14:43:21 Hi, all 14:45:51 I have a question about ZFS file system and Omni 14:46:52 Is using ZFS under OmniOS mandatory or can one use a different file system? 14:47:57 This is the ONLY issue keeping me from switching to Tribblex! UFS maybe? 14:49:02 I’d like to run OMNI instead 14:50:02 The bootable disk on OmniOS must be ZFS (the beadm(8) command requires it). 14:50:17 Do you have a UFS set of data you wanna migrate? or is there some other reason you want UFS? 14:51:58 I dislike ZFS! DragonflyBSD had hammer2, which is a nice alternative to ZFS! 14:52:22 As far as I know, Tribblix is the only illumos distribution where root-on-ufs is a first-class citizen 14:52:24 Matt was very nice! 14:53:33 tribble thanks, Peter 14:54:06 Do, Omni is only ZFS? 14:54:19 So 14:54:20 But generally there are a lot of assumptions about using zfs now embedded in things like install, packaging, and zones 14:58:07 I did root on ufs for a couple of reasons, (1) to prove I could, and (2) for systems with limited memory (much less than 1G). Outside those 2 crazy cases, use ZFS. 17:17:16 danmcd, I have collected a bunch more data about the IPv4 Vs IPv6 speed difference. The raw data is at https://pastebin.com/zkcs3iWa 17:17:52 but it doesn't look like the aggregate is the problem. To cherry-pick the 'best' result from one direction of the tests: 17:18:19 IPv4: 17:18:21 [ ID] Interval Transfer Bandwidth 17:18:22 [ 4] 0.00-60.00 sec 54.2 GBytes 7.76 Gbits/sec sender 17:18:22 [ 4] 0.00-60.00 sec 54.2 GBytes 7.76 Gbits/sec receiver 17:18:26 IPv6: 17:18:38 [ 4] 0.00-60.00 sec 29.9 GBytes 4.28 Gbits/sec sender 17:18:38 [ 4] 0.00-60.00 sec 29.9 GBytes 4.28 Gbits/sec receiver 17:19:03 (please check the raw data in the pastebin, this is a very surface level summary here.) 17:19:47 Note those numbers are just one ixgbe device directly cabled to another ixgbe device. No switch, no aggregate. 17:32:57 sommerfeld, I think you were also looking at this. 17:33:22 Note that I did not try changing MTU in these tests as I'm trying to match the real world these hosts work in. 17:57:47 When I was asking about aggrs vs. no aggrs, that needs to be the case on both sides. 17:58:02 I'm trying to eliminate "aggr" as part of the problem. 17:58:32 (Pardon latency ; meeting in 2min) 18:03:18 both sides were configured to match for each test. 18:03:26 testing w/o aggr was done w/o aggr on both hosts. 18:03:47 I'll update the notes to make that clearer. 18:23:04 I think at this point profiling/flamegraphs would be useful to see where the cpu time is going for v4 vs v6. 18:24:31 happy to run any tests you'd like to suggest. 18:25:17 right now the hosts are set up with an aggregate. I'd like to get whatever tests need that done before blowing them away (again). 18:26:11 My goal is to gather whatever data is needed to open a ticket (I presume with illumos). 18:28:21 Based on the numbers I'm seeing, I think aggregate isn't actually relevant to this problem. However, I'm not really comfortable calling that fact and ignoring it w/o someone else's eyes on it as well. 20:39:40 If aggr's a factor it's a minimal factor. Thanks for the clarification. 20:41:09 I should find a cat6 cable and link two unused ixgbes on two ixgbe-wielding kebecloud nodes... can probably duplicate your non-aggr experiments quickly, and without a switch in the way. 20:47:10 there's no switch in this configuration. 20:47:21 (just to be clear) 20:47:34 Understood. 20:47:55 I do have a switch I can add if you want to test that but I don't expect you'll want/need such a thing. 20:48:02 This means I should easily be able to see similar results (albeit maybe CPU-capped since these are old Haswell E boxes from 2014). 20:48:21 Don't bother! Knowing there's no switch in your tests is good enough. 20:51:28 * nomad nods 20:52:14 Should I post what I've pasted (with clarifications about no aggr and no switch) as a ticket for illumos or should I wait for further instructions? 20:52:42 filing tickets is always good, they can be updated later 20:55:31 also, I don't have i40e cards in the test hosts, we're seeing the same behavior with those connections. (Can't speak definitively because those are aggregates and go through a Juniper switch.) 21:04:57 https://www.illumos.org/issues/16590 21:04:58 → BUG 16590: speed difference between IPv4 & IPv6 on 10Gbit connections (New) 21:07:56 thanks