Hi, all

I have a question about ZFS file system and Omni

Is using ZFS under OmniOS mandatory or can one use a different file system?

This is the ONLY issue keeping me from switching to Tribblex! UFS maybe?

I’d like to run OMNI instead

The bootable disk on OmniOS must be ZFS (the beadm(8) command requires it).

Do you have a UFS set of data you wanna migrate? or is there some other reason you want UFS?

I dislike ZFS! DragonflyBSD had hammer2, which is a nice alternative to ZFS!

As far as I know, Tribblix is the only illumos distribution where root-on-ufs is a first-class citizen

Matt was very nice!

tribble thanks, Peter

Do, Omni is only ZFS?

So

But generally there are a lot of assumptions about using zfs now embedded in things like install, packaging, and zones

I did root on ufs for a couple of reasons, (1) to prove I could, and (2) for systems with limited memory (much less than 1G). Outside those 2 crazy cases, use ZFS.

danmcd, I have collected a bunch more data about the IPv4 Vs IPv6 speed difference. The raw data is at pastebin.com/zkcs3iWa

but it doesn't look like the aggregate is the problem. To cherry-pick the 'best' result from one direction of the tests:

IPv4:

[ ID] Interval Transfer Bandwidth

[ 4] 0.00-60.00 sec 54.2 GBytes 7.76 Gbits/sec sender

[ 4] 0.00-60.00 sec 54.2 GBytes 7.76 Gbits/sec receiver

IPv6:

[ 4] 0.00-60.00 sec 29.9 GBytes 4.28 Gbits/sec sender

[ 4] 0.00-60.00 sec 29.9 GBytes 4.28 Gbits/sec receiver

(please check the raw data in the pastebin, this is a very surface level summary here.)

Note those numbers are just one ixgbe device directly cabled to another ixgbe device. No switch, no aggregate.

sommerfeld, I think you were also looking at this.

Note that I did not try changing MTU in these tests as I'm trying to match the real world these hosts work in.

When I was asking about aggrs vs. no aggrs, that needs to be the case on both sides.

I'm trying to eliminate "aggr" as part of the problem.

(Pardon latency ; meeting in 2min)

both sides were configured to match for each test.

testing w/o aggr was done w/o aggr on both hosts.

I'll update the notes to make that clearer.

I think at this point profiling/flamegraphs would be useful to see where the cpu time is going for v4 vs v6.

happy to run any tests you'd like to suggest.

right now the hosts are set up with an aggregate. I'd like to get whatever tests need that done before blowing them away (again).

My goal is to gather whatever data is needed to open a ticket (I presume with illumos).

Based on the numbers I'm seeing, I think aggregate isn't actually relevant to this problem. However, I'm not really comfortable calling that fact and ignoring it w/o someone else's eyes on it as well.

If aggr's a factor it's a minimal factor. Thanks for the clarification.

I should find a cat6 cable and link two unused ixgbes on two ixgbe-wielding kebecloud nodes... can probably duplicate your non-aggr experiments quickly, and without a switch in the way.

there's no switch in this configuration.

(just to be clear)

Understood.

I do have a switch I can add if you want to test that but I don't expect you'll want/need such a thing.

This means I should easily be able to see similar results (albeit maybe CPU-capped since these are old Haswell E boxes from 2014).

Don't bother! Knowing there's no switch in your tests is good enough.

Should I post what I've pasted (with clarifications about no aggr and no switch) as a ticket for illumos or should I wait for further instructions?

filing tickets is always good, they can be updated later

also, I don't have i40e cards in the test hosts, we're seeing the same behavior with those connections. (Can't speak definitively because those are aggregates and go through a Juniper switch.)

→ BUG 16590: speed difference between IPv4 & IPv6 on 10Gbit connections (New)

thanks