hello all, does illumos / smartos support SR-IOV ? can RDMA run with bhyve vm?

thanks

i'm 90% sure we do not do SR-IOV

sjorge: if so, can’t use RDMA with the Bhyve VM?

I doubt that would work

where had related docs? I have not find out how to use RDMA with bhyve, such as NoF (NVMe over RDMA/RoCEv2)

any advice, best wishes

i think your best best would be doing pci pass through of the device if possible..

Most of the SR-IOV configuration to create the virtual pci functions is in the driver for the respective devices when they are setup on boot

e.g. with my connectx 4 nics docs.nvidia.com/networking/display/…gle+root+io+virtualization+(sr-iov)

For our NIC drivers we tend to prefer to use the device's packet processing and classification features to steer traffic into dedicated rings which can themselves be mapped to individual VNICs. For example, Intel's VMDq support. I don't believe we've setup any SR-IOV support in any of our drivers, at least not the ones I'm familiar with.

rzezeski is correct. And our mlxcx(4D), which drives CX-4, does not.

danmcd - to be clear to myself - mxlxcx uses the packet processing and classification features, but does not setup SR-IOV?

mlxcx does exactly what rzezeski says, so yes no SR-IOV.

[illumos-gate] 16228 convert git-pbchk.1onbld to mdoc -- Bill Sommerfeld <sommerfeld⊙ho>

Looping back on a conversation from a couple of weeks ago, I have taken a 240GB NVMe drive, used format(8) on it to slice it up into three roughly-equal-sized parttions, and now have slice0 of it as my zpool's slog.

Given this device doesn't SEEM to have any namespaces beyond the one, it seemed the best thing to do.

I've had to slice ssds much in the past for slog/l2arc purposes (working primarily for small businesses we are usually dealing with limited physical disk space), and even though it isn't as clean, I haven't had a problem with it.

I've done it too, despite the heckling from the openzfs irc channel.

danmcd: speaking of mlxcx, have you had any luck w the 100gb cards and using a breakout cable into 4x25gb SFPs?

or is that something that's even been attempted?

Treating it as 4 distinct interfaces?

no.. just one of them

Is the far end joining them together?

@jbk --> forgot to update on that.

Turns out I had a test-tool problem. Namely iperf3 is single-threaded, something unlike netperf isn't made plainly obvious.

Once I started concurrent iperf3s on different ports I got better results esp. on the EPYC.

It needs work (tops out at 50-60Gbits w/o any LSO or some of arekinath's other goodies) but it's not horrid as I'd feared.

I'll be very interested to see when they get returned to their low-powered original home.

i think the idea is to plug in a 100gb NIC into a 25gb switch port.. I'm far, far away from the physical hardware, but I'm guessing there's some sort of form factor issue where you can't just plug it in and have it auto negotiate down

and need different sfps

that sounds vaguely familiar. jbk: probably sfp28 (25gbit/s) vs qsfp28 (4x25gbit/s)

(there is also apparently OSFP out there -- 8x whatever)

yeah.. IIUC, the card itself only accepts qsfp28

danmcd: honestly, that's pretty decent compared to what I would have expected you to report. so that's good

rzezeski: that's with our new-DC-build EPYC server. The test machines I am now back to are literally Atom (Skylake era). I'm literally about to see how they do again, now that I understand WTaF was going on.

Those numbers are also with MTU=9k and max_buf cranked to 16M (prob. overkill on that one).

If it's for intra-DC comms I think large MTU is totally fair, we can have nice things.

Ahhh I'm seeing the CPU limitations (of course, one wonders how much of that is code we have that could do better with less).

danmcd: i take it the system must have a lot of ram :)

The EPYC, oh yes.

i think i mentioned, we had a system where the integrator messed up and shipped it w/ 64gb of ram

turns out that wasn't enough for 2 4-way 25gb aggrs + vnics w/ mlxcx :)

There are tunables you can adjust on all that you know, right?

The driver memory, yes.

i was more impressed than anything

(and once we fixed the integrators goof up, it was fine)

I just tested oci out of boredom again, I still need to set pit_is_broken=1 to be able to boot and virtio_scsi still does not work, disks are not available

this is what I have for prtconf -vp

the odd thing that smartos booted without modifying pit