neirac: I'd be curious if you compiled and ran gist.github.com/jasonbking/6f1475e4fe1684042b9fb17c617d3151 on there to see what it returns

(might be nothing)

jbk I thought of that, but omnios bloody did not have that change, I'll try to test it

well this will tell if it'd help or not

jbk: rmustacc: rzezeski, some NIC support RoCEv2, to use SR-IOV for a NIC can pass through the configured VFs to Bhyve VMs to run RDMA over RoCE; not sure if vnic would run RDMA or support RoCEv2 ? I have not find document say vnic support RoCEv2, a big thanks

tozhu: No, VNICs are not setup for that. What you are saying is technically possible. But we currently don't support setting up VFs and handing them off to VMs. In our deployments we have preferred to have the hypervisor (illumos) control the classification/switching with hardware acceleration from the NIC by mapping MAC addresses and VLANs to specific queues, and use the viona/VIRTIO device to deliver traffic.

rzezeski: Thank you, so there is no way to do RDMA/RoCE on Bhyve? my usage case is rdbms,the SDS (Software Defined Storage) targets supports RDMA over RoCEv2, I’m looking for the solution for RDMA over RoCEv2 for the Bhyve VM for the initinator,

I read this link say virtio-net added RoCE support on Linux ,lore.kernel.org/all/20220511095900.343-1-xieyongji⊙bc/T , not sure if illumos had it now. best wishes

there really hasn't been much interest in rdma in illumos to date

(doesn't mean taht can't change, just how it's been historically)

hrm..

i think i'll save this crash for tomorrow

in mac_util.c

...hrm.. and i feel like i've seen this before

with emulated e1000g panic on DEBUG in mac_sw_lso

I think the RDMA based application is very importand, esp for NVMe over Fabric, with high performance database application, works with Bhyve on illumos, need very low latency, else the DB size is limited by the phy host configured disk, or have to work with very low performance mode due to very high latency with network

anyone have access to the opengrok instance? seems like it might need to be kicked

jbk I did not read your message I thought it was a vioscsi commit, because I saw a commit sometime ago for vioscsi, I cannot compile the program omnios boots I cannot install software

jbk: keep that panic (in mac_sw_lso) around, that's precisely the sort of function that can be tested well with ktest. We should be able to extract your input mblk from the dump and make a regression test along with the fix

I'm happy to grab a copy of it and give it a look

rzezeski: looking forward to playing with ktest..

sommerfeld: Good to know. I kind of let it stall out because I thought maybe there wasn't too much interest (also dealt with some life stuff that kept me out of the game for a bit). But I was recently assured that it would be useful. So I'm happy to do the work to get it in an acceptable state.

[illumos-gate] 15973 Win11 SMB set atime=-2 gets internal error -- Toomas Soome <tsoome⊙mc>

[illumos-gate] 15974 SMB logoff door up-call can suffer a thundering herd -- Gordon Ross <gwr⊙rc>

rzezeski: ok... it looks like somesort of mismatch w/ checksum offload and lso when i looked at it briefly

it could be the emulated e1000 is somewhat brain dead

but i'll keep that panic around

tozhu What OS are you running your DBS on inside of Bhyve?

Are you wanting to pass in a PCIe device so that you can run an RDMA stack inside of it? Or are you thinking that the paravirtualized NVMe in Bhyve can be pass through to the host through NVMoF? Sorry, what you said brings up a lot of questions to me

Although there isn't SR-IOV support to pass through a virtual PCI function with illumos as a host in the drivers, NVMe/TCP is pretty close performance wise, with less CPU overhead and additional latency than one would think

Actually, with NVMe/TCP, the only applications that seem worth it to benefit from RDMA is HPC stuff with MPI. With the availability of CPU cores these days I'd rather have something IP/Ethernet based

neirac: i can give you a compiled binary if that'd work

rzezeski: also, I'll hopefully try to look again at the ktest stuff soon...