jbk: you said you had some i40e patches you wanted to share. I'm seeing some anomalous behavior on a couple i40e's. Running iperf3 from a gigabit client; it can sustain gigabit line rate for ipv4 but not ipv6. But one of the affected machines hosts a bhyve VM and it can sustain gigabit line rate over v6 to that vm.

i don't think it'd impact that.. the changes are more aroudn capping the size of DMA memory allocation, but I do need to put them up...

yeah, I'm mostly looking to avoid reinventing the wheel when I start digging through the driver.

There are open bugs either on illumos or on SmartOS (pre Oxide diaspora likely) about IPv6 vs. i40e.

I suspect Dan's common-code update is more likely to be relevant.

ahh.. jasonbking/illumos-gate cf5a84c

→ GitHub commit cf5a84c: BSR-14223 / BSR-13428 / BSR-14814 (committed)

i figured i had it in a branch

Possibly. Also there have been recent IPv6 things for generic improvements (Oxide written) that latest illumos can exploit. If i40e still can't do it, then we know it's i40e-specific.

connstat shows bursts of inunordersegs so either packet reordering or drops (haven't sorted out which yet)

Looks like drops. Shrinking tcp window size (which I'd tuned up for better WAN performance..) appears to mitigate it.

perhaps we're not scheduling rx ring polling frequently enough?

there's plenty of idle cpu on the receiver (no cpu less than about 88% idle)

are you polling? it should be raising an interrupt unless mac has switched to polling

not entirely sure.

maybe the rx ring is too small?

i can't recall the default sizing offhand

looks like rx_bind_norcb is getting bumped when it's dropping

looks like 1K default, 4K max (entries)

this looks like an i40e.conf tunable?

(just put "rx_ring_size=4096" on a line by itself in i40e.conf ?)

i think so.. I don't think it's wired up to dladm (i haven't looked closely at the driver in a while)

rmustacc: I noticed uts/common/sys/pci_cfgacc.h in your stlouis branch doesn't have an oxide copyright for the added PCI_CFGACC_xxx() macros (I was hoping to use the pcie_cfgspace.c as the basis for a 'pci_mech_mmio' (i.e. what the pci_xxx_func pointers get assigned) and noticed that..

(in case it was an oversight)

didn't stlouis already have the concrete impls too?

the way I did my stuff is unfortunately I started with stlouis and then I removed 99% of it, and now I don't remember

yeah, I was trying to adapt the pcie_cfgspace.c

yeah, keith made some assumptions that you get from designing your own hardware

(basically with a few small tweaks)..

since on the non-oxide stuff, AFAIK if it's going to be used, we need to use the MCFG table

jbk: If we didn't put something there then don't worry about it.

Let me konw if I can help usefully on this.