17:30:35 <sommerfeld> jbk: you said you had some i40e patches you wanted to share.  I'm seeing some anomalous behavior on a couple i40e's.  Running iperf3 from a gigabit client; it can sustain gigabit line rate for ipv4 but not ipv6.  But one of the affected machines hosts a bhyve VM and it can sustain gigabit line rate over v6 to that vm.
17:33:08 <jbk> i don't think it'd impact that.. the changes are more aroudn capping the size of DMA memory allocation, but I do need to put them up...
17:34:33 <sommerfeld> yeah, I'm mostly looking to avoid reinventing the wheel when I start digging through the driver.
17:35:33 <danmcd> There are open bugs either on illumos or on SmartOS (pre Oxide diaspora likely) about IPv6 vs. i40e.
17:35:42 <sommerfeld> I suspect Dan's common-code update is more likely to be relevant.
17:36:23 <jbk> ahh.. https://github.com/jasonbking/illumos-gate/commit/cf5a84c2df46d1c39748a270e50ad373b510200e
17:36:24 <fenix> → GitHub commit cf5a84c: BSR-14223 / BSR-13428 / BSR-14814 (committed)
17:36:29 <jbk> i figured i had it in a branch
17:37:03 <danmcd> Possibly.  Also there have been recent IPv6 things for generic improvements (Oxide written) that latest illumos can exploit.  If i40e still can't do it, then we know it's i40e-specific.
17:45:13 <sommerfeld> connstat shows bursts of inunordersegs so either packet reordering or drops (haven't sorted out which yet)
19:20:14 <sommerfeld> Looks like drops.   Shrinking tcp window size (which I'd tuned up for better WAN performance..) appears to mitigate it.   
19:24:48 <sommerfeld> perhaps we're not scheduling rx ring polling frequently enough?
19:28:10 <sommerfeld> there's plenty of idle cpu on the receiver (no cpu less than about 88% idle)
19:33:29 <jbk> are you polling? it should be raising an interrupt unless mac has switched to polling
19:33:39 <sommerfeld> not entirely sure.
19:34:08 <jbk> maybe the rx ring is too small?
19:34:18 <jbk> i can't recall the default sizing offhand
19:34:42 <sommerfeld> looks like rx_bind_norcb is getting bumped when it's dropping
19:37:03 <sommerfeld> looks like 1K default, 4K max (entries)
19:37:35 <sommerfeld> this looks like an i40e.conf tunable?  
19:48:38 <sommerfeld> (just put "rx_ring_size=4096" on a line by itself in i40e.conf ?)
20:47:32 <jbk> i think so.. I don't think it's wired up to dladm (i haven't looked closely at the driver in a while)
21:09:22 <jbk> rmustacc: I noticed uts/common/sys/pci_cfgacc.h in your stlouis branch doesn't have an oxide copyright for the added PCI_CFGACC_xxx() macros (I was hoping to use the pcie_cfgspace.c as the basis for a 'pci_mech_mmio' (i.e. what the pci_xxx_func pointers get assigned) and noticed that..
21:09:33 <jbk> (in case it was an oversight)
21:11:49 <richlowe> didn't stlouis already have the concrete impls too?
21:12:14 <richlowe> the way I did my stuff is unfortunately I started with stlouis and then I removed 99% of it, and now I don't remember
21:12:28 <jbk> yeah, I was trying to adapt the pcie_cfgspace.c
21:12:50 <richlowe> yeah, keith made some assumptions that you get from designing your own hardware
21:13:25 <jbk> (basically with a few small tweaks)..
21:14:20 <jbk> since on the non-oxide stuff, AFAIK if it's going to be used, we need to use the MCFG table
22:16:18 <rmustacc> jbk: If we didn't put something there then don't worry about it.
22:16:38 <rmustacc> Let me konw if I can help usefully on this.