16:19:52 <sjorge> jbk / papertigers would the tcp CC have any effect on when a zone/socket would release it's held buffers?
16:20:22 <sjorge> I switched to cubic on the host side and switched the linux VMs back cubic (had the VMs on newreno to match the native stuff)
16:20:37 <sjorge> I seem to hit the viona starvation less
16:20:44 <sjorge> Still hit it, though
16:20:48 <sjorge> Not sure if placebo or not
17:13:54 <jclulow> I would have to guess that it's a placebo 
17:14:01 <jclulow> But anything is possible 
17:55:49 <sjorge> yeah I'm leaning towards placebo too
17:56:21 <sjorge> as it has been a mixed bag in the past too where things go fine for a bit, then i keep hitting the issue multiple times a day,...
18:09:09 <nbjoerg> would be nice to have a vm flag that packets to/from the vm should just be copied...
18:37:11 <sjorge> in theory, swapping from viona to plain virtio-net, but the performance is like 10-20MB/s max which makes all the nfs mounts basically unusable and eventually whatever is using thme starves for IO in the D state.
18:37:45 <sjorge> IIRC most of viona's perf comes from only doing all the extra work once it needs to leave the host
18:43:02 <nbjoerg> in this case, we kind of want the opposite
18:43:30 <nbjoerg> maybe with some extra accounting: keep track of how many queue entries are left and if it is below some threshold, switch to copying
19:11:25 <sjorge> From what I understand, it's something in sockfs (?) not releasing the buffers when it's done with it and hits some sort of low/high watermark that is in place to prevent it from holding onto those buffers.
19:11:50 <sjorge> So there is already some sort of accounting tracking it (not just coming from viona, the receiving netstack doesn't care where it  comes from)
19:12:47 <rmustacc> It only matters because they're loaned buffers.
19:15:09 <sjorge> Correct, as reducing vqsize makes it trigger more often and increasing it makes it trigger less often (I have it at max vqsize at the moment0
21:51:09 <sjorge> is the 16k a virtio max or is that something on our side?
21:51:17 <sjorge> (for vqsize)
21:56:17 <richlowe> that might be a legacy thing
21:56:27 <richlowe> in that I remember thinking that, pmooney is the expert.
21:57:45 <richlowe> oh, that's memory from me merging the virtio drivers together, so especially don't trust that.
21:58:11 <pmooney> max virtqueue length is 32k
21:58:21 <pmooney> that is, the number of entries in the virtqueue
21:58:48 <richlowe> the mmio stuff caps at some low number, the pci stuff just lets stuff default I think?
22:00:17 <pmooney> well and for legacy virtio, the virtqueue size is not negotiated, but rather specified by the device
22:00:44 <pmooney> so cranking it way up can be a bit onerous for the guest, having to allocate adequate memory for that dictated size
22:01:50 <richlowe> ok, that'd be what I was seeing and confused by 
22:02:53 <richlowe> at some point I hope someone smart updates us to the current spec, and takes a look at non-pci stuff.
22:02:59 <richlowe> because I'm trying to get weasel out of it
22:03:02 <richlowe> s/get//
22:07:04 <sjorge> i'll see if i can bump it to 32k tomorrow, i'd rather wast memory if it means less issues
22:08:22 <richlowe> what issues?
22:08:26 <richlowe> sorry, I'm bad at keeping up
22:08:58 <jclulow> Is there a bug open for this problem?
22:11:02 <richlowe> oh yeah, definitely file a bug if you haven't.
22:11:34 <andyf> I think it's illumos 13463 (fenix)
22:11:35 <fenix> BUG 13463: bhyve locks up when viona ring buffers on loan to a local zone are not returned by sockfs (New)
22:11:35 <fenix> ↳ https://www.illumos.org/issues/13463
22:12:00 <sjorge> bingo
22:12:20 <sjorge> looks like 32k does work
22:13:02 <sjorge> s/work/gets accepted by zadm/
22:13:27 <andyf> oh, well that could just be zadm. I just looked at bhyve and it should accept 32K for viona (that's the max)
22:14:18 <andyf> did you put in 32768?
22:14:24 <sjorge> yep
22:14:27 <sjorge> it works
22:14:35 <andyf> I ask because zadm checks for the specific powers of two
22:14:59 <andyf> Time for bed for me, I read that as does not work, sorry.
22:15:10 <sjorge> my notes say to set it to 16k (max)
22:15:23 <sjorge> and i winder if it could go higher, so yes it can
22:15:25 <jclulow> so the VM transmits packets to the local zone, by loaning buffers, basically?
22:15:58 <sjorge> As far as i understand it, yes
22:16:11 <jclulow> It really feels like we could at least add an /etc/system tuneable or something to just force it to copy always
22:16:18 <sjorge> in my case its our nfs/server in a zone not returning them in some cases
22:16:20 <jclulow> so that it would at least work
22:18:05 <richlowe> wait, is it in nfs _in specific_ doing something weird?
22:18:12 <richlowe> or does that just happen to be what screws you, but anything could?
22:18:47 <jclulow> I believe if you have a process in a zone that accepts a connection but never reads from it
22:18:50 <jclulow> you would end up in the same boat
22:19:06 <jclulow> nothing releases that memory back, it'll just sit in your recvq
22:19:18 <sjorge> not nfs specific
22:19:24 <jclulow> The complex but complete solution would be to claw them back from queues
22:19:47 <jclulow> But the absolute minimum bandaid here is to force a copy instead of a loan in this case
22:19:53 <jclulow> so that it just never happens
22:20:14 <sjorge> i think one customer at joyent also hit the same
22:20:19 <sjorge> issue
23:33:08 <jbk> jclulow: at least from what I recall, and I think is in the ticket, even a process using port_get() or poll() to wait to read can trigger it (as it seems that somehow we're not signaling when data is there, so the mblk_ts keep getting linked)