jbk / papertigers would the tcp CC have any effect on when a zone/socket would release it's held buffers?

I switched to cubic on the host side and switched the linux VMs back cubic (had the VMs on newreno to match the native stuff)

I seem to hit the viona starvation less

Still hit it, though

Not sure if placebo or not

I would have to guess that it's a placebo

But anything is possible

yeah I'm leaning towards placebo too

as it has been a mixed bag in the past too where things go fine for a bit, then i keep hitting the issue multiple times a day,...

would be nice to have a vm flag that packets to/from the vm should just be copied...

in theory, swapping from viona to plain virtio-net, but the performance is like 10-20MB/s max which makes all the nfs mounts basically unusable and eventually whatever is using thme starves for IO in the D state.

IIRC most of viona's perf comes from only doing all the extra work once it needs to leave the host

in this case, we kind of want the opposite

maybe with some extra accounting: keep track of how many queue entries are left and if it is below some threshold, switch to copying

From what I understand, it's something in sockfs (?) not releasing the buffers when it's done with it and hits some sort of low/high watermark that is in place to prevent it from holding onto those buffers.

So there is already some sort of accounting tracking it (not just coming from viona, the receiving netstack doesn't care where it comes from)

It only matters because they're loaned buffers.

Correct, as reducing vqsize makes it trigger more often and increasing it makes it trigger less often (I have it at max vqsize at the moment0

is the 16k a virtio max or is that something on our side?

(for vqsize)

that might be a legacy thing

in that I remember thinking that, pmooney is the expert.

oh, that's memory from me merging the virtio drivers together, so especially don't trust that.

max virtqueue length is 32k

that is, the number of entries in the virtqueue

the mmio stuff caps at some low number, the pci stuff just lets stuff default I think?

well and for legacy virtio, the virtqueue size is not negotiated, but rather specified by the device

so cranking it way up can be a bit onerous for the guest, having to allocate adequate memory for that dictated size

ok, that'd be what I was seeing and confused by

at some point I hope someone smart updates us to the current spec, and takes a look at non-pci stuff.

because I'm trying to get weasel out of it

s/get//

i'll see if i can bump it to 32k tomorrow, i'd rather wast memory if it means less issues

what issues?

sorry, I'm bad at keeping up

Is there a bug open for this problem?

oh yeah, definitely file a bug if you haven't.

I think it's illumos 13463 (fenix)

BUG 13463: bhyve locks up when viona ring buffers on loan to a local zone are not returned by sockfs (New)

↳ illumos.org/issues/13463

bingo

looks like 32k does work

s/work/gets accepted by zadm/

oh, well that could just be zadm. I just looked at bhyve and it should accept 32K for viona (that's the max)

did you put in 32768?

yep

it works

I ask because zadm checks for the specific powers of two

Time for bed for me, I read that as does not work, sorry.

my notes say to set it to 16k (max)

and i winder if it could go higher, so yes it can

so the VM transmits packets to the local zone, by loaning buffers, basically?

As far as i understand it, yes

It really feels like we could at least add an /etc/system tuneable or something to just force it to copy always

in my case its our nfs/server in a zone not returning them in some cases

so that it would at least work

wait, is it in nfs _in specific_ doing something weird?

or does that just happen to be what screws you, but anything could?

I believe if you have a process in a zone that accepts a connection but never reads from it

you would end up in the same boat

nothing releases that memory back, it'll just sit in your recvq

not nfs specific

The complex but complete solution would be to claw them back from queues

But the absolute minimum bandaid here is to force a copy instead of a loan in this case

so that it just never happens

i think one customer at joyent also hit the same

issue

jclulow: at least from what I recall, and I think is in the ticket, even a process using port_get() or poll() to wait to read can trigger it (as it seems that somehow we're not signaling when data is there, so the mblk_ts keep getting linked)