rmustacc: one some debug kernels, I see src.illumos.org/source/xref/illumos….c?r=bbf21555&mo=88163&fi=3261#3261 get emitted from the pvscsi driver -- I think src.illumos.org/source/xref/illumos…pters/pvscsi/pvscsi.c?r=7a73cc88#70 is probably the culprit (I suspect that value is a 32-bit kernel holdover), but wanted to know if

that seems reasonable

looking at the pvscsi freebsd src, i can't find any similar restrictions for the DMA memory (it seems to allow anything if I read the source correctly)

You suspect which value is a 32-bit kernel holdover?

The count-max?

the dma_attr_count_max

We'll have to look at the driver.

I assume this is only documented by vmware via a linux driver.

yes... there's also a freebsd driver as well (what I was looking at)

Sure, did VMware write that though?

that i don't know

But if we're exceeding that, that means someone asked for a >2 GiB DMA alloc.

That seems... excessive.

If I counted my bits right.

hmm.. might need to run some dtrace and see if I can see what's doing that

err calling that to go down that path

This is reproducible?

not consistently

but i've hit it not too infrequently as well

there is a correlation with a hung I/O and those messages as well as a failure to bind the ARQ buffer (which I suppose if somethings causing a large DMA allocation, would explain that)

It seems weird to be asking for >2 GiB of DMA.

So I think we want to understand if that's actually it or something else.

So yeah, using DTrace to see what the request was that failed would be good.

Sorry, I was misreading the attr a little bit.

If count_max is 2 GiB, we're saying that's how much fits in a single cookie.

Now, the maxxfer here is which basically a lot lower 256 * 4K

What led you to the count?

it was the only thing that seemed to have any restrictions compared to the freebsd src

but might need to go back through it

or look at the linux source

... well maybe later

(customer problem... as an aside, I hate iSCSI)

:P

jbk: I think we'll just want to see what the actual DMA request is. After that it'll become a lot clearer.

hrm.. no luck yet (since it happens after userland is up, I may just create a simple service to start dtrace to try to capture it)

though digging more, I see that pvscsi doesn't report a SCSI_CAP_DMA_MAX value.. i wonder if that might be a contributing factor

That's ceratinlynot going to help.

The max request size there is about 1 MiB, which I think we could exceed in a single request.

So if someone has asked for more than 1 MiB then you'll be in trouble.

jbk: This could be a use for anonymous DTrace fwiw.