19:45:24 <jbk> rmustacc: one some debug kernels, I see https://src.illumos.org/source/xref/illumos-gate/usr/src/uts/i86pc/io/rootnex.c?r=bbf21555&mo=88163&fi=3261#3261 get emitted from the pvscsi driver -- I think https://src.illumos.org/source/xref/illumos-gate/usr/src/uts/common/io/scsi/adapters/pvscsi/pvscsi.c?r=7a73cc88#70 is probably the culprit (I suspect that value is a 32-bit kernel holdover), but wanted to know if 
19:45:30 <jbk> that seems reasonable
19:46:13 <jbk> looking at the pvscsi freebsd src, i can't find any similar restrictions for the DMA memory (it seems to allow anything if I read the source correctly)
19:48:05 <rmustacc> You suspect which value is a 32-bit kernel holdover?
19:48:22 <rmustacc> The count-max?
19:48:23 <jbk> the dma_attr_count_max
19:49:54 <rmustacc> We'll have to look at the driver.
19:50:09 <rmustacc> I assume this is only documented by vmware via a linux driver.
19:50:28 <jbk> yes... there's also a freebsd driver as well (what I was looking at)
19:50:44 <rmustacc> Sure, did VMware write that though?
19:51:03 <jbk> that i don't know
19:51:07 <rmustacc> But if we're exceeding that, that means someone asked for a >2 GiB DMA alloc.
19:51:11 <rmustacc> That seems... excessive.
19:51:29 <rmustacc> If I counted my bits right.
19:53:40 <jbk> hmm.. might need to run some dtrace and see if I can see what's doing that
19:53:51 <jbk> err calling that to go down that path
19:54:47 <rmustacc> This is reproducible?
19:54:58 <jbk> not consistently
19:55:10 <jbk> but i've hit it not too infrequently as well
19:55:54 <jbk> there is a correlation with a hung I/O and those messages as well as a failure to bind the ARQ buffer (which I suppose if somethings causing a large DMA allocation, would explain that)
19:56:18 <rmustacc> It seems weird to be asking for >2 GiB of DMA.
19:56:28 <rmustacc> So I think we want to understand if that's actually it or something else.
19:56:42 <rmustacc> So yeah, using DTrace to see what the request was that failed would be good.
19:57:31 <rmustacc> Sorry, I was misreading the attr a little bit.
19:57:43 <rmustacc> If count_max is 2 GiB, we're saying that's how much fits in a single cookie.
19:58:25 <rmustacc> Now, the maxxfer here is which basically a lot lower 256 * 4K
19:58:58 <rmustacc> What led you to the count?
20:05:32 <jbk> it was the only thing that seemed to have any restrictions compared to the freebsd src
20:05:38 <jbk> but might need to go back through it
20:05:42 <jbk> or look at the linux source
20:15:13 <jbk> ... well maybe later
20:15:28 <jbk> (customer problem... as an aside, I hate iSCSI)
20:15:30 <jbk> :P
21:09:52 <rmustacc> jbk: I think we'll just want to see what the actual DMA request is. After that it'll become a lot clearer.
22:36:16 <jbk> hrm.. no luck yet (since it happens after userland is up, I may just create a simple service to start dtrace to try to capture it)
22:36:54 <jbk> though digging more, I see that pvscsi doesn't report a SCSI_CAP_DMA_MAX value.. i wonder if that might be a contributing factor
22:52:47 <rmustacc> That's ceratinlynot going to help.
22:53:19 <rmustacc> The max request size there is about 1 MiB, which I think we could exceed in a single request.
22:53:29 <rmustacc> So if someone has asked for more than 1 MiB then you'll be in trouble.
22:53:38 <rmustacc> jbk: This could be a use for anonymous DTrace fwiw.