I left the poor guy at the datacenter at 4 in the morning

any new message in /var/adm/messages ?

neirac nope :/ altho I asked ararat to go home now. I'll go tomorrow and check myself.

neirac what's the best NIC that I can get for basic support? We just need to run DNS and DHCP zones

antranigv check this: docs.tritondatacenter.com/private-c…ud/hardware#network-interface-cards

antranigv there is also this list illumos.org/hcl

[illumos-gate] 16202 swapfs_minfree defaults to an absurdly high value -- Bryan Cantrill <bryan⊙oc>

Hibernate to disk like in Linux, would be a very nice feature for OpenIndiana illumos.

antranigv I got stock overnight once, security guard forgot about me. Couldn't leave afterwards.

*stuck

On the upside, I got paid the entire time I was stuck so ...

sjorge omgggg same story here! :D and I was like... 16, so panicked like hell. then I said "fuck it, back to work" and delivered all of my weekly tasks in a night xD

I'm having a hard time understanding if illumos has SIGINFO (^T) or not.

It's there, not much actually uses it though (can't think of anything other than dd offhand)

see illumos.org/issues/4493

→ FEATURE 4493: want siginfo (Resolved)

ptribble looks like I know what I want to contribute :))

sommerfeld: FWIW, I suspect you can just move to test and RTI on 16163. If you want to get more review that's fine, but I don't think it's needed!

jclulow: just wanted to let anyone else speak up on it. As I noted in the bug I've run the zfs test suite; there are a few failures but none in scrub-related tests.

probably could use someone with experience with the test suite looking over the fails to make sure I didn't miss anything.

I think that's pretty much on us to try and figure out at the RTI point generally, FWIW. It's a bit of a mess in there right now

Would love to have a reliably green board on those tests, but we're not quite there haha

i have a perhaps related fix as well but i've been drowning in customer issues, so I didn't have a chance to look at it closely

also arguably fallout from the sequential scrub work

in that we try to keep the distribution of I/Os across the pool roughly balanced

or rather there's supposed to be a limit on the # of outstanding zios per top level vdev

except that the way it's enforced means things can get extremely lopsided

(basically instead of lookint at each top level vdev, it takes the per top level vdev limit * # top level vdevs and forces that total limit on the whole pool)

my change fixes that

the problem with that is since (aside from syncing zios) we issue zios sorted by (priority, LBA)

you can get starvation for zios to higher numbered LBAs

jbk: not that closely related (mine puts on the brakes for the "scan metadata for block pointers" side of sequential scrub if the system is short on memory) but probably would also help..

which normally isn't a problem

unless the queue for the top level vdev gets too large

which can happen because of the way that limit is enforced

at which point, it's possible the zio starvation ends up triggering the spa deadman

since it looks at the time the zio enters the queue, and not when it's issued to the disk)

so if it's queued for 16 minutes, the instant it finally gets to run, *boom*

jbk: I was seeing pageout deadman triggers, not spa deadman.

yeah, we'd see that from snapshot activity on vmware vms :)

well snapshot and/or zfs diff

when vmware deletes one of it's snapshots, your disk I/O is better measured in IOs per minute instead of per second

what really needs to happen is a backpressure mechanism in the zio pipeline

so if things are backing up, but still progressing, it can throttle stuff at the higher layers in zfs

AFAICT, zfs diff basically just issues all the zios as fast as it can and hopes the disks can keep up

and if they can't, it'll just keep doing it until it exhausts memory

backpressure and some sort of fair-share-scheduler-like thing.

annoyingly, we can get an event from the hypervisor (via vmtoolsd) when things start, but not when it ends

so we can quiesce things to avoid that from happening, but have no way to bring things back

That reminds me, might illumos.org/issues/15884 be related to bug 14526 or am I just triggering on QEMU being in both?

→ BUG 15884: NVMe driver panics on xcp-ng when trying to enable volatile write cache (New)

no

that's omnios specific

but I suspect the PR I have up for it (those drivers aren't upstream yet) might help

the problem is that somehow the VM guest xcp-ng is presenting looks enough like a Hyper-V VM that the Hyper-V drivers are loading and get out of sorts when then things aren't working as expected

oh he has two different problems in that ticket

the one in the description is a different one from what he reported in the comments

wait? I do?

oh right, we updated the ticket after additional debugging.

it was long ago so I've forgotten all the details.

the xcp-ng is related to the hyper-v drivers in omnios

but the nvme on the physical box is arguably two issues

the namespace thing rmustacc mentioned, but also (as he mentioned) we shouldn't panic if the command fails

(a similar thing happened to me with a VMware VM in that their emulated NVMe -- much like the rest of their storage emulation -- didn't seem to like certain things it should, and caused a panic loop

IIRC (mind you, 5 months ago so bad memory) all tests were done on hypervisors running XCP-ng.

supermicro bare-iron had spinning rust and the dell was nVME.

but I won't swear to that now.

entry #4 seems to bear that out: using

boot -B disable-nvme=true,disable-hv_vmbus=true

from the bootloader gets a running installer on both the dell and SuperMicro hypervisors.

anyway, if it's not related it's not related.

I had completely forgotten about the ticket until today's conversation made it bubble back up from the recesses of memory.

I have a PR open with omnios (the people that'll review it were away for a few days) that adds Hyper-V Gen2VM support.. as part of that, the way the vmbus driver is instantiated is smarter

cool

basically instead of always trying and getting fooled (though i thought it looked at the hypervisor signature, which xcp-ng shouldn't be returning 'Microsoft')

If someone can help with xcp-ng, I did a bit more digging there, but wasn't able to really understand exactly why it was happening the way it was.

I'll be happy to test it when you're ready.

rmustacc: it looks like the vmbus driver thought it should attach and try to talk to the hypervisor

Oh, if that's easy, then yeah we can. The one thing I was still trying to figure out is what the actual underlying hypervisor was in xcp-ng.

ISTR it is Xen.

OK. I was having trouble finding the underlying nvme source.

for the vmbus panic, i'd need to dig into it, but my guess is that it's actually the hypercall that's causign things to crash

I wasn't sure if it was leveraging QEMU or something else there. nomad is it doing xpv or not?

xcp-ng.org says XenServer.

it works similar to xen in that you write the address of a physical page to an msr, and the hypervisor maps in a page of code that contains the hw-specific instructions for making the hypercall

rmustacc, I lack that information. Is there an easy way for me to check?

i'm guesssing the msr values are probably different

so then that page it's using is empty

so when it jumps to that page to make the hypercall

sadness ensues

nomad: No worries. If I can figure out an easy way to ask that. I'll reacho ut. I didn't realize you still had access to that.

I'll try to spin something on top of all the other NVMe work I have outstanding.

I have access on the SuperMicro hardware. It's our prod cluster.

i'd offer you an omnios build w/ the xcp-ng fixes (though as you found, you can just disable the driver to work around it), but i wouldn't have a good way to get you the image

I'm wrapping up work in that lab for the week. I'll be able to take a look again next week. Maybe we can find a way to do the exchange, depending on how large the build is.