updating neovim to 0.10.3, and working around the crashing problem: omniosorg/omnios-extra #1573

Well... work happened and just getting back to bhyve zones. Trying to install Rocky 9.5 from ISO. I get as far as connecting to VNC and seeing the GRUB boot menu. Then I'm getting a crash and not sure what to do next: "VM-entry failure due to invalid guest state" // Has anyone run into this error with RHEL/EL/etc? Not sure if it's a uefi vs bios boot issue?

found a tutorial for getting Void running, and their settings are very close to mine. So, maybe a hardware issue. I'll try the same config on a different machine tomorrow.

i guess i could try void and follow the tutorial exactly and then (if it works) start bisecting the differences.

Hello! I'm using an OmniOS r151046 machine and I wanted to upgrade the storage, replacing the current NVME drives with a pair of 2TB Samsung 990 pros. The drives work fine under linux and are reported correctly in the BIOS but omnios doesn't see them at all. Neither format, nor iostat -eE show anything and nvmeadm list says there are no controllers. The old NVME disk (SK hynix BC711 HFM512GD3JX013N) is

shown correctly in all three. Is the drive not supported in illumos?

pilonsi - if they are not showing up that seems like a bug.

Is there anything useful in the logs? Try `dmesg | grep nvme` or similar.

also prtconf -vD

I am just checking which OmniOS version got support for newer NVMe standards, but I think it was backported.

In OmniOS r151046 there is no support for NVMe 2.x, so that's a likely reason. If that turns out to be the reason, try uncommenting the 'strict-version=0' line in /kernel/drv/nvme.conf

andyf the new drives are nvme 2.0, not sure about the old ones

Aaah so yeah that must be it

Relaxing that strict version check should be fine, or you could upgrade to r151052 if you don't have a particular desire to stay on the LTS version

May's r151054 will be the next LTS.

I was going to say that, if there may be some stability issues by skipping the version check I may just upgrade. I have no special requirement to be in that version, just chose the LTS when I set up the system last year and so far had no reason to upgrade

Thanks!

You're welcome. The reason I say it should be fine is that relaxing that restriction mostly involved a validation exercise. When the actual change was integrated it was effectively just bumping the maximum allowed version in the driver - illumos/illumos-gate f7ce6a7

Having said that, there have been several further improvements in the nvme driver since r46 so newer should be better!

andyf: I'm on r52 now and it's improved, but the drive is still not fully recognized

iostat -eE shows the following

blkdev3 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0

Model: (null) Revision: (null) Serial No: (null)

Size: 2000,40GB <2000398934016 bytes>

Media Error: 0 Device Not Ready: 0 No Device: 0 Recoverable: 0

Illegal Request: 0 Predictive Failure Analysis: 0

while nvmeadm list

nvmeadm: failed to open 'nvme3': failed to open device path /devices/pci@0,0/pci8086,6c0@1b/pci144d,a801@0:devctl: No such device or address: NVME_ERR_OPEN_DEV (libnvme: 0xd, sys: 6)

Anything in the logs this time? (dmesg)

Seems like it initializes ok

Jan 22 14:20:26 localhost nvme: [ID 259564 kern.info] nvme3: NVMe spec version 2.0

Jan 22 14:20:26 localhost nvme: [ID 259564 kern.info] nvme2: NVMe spec version 1.3

Jan 22 14:20:26 localhost blkdev: [ID 643073 kern.info] NOTICE: blkdev2: dynamic LUN expansion

Jan 22 14:20:26 localhost blkdev: [ID 348765 kern.info] Block device: blkdev@wACE42E00156E7F45,0, blkdev2

Jan 22 14:20:26 localhost genunix: [ID 936769 kern.info] blkdev2 is /pci@0,0/pci8086,6ac@1b,4/pci1c5c,174a@0/blkdev@wACE42E00156E7F45,0

Jan 22 14:20:26 localhost genunix: [ID 408114 kern.info] /pci@0,0/pci8086,6ac@1b,4/pci1c5c,174a@0/blkdev@wACE42E00156E7F45,0 (blkdev2) online

Jan 22 14:20:26 localhost blkdev: [ID 643073 kern.info] NOTICE: blkdev3: dynamic LUN expansion

Jan 22 14:20:26 localhost blkdev: [ID 348765 kern.info] Block device: blkdev@w0025384A41402A94,0, blkdev3

Jan 22 14:20:26 localhost genunix: [ID 936769 kern.info] blkdev3 is /pci@0,0/pci8086,6c0@1b/pci144d,a801@0/blkdev@w0025384A41402A94,0

Jan 22 14:20:26 localhost genunix: [ID 408

Can you try a reconfigure with `devfsadm -vC` ?

actually maybe it's a hardware issue :(

I've rebooted and now it's not detected at all. The drive that was in that slot failed previously as well (wasnt recognized). I thought it was a drive issue since linux could see both samsungs but now with this inconsistencies im starting to think that maybe there is some issue with the connector itself. I'm going to swap drives and check in case it's that and ill report back

so after moving the disk to the slot that hasn't given any issues I still see the same

and running devfsadm -vC removes a lot of device nodes related to disks:

Here is the full output of devfsadm

Can you go into the kernel debugger with `mdb -k` and then enter `::prtconf -d nvme -i 3 | ::devinfo -d | ::print nvme_t` at the prompt, and pastebin that output?

to exit the debugger again you can press ^d or enter $q

or ::quit

Or $%

(no idea why)

$ prefix is originating from adb

if you have the good old "Panic!" book, it is all based on adb:)

after I once did blow up some production DB system with mdb -kw, I did switch to use ::quit for safety:D

I didn't use adb much, I used SCAT a few times on panics. Maybe $% was adb then, but it was definitely carried over to mdb but not documented in the man page or help output.

andyf: I'm going to try that. Meanwhile I swapped the disk in case it was faulty, with the other copy I had. Then it wasn't recognized at all. Put the 1st samsung back in and it wasnt recognized at all either (while before it was). So i loaded linux and it is recognized fine. I have populated the other slot with the second disk and linux does recognize both. So I would discard hardware issues then.

So now, with both disks connected to the board

I see one of them properly recognized, but the other one does not appear at all... No strange things in dmesg either

It's strange because wether it is recognized completely, not at all, or partially is not consistent and I'm not sure what makes it change

~ > pfexec nvmeadm list

nvme4: model: Samsung SSD 990 PRO 2TB, serial: S7PJNJ0XA07900D, FW rev: 4B2QJXD7, NVMe v2.0 nvme4/1 (c3t0025384A41402A94d0): Size = 1.82 TB, Capacity = 1.82 TB, Used = 0 B

This is the output of the debugger command. Although now the disk that does appear is fully recognized. I changed -i to 4 since it is nvme4 now

prtconf -d shows both disks, but one of them appears as (retired)

Could it be fmadm? Just learned about it

But maybe when the old disk failed it disabled that port in the mobo?

Since it failed i see a Notice: One or more i/o devices have been retired message at boot, which still appears now, i thought it was related to the zfs pool the disk was part of

That's interesting. We can clear the retire store and see what happens.

Try moving '/etc/devices/retire_store' to one side and rebooting

Ok so I've moved the retire_store and now I see both of them, but the one in the slot that failed is weird still

~ > pfexec nvmeadm list

nvmeadm: failed to open 'nvme3': failed to open device path /devices/pci@0,0/pci8086,6c0@1b/pci144d,a801@0:devctl: No such device or address: NVME_ERR_OPEN_DEV (libnvme: 0xd, sys: 6)

nvme4: model: Samsung SSD 990 PRO 2TB, serial: S7PJNJ0XA07900D, FW rev: 4B2QJXD7, NVMe v2.0 nvme4/1 (c3t0025384A41402A94d0): Size = 1.82 TB, Capacity = 1.82 TB, Used = 0 B

~ > pastebin.com/PPmg6WZE

This is the mdb command for the nvme3 disk

And this is the fmadm output, the device it listed as failed is the same one that is not being read properly

So maybe there is indeed something wrong with the motherboard?

.join #illumos

/ not .

Got it working :)

It was fmadm

I had renamed the retire_store file to something else, but I noticed it had been recreated, and the drive still appeared as retired

(Renaming it is what i understood by moving it to the side, maybe you meant something else)

Oh, you had to do something like `fmadm repaired` ?

But in any case since Im trying to actually replace the faluted disk i marked it as replaced in fmadm, and both are now detected correclty

Yes!

Maybe there was a race condition between the driver loading and fmadm retiring the device

And sometimes it was retired before the driver loaded (So, not detected)

And sometimes after (So, listed as present, but when trying to access it later it appeared as missing)

Very interesting

Illumos is indeed in another league compared to linux :)

Thank you very much for your help

I'm glad you got it working!