there is small bug in /opt/onbld/env/omnios-illumos-gate -- export GNUC_ROOT=/opt/gcc-7/ is overwriting export GNUC_ROOT=/usr/gcc/10

tsoome - thanks, I'll open a PR to fix it.

thanks:)

OmniOS r151050 has been released - omnios.org/article/r50

r151038, the old LTS, is now end-of-life. Please upgrade to r151046 or r151050 to stay on a supported track. r151046 is an LTS release.

Does that mean 050 has been released?

Yes, it was released earlier today - omnios.org/article/r50

Thanks

andyf, any chance you could bump illumos.org/issues/16429 for me?

→ BUG 16429: zpool hot spare replacement not grabbing similar/appropriate sized drive (New)

ok, I'm blind. I didn't even see this from fenix: <fenix> OmniOS r151050 has been released - omnios.org/article/r50

Hi nomad

I'm having such. a. day.

I actually have no idea how hot spare selection works, that's interesting.

I'm sure it's nontrivial and impacts on at least three things you'd never expect to be impacted.

but I'd love something as "simple" as "if there's another drive that matches the model of the failing one, use it."

if not, do whatever it currently does.

By 'another drive' I mean hot spare.

andyf: I went digging into the hot spare selection logic a while back. it's pretty simple.

walk through the hot spares in order, try to do a replace. if it fails (perhaps because it's too small), move to the next one

It is - zfs_retire.c is a pretty short fm module

sommerfeld, what order?

If the order were "smallest to largest" that would solve the problem for me :)

whatever order they're listed in the config, looks like.

But, so far, we've had at least two instances where the largest hot spare was picked, even though there was another drive of the same size as the failing one ready to go.

possibly the order they were originally added to the pool, but I'd have to do more digging.

It could also be effectively random, I don't know

One could envision introducing a "sort_spares (spares, nspares, dev_name)" in the middle of replace_with_spare() which ranks them in some order.

yeah, even I, a complete not programmer, can tell that's a short simpleish module.

not that I would be at all comfortable trying to modify it.

nomad - I would be interested to see what order the disks appear in the output of this - mdb -ke '::spa -v'

give me a moment

I presume you also want the output of iostat -En to match types?

just see if the order there is the order you've seen replacements use - as in, is the biggest disk first?

You'll have to cross reference, yes

That can't be the order then. Fair enough. Ordering by size does seem like a reasonable general approach.

"/dev/dsk/c0t5000C500D863CF37d0s0 - ST12000NM002G - 1200G

"/dev/dsk/c3t5000CCA2957A2819d0s0 - WUH721816AL5204 - 1600G

that's *after* we replaced drives, though.

so I wouldn't be surprised if it were in a different order before the most recent failure.

: || lvd@chrufs ~ [508] ; ls -ld /dev/dsk/c0t5000C500D863CF37d0s0 /dev/dsk/c3t5000CCA2957A2819d0s0

lrwxrwxrwx 1 root root 48 Mar 12 17:32 /dev/dsk/c0t5000C500D863CF37d0s0 -> ../../devices/scsi_vhci/disk@g5000c500d863cf37:a

lrwxrwxrwx 1 root root 88 Dec 6 2021 /dev/dsk/c3t5000CCA2957A2819d0s0 -> ../../devices/pci@7d,0/pci8086,2030@0/pci1000,30e0@0/iport@ff/disk@w5000cca2957a2819,0:a

going by that list, the smaller drive (which is listed first) was added more recently.

I found a second host with different sizes of hot spares.

c3t5000CCA2A1CE1FEDd0s0 - WUH721816AL5204 - 16000.90GB - Oct 31 2023

c0t5000C500DAA00C1Bd0s0 - ST12000NM004J - 12000.14GB - Dec 8 17:35

that's the order they're listed in the mdb output.

based on this incredibly small sample sit it might be that the newer drive is always listed first. Maybe. Possibly. Too little data to be sure.

I uploaded a small program to dump the spares the way that the retire module sees them

it's downloads/omnios.org/misc/dump_spares

No size information there, so there would be a bit more work needed to sort them

Wait, I'm about to download and run something from IRC? As root? Can this really be happening? :)

did you mean downloads.omnios.org?

Yes, I did!

heh, thought so.

The output is in the same order as I pasted here.

glad to see I didn't need to sudo to run it.

... on both hosts.

(same order output ... on both hosts)

That's the order that the disks are spared in, anyway. We probably would just need to look up the drive size and try them in ascending order to make things better for you.

That certainly sounds reasonable to me.

In the output from that `dump_spares`, what do the `vdev_stats` lines look like?

The 7th field should be the size

Looks like we even have the information to hand then

sure looks like it