[illumos-gate] 16289 gss_mechs: left shift of negative value -- Toomas Soome <tsoome⊙mc>

[illumos-gate] 16281 libnsl: left shift of negative value -- Toomas Soome <tsoome⊙mc>

[illumos-gate] 16292 gss_mechs: symbol 'asn1buf_insert_octet' is multiply-defined -- Toomas Soome <tsoome⊙mc>

[illumos-gate] 16293 libsldap: passing argument 1 to 'restrict'-qualified parameter aliases with argument 4 -- Toomas Soome <tsoome⊙mc>

@tsoome_ --> I'd prefer richlowe look at 16301 (sgs related).

I figured so:)

[illumos-gate] 16295 nc: 'restrict'-qualified parameter aliases with argument -- Toomas Soome <tsoome⊙mc>

[illumos-gate] 16296 snoop: 'restrict'-qualified parameter aliases with argument 4 -- Toomas Soome <tsoome⊙mc>

[illumos-gate] 16298 mem: 'restrict'-qualified parameter aliases with argument 4 -- Toomas Soome <tsoome⊙mc>

jbk: sorry, I realize this conversation has like 1.5 day latency, but what am I missing to make this fully light up as failed automatically?

Thanks richlowe

KungFuJesus: The lights are generally managed by FMA, and for the fault light to be on, the system needs to think that the disk is broken and match that up with an FMA case as far as I know. So we'd need to look at things like: why do you think the disk is faulty, and why does the system apparently not think that; also, what actions have you taken, etc, since determining the disk is faulty.

do you need topo for the chassis too?

Earlier they mentioned they have a SES enclosure, which is self-describing

Also that the lights _are_ able to be turned on, manually, we just turn them off later.

There is often no good way to ask the firmware in these things "is your light on already?" so we tend to resort to flushing out new LED state periodically

oh, I missed that

In the limit if you want to be able to manually override the LED state, we'll need to come up with some way to allow that -- to mediate the desires of both the operator _and_ the fault management stuff, basicaly

Which is totally possible but is not a mechanism that exists today

unrelated: I want to split pkg:/system/kernel/platform, because it's a mess if there's more than one. Do people prefer pkg:/system/kernel/platform/{i386,aarch64} (cf. SUNWcakr.x), or two distinct manifests that each deliver the same FMRI?

rather than the mess of $(i386_ONLY) that happens otherwise

you can see the mess, in fact, even _without_ 2 platforms.

technically there's a 3rd option: make pkg:/system/kernel/platform i386-only, and other platforms deliver pkg:/system/kernel/platform/<name>, but I hate that one

[illumos-gate] 16301 sgs: 'restrict'-qualified parameter aliases with argument 4 -- Toomas Soome <tsoome⊙mc>

I think I prefer a separate FMRI for each. There is some precedent with pkg:/driver/i86pc/platform for the arch to come before 'platform' though

I definitely support splitting it up!

jclulow: so, is fmd the thing that's actually turning the LED back off?

so if that's the case, fmd apparently then isn't properly connecting that bay and disk. What's weird is that I used fmtopo -V to tell me the faulted disk in the pool

fmd reported the disk as faulted and fmtopo knows the bay it's in

one thing that perhaps makes it weird is that it's going through VHCI for multipathing. Is that perhaps part of the problem?

unfortunately at the moment the disk is already being replaced, so we maybe lost our chance at a guinea pig opportunity

jclulow: the fault event: pastebin.com/7dnpKUhi

wtf, it even names the slot in the event, lol

at this point I don't know what else I could give fmd to know any better than it does

KungFuJesus: That's good information!

indeed, why doesn't it know how to act on it?

An excellent question! This fmd module is supposed to drive the lights based on that sort of thing: github.com/illumos/illumos-gate/tre…c/cmd/fm/modules/common/disk-lights

So you have a FRU with the right shaped FMRI: hc://:product-id=SMC-SC846P:server-id=:chassis-id=5003048017d2467f:serial=8DK8LE1H:part=HGST-HUH721212AL4200:revision=A3D0/ses-enclosure=0/bay=12/disk=0

i.e., a DISK under a BAY

I think maybe the disconnect here is that we've got a "fault.fs.zfs.vdev.io" fault

But the disk-lights module is subscribing to fault.io.{disk,scsi}.*

so, modify the config file to point to fault.fs.zfs.vdev.io?

Well

I am not sure

But perhaps!

hah, could it hurt?

Well, anything could hurt haha -- but in this case I think the likelihood is low

It doesn't look like the actual C code has anything specific about the fault classes, github.com/illumos/illumos-gate/blo…disk-lights/disk_lights.c#L188-L195

So maybe subscribing it to more fault classes will be enough, if fmd_nvl_fmri_has_fault() manages to link up the vdev fault with the disk FMRI -- but I don't know if it will

It's also possible that the problem is at a much higher level, and that we should have some kind of translation of a vdev-level fault into a disk-level fault

But yeah I think this is where the disconnect is!

I also think this will be reproducible without your specific system

But you could preserve the contents of /var/fm right now to be sure that we have all the data

(it has all the log data and stuff that fmd uses to make these diagnoses)

sure thing - the disk itself has been removed from the pool and is being replaced but I suspect that won't matter if it's historically logged

I'll tar it up, where should I send it to?

jclulow: thanks for making me look at what `fmsim` does :\

KungFuJesus: how big is the tar.gz

~2MB

KungFuJesus: I think you can just upload it to this issue as an attachment probably: illumos.org/issues/16353

→ BUG 16353: disk fault lights should come on for ZFS vdev-level faults (New)

cool, uploaded

andyf: (and others interested), my current preference is to treat platform as a variable, that is, s/platform/i86pc/ s/platform/aarch64/ s/platform/raspberry-pi-4/ in those (and other?) packages.

looking at *platform*.p5m, it looks the most reasonable

richlowe: I would like to be able to install _just_ the oxide parts and not the i86pc parts, if that helps

(oxide being an i86pc-level peer in the taxonomy)

Which I suspect means we should codify some kind of facet structure for pulling those in, as well

Although I suppose it's possible nothing explicitly depends on /system/kernel/platform today so maybe that won't be needed

jclulow: In the model I suggested above, oxide would be system/kernel/oxide

oxide as a moral equivalent to a raspberry-pi-4

except a touch more expensive

so system/kernel/i86pc system/kernel/i86xpv(?) system/kernel/oxide system/kernel/aarch64 system/library/i86pc

etc.

literally replace the word "platform" with the platform

and split the shitty ones as necessary (that's just system/kernel/platform)

rhetorical question: what else lives under system/kernel/foo that isn't a platform?

oh, hmmmm, specific kernel features.

system/kernel/cpu-counters for eg

Yeah I think you want /system/kernel/platform/$platform

yeah, and then that really needs to be /system/kernel/$platform/platform, maybe, to follow the majority of the mess we've made

see driver/i86pc/* etc.

Apparently there was once a /system/kernel/platform/netra so this is not really new

I hate naming stuff

though not as much as I hate that package

jclulow: oh, in that case I can go back to how I did it at first!

that's good to know

But yeah we currently, when building an oxide-specific ramdisk, and up just removing a lot of the stuff from that (and similar) packages anyway: github.com/oxidecomputer/helios/blo…gimlet/ramdisk-02-trim.json#L28-L35

yeah, hopefully I can split this in a way that makes that better.

Incremental improvements to making it easier to do that by just installing the bits you need would be awesome

I'll do what I can, I just hate having a manifest that is literally two entirely distinct packages :)