07:18:16 <gitomat> [illumos-gate] 16289 gss_mechs: left shift of negative value -- Toomas Soome <tsoome⊙mc>
07:21:46 <gitomat> [illumos-gate] 16281 libnsl: left shift of negative value -- Toomas Soome <tsoome⊙mc>
07:24:16 <gitomat> [illumos-gate] 16292 gss_mechs: symbol 'asn1buf_insert_octet' is multiply-defined -- Toomas Soome <tsoome⊙mc>
16:29:35 <gitomat> [illumos-gate] 16293 libsldap: passing argument 1 to 'restrict'-qualified parameter aliases with argument 4 -- Toomas Soome <tsoome⊙mc>
16:32:07 <danmcd> @tsoome_ --> I'd prefer richlowe look at 16301 (sgs related).
16:32:19 <tsoome_> I figured so:)
16:33:40 <gitomat> [illumos-gate] 16295 nc: 'restrict'-qualified parameter aliases with argument -- Toomas Soome <tsoome⊙mc>
16:37:29 <gitomat> [illumos-gate] 16296 snoop: 'restrict'-qualified parameter aliases with argument 4 -- Toomas Soome <tsoome⊙mc>
16:41:05 <gitomat> [illumos-gate] 16298 mem: 'restrict'-qualified parameter aliases with argument 4 -- Toomas Soome <tsoome⊙mc>
19:32:36 <KungFuJesus> jbk: sorry, I realize this conversation has like 1.5 day latency, but what am I missing to make this fully light up as failed automatically?
19:45:59 <danmcd> Thanks richlowe
20:01:06 <jclulow> KungFuJesus: The lights are generally managed by FMA, and for the fault light to be on, the system needs to think that the disk is broken and match that up with an FMA case as far as I know.  So we'd need to look at things like: why do you think the disk is faulty, and why does the system apparently not think that; also, what actions have you taken, etc, since determining the disk is faulty.
20:01:30 <richlowe> do you need topo for the chassis too?
20:02:21 <jclulow> Earlier they mentioned they have a SES enclosure, which is self-describing
20:02:36 <jclulow> Also that the lights _are_ able to be turned on, manually, we just turn them off later.
20:03:15 <jclulow> There is often no good way to ask the firmware in these things "is your light on already?" so we tend to resort to flushing out new LED state periodically
20:03:30 <richlowe> oh, I missed that
20:04:07 <jclulow> In the limit if you want to be able to manually override the LED state, we'll need to come up with some way to allow that -- to mediate the desires of both the operator _and_ the fault management stuff, basicaly
20:04:17 <jclulow> Which is totally possible but is not a mechanism that exists today
20:05:03 <richlowe> unrelated: I want to split pkg:/system/kernel/platform, because it's a mess if there's more than one.  Do people prefer pkg:/system/kernel/platform/{i386,aarch64} (cf. SUNWcakr.x), or two distinct manifests that each deliver the same FMRI?
20:05:13 <richlowe> rather than the mess of $(i386_ONLY) that happens otherwise
20:05:49 <richlowe> you can see the mess, in fact, even _without_ 2 platforms.
20:08:35 <richlowe> technically there's a 3rd option: make pkg:/system/kernel/platform i386-only, and other platforms deliver pkg:/system/kernel/platform/<name>, but I hate that one
20:29:40 <gitomat> [illumos-gate] 16301 sgs: 'restrict'-qualified parameter aliases with argument 4 -- Toomas Soome <tsoome⊙mc>
20:31:42 <andyf> I think I prefer a separate FMRI for each. There is some precedent with pkg:/driver/i86pc/platform for the arch to come before 'platform' though
20:32:22 <andyf> I definitely support splitting it up!
21:14:40 <KungFuJesus> jclulow: so, is fmd the thing that's actually turning the LED back off?
21:15:21 <KungFuJesus> so if that's the case, fmd apparently then isn't properly connecting that bay and disk.  What's weird is that I used fmtopo -V to tell me the faulted disk in the pool
21:15:32 <KungFuJesus> fmd reported the disk as faulted and fmtopo knows the bay it's in
21:15:52 <KungFuJesus> one thing that perhaps makes it weird is that it's going through VHCI for multipathing.  Is that perhaps part of the problem?
21:17:32 <KungFuJesus> unfortunately at the moment the disk is already being replaced, so we maybe lost our chance at a guinea pig opportunity
21:18:40 <KungFuJesus> https://pastebin.com/LXsP5cEs
21:20:28 <KungFuJesus> jclulow: the fault event: https://pastebin.com/7dnpKUhi
21:20:54 <KungFuJesus> wtf, it even names the slot in the event, lol
21:21:25 <KungFuJesus> at this point I don't know what else I could give fmd to know any better than it does
21:22:20 <jclulow> KungFuJesus: That's good information!
21:22:33 <KungFuJesus> indeed, why doesn't it know how to act on it?
21:30:33 <jclulow> An excellent question!  This fmd module is supposed to drive the lights based on that sort of thing: https://github.com/illumos/illumos-gate/tree/master/usr/src/cmd/fm/modules/common/disk-lights
21:31:36 <jclulow> So you have a FRU with the right shaped FMRI: hc://:product-id=SMC-SC846P:server-id=:chassis-id=5003048017d2467f:serial=8DK8LE1H:part=HGST-HUH721212AL4200:revision=A3D0/ses-enclosure=0/bay=12/disk=0
21:31:46 <jclulow> i.e., a DISK under a BAY
21:32:09 <jclulow> I think maybe the disconnect here is that we've got a "fault.fs.zfs.vdev.io" fault
21:32:33 <jclulow> But the disk-lights module is subscribing to fault.io.{disk,scsi}.*
21:32:52 <KungFuJesus> so, modify the config file to point to fault.fs.zfs.vdev.io?
21:33:36 <jclulow> Well
21:33:40 <jclulow> I am not sure
21:34:00 <jclulow> But perhaps!
21:34:06 <KungFuJesus> hah, could it hurt?
21:34:49 <jclulow> Well, anything could hurt haha -- but in this case I think the likelihood is low
21:35:01 <jclulow> It doesn't look like the actual C code has anything specific about the fault classes, https://github.com/illumos/illumos-gate/blob/master/usr/src/cmd/fm/modules/common/disk-lights/disk_lights.c#L188-L195
21:35:28 <jclulow> So maybe subscribing it to more fault classes will be enough, if fmd_nvl_fmri_has_fault() manages to link up the vdev fault with the disk FMRI -- but I don't know if it will
21:35:53 <jclulow> It's also possible that the problem is at a much higher level, and that we should have some kind of translation of a vdev-level fault into a disk-level fault
21:36:32 <jclulow> But yeah I think this is where the disconnect is!
21:36:46 <jclulow> I also think this will be reproducible without your specific system
21:37:10 <jclulow> But you could preserve the contents of /var/fm right now to be sure that we have all the data
21:37:31 <jclulow> (it has all the log data and stuff that fmd uses to make these diagnoses)
21:37:47 <KungFuJesus> sure thing - the disk itself has been removed from the pool and is being replaced but I suspect that won't matter if it's historically logged
21:38:06 <KungFuJesus> I'll tar it up, where should I send it to?
21:55:32 <richlowe> jclulow: thanks for making me look at what `fmsim` does :\
21:55:58 <jclulow> KungFuJesus: how big is the tar.gz
21:57:23 <KungFuJesus> ~2MB
21:59:46 <jclulow> KungFuJesus: I think you can just upload it to this issue as an attachment probably: https://www.illumos.org/issues/16353
21:59:47 <fenix> → BUG 16353: disk fault lights should come on for ZFS vdev-level faults (New)
22:03:08 <KungFuJesus> cool, uploaded
22:07:04 <richlowe> andyf: (and others interested), my current preference is to treat platform as a variable, that is, s/platform/i86pc/ s/platform/aarch64/ s/platform/raspberry-pi-4/ in those (and other?) packages.
22:07:55 <richlowe> looking at *platform*.p5m, it looks the most reasonable
22:21:30 <jclulow> richlowe: I would like to be able to install _just_ the oxide parts and not the i86pc parts, if that helps
22:21:48 <jclulow> (oxide being an i86pc-level peer in the taxonomy)
22:31:13 <jclulow> Which I suspect means we should codify some kind of facet structure for pulling those in, as well
22:33:18 <jclulow> Although I suppose it's possible nothing explicitly depends on /system/kernel/platform today so maybe that won't be needed
22:33:40 <richlowe> jclulow: In the model I suggested above, oxide would be system/kernel/oxide
22:33:54 <richlowe> oxide as a moral equivalent to a raspberry-pi-4
22:33:57 <richlowe> except a touch more expensive
22:34:46 <richlowe> so system/kernel/i86pc system/kernel/i86xpv(?) system/kernel/oxide system/kernel/aarch64 system/library/i86pc
22:34:51 <richlowe> etc.
22:35:03 <richlowe> literally replace the word "platform" with the platform
22:35:13 <richlowe> and split the shitty ones as necessary (that's just system/kernel/platform)
22:35:52 <sommerfeld> rhetorical question: what else lives under system/kernel/foo that isn't a platform?
22:37:01 <richlowe> oh, hmmmm, specific kernel features.
22:37:15 <richlowe> system/kernel/cpu-counters for eg
22:37:28 <jclulow> Yeah I think you want /system/kernel/platform/$platform
22:37:49 <richlowe> yeah, and then that really needs to be /system/kernel/$platform/platform, maybe, to follow the majority of the mess we've made
22:38:09 <richlowe> see driver/i86pc/* etc.
22:38:12 <jclulow> Apparently there was once a /system/kernel/platform/netra so this is not really new
22:38:14 <richlowe> I hate naming stuff
22:38:22 <richlowe> though not as much as I hate that package
22:38:31 <richlowe> jclulow: oh, in that case I can go back to how I did it at first!
22:38:36 <richlowe> that's good to know
22:41:44 <jclulow> But yeah we currently, when building an oxide-specific ramdisk, and up just removing a lot of the stuff from that (and similar) packages anyway: https://github.com/oxidecomputer/helios/blob/master/image/templates/gimlet/ramdisk-02-trim.json#L28-L35
22:42:22 <richlowe> yeah, hopefully I can split this in a way that makes that better.
22:42:26 <jclulow> Incremental improvements to making it easier to do that by just installing the bits you need would be awesome
22:44:49 <richlowe> I'll do what I can, I just hate having a manifest that is literally two entirely distinct packages :)