00:53:24 <jclulow> Yeah not just yet, unfortunately -- it's not quite finished haha
03:44:17 <jbk> sounds like my tpm driver :)
13:55:48 <KungFuJesus> hmm, boot seems to be hanging on "configuring devices" after a giant leap in updates
13:58:41 <KungFuJesus> anything I can do to diagnose this?  Verbose mode doesn't print any more info after configuring devices
14:00:34 <KungFuJesus> https://imgur.com/a/M4AiMQM
14:18:13 <rmustacc> First thing I usually try to do here is boot into kmdb and then inject an nmi and see if that does it.
14:18:18 <rmustacc> And then I can inspect state.
14:24:25 <KungFuJesus> ok, so I saw the kmdb option on the bootloader
14:24:37 <KungFuJesus> I can boot to that, then can you give me the command?
14:24:53 <rmustacc> Command for what?
14:24:57 <rmustacc> What to look at?
14:25:10 <KungFuJesus> yeah, what to do in kmdb
14:25:30 <rmustacc> I can give some high level bits, I am not really around IRC much today.
14:26:36 <KungFuJesus> I assume you mean the option from the bootloader option that says: "kmdb mode"?
14:26:51 <KungFuJesus> err s/option/g
14:27:29 <KungFuJesus> fwiw I can boot into the ancient boot environment still because it doesn't attempt a device reconfigure
14:32:18 <rmustacc> There is the kmdb mode that says something like 'At Boot'?
14:32:19 <rmustacc> IIRC.
14:36:32 <KungFuJesus> yep, found "On Boot"
14:36:52 <KungFuJesus> anything else I should set while in the boot loader?
14:38:16 <rmustacc> Nothing really comes to mind given your comments about verbose mode.
14:38:31 <KungFuJesus> ok, I'll boot and let you know when I'm at the prompt.  Sounds like it should be immediately
14:38:55 <KungFuJesus> yep, I'm there
14:38:56 <rmustacc> Yeah, when you do just hit ':c'.
14:39:07 <rmustacc> Then when it hands, try to inject an NMI and see if that drops you back into kmdb.
14:39:26 <rmustacc> That'd be via something like ipmitool chassis power diag IIRC.
14:40:26 <KungFuJesus> Hmm, not sure I have a machine with ipmi tool, let me see if I can find it from the web UI
14:42:29 <rmustacc> If you do get in, the things I'd look at are ::cpuinfo, $C, ::stacks, etc.
14:43:12 <rmustacc> If that doesn't work, there are much more manual ways to go through it all using moddebug and breakpoints.
14:44:08 <KungFuJesus> did this: 
14:44:08 <KungFuJesus> Adam Stylinski 10:43 AM
14:44:08 <KungFuJesus> [astylinski@fedoravm ~]$ ipmitool -H 192.168.8.45 -U ADMIN -P ADMIN chassis power diag
14:44:11 <KungFuJesus> Chassis Power Control: Diag
14:44:16 <KungFuJesus> didn't drop me to mdb, though
14:45:20 <KungFuJesus> there was another option for mdb for "on NMI" rather than on boot
14:45:23 <KungFuJesus> did I need that instead?
14:45:47 <KungFuJesus> I'll try that
14:54:29 <KungFuJesus> looks like IPMI tool is sending it but this supermicro BMC is deaf to it for whatever reason
14:55:20 <KungFuJesus> it is _fairly_ old.  I'll drop back to the mdb prompt at boot for the long route
14:55:22 <rmustacc> No, you wouldn't need to switch that. There are reasons why it may not work.
14:58:29 <KungFuJesus> alright, back at the prompt
15:01:23 <KungFuJesus> there's no way to set a magic key sequence?  Can I get a stop-a, heh?
15:04:59 <Agnar> on x86 iirc there was also F1-a on the physical keyboard, but I never used it. Also Pause-A should work iirc
15:05:55 <KungFuJesus> I'll give it a try
15:06:52 <Agnar> but your BMC should be able to send a NMI to the system, that should do the trick
15:07:29 <Agnar> if it doesn't work, it's probably stuck somewhere weird. I have a Dell Laptop that I can't break into mdb also
15:09:16 <KungFuJesus> yeah those things aren't working either through the virtual keyboard or my physical one (I'm stuck dealing with iKVM for this thing)
15:09:50 <Agnar> is there a way to send a NMI through the BMCs cli?
15:10:21 <KungFuJesus> supermicro claims the chassis power diag should do it
15:10:33 <KungFuJesus> though, I'm uncertain that works for everything supermicro
15:10:49 <Agnar> hmm.
15:14:46 <KungFuJesus> hmm, there may be an option to enable it in the bios? Let me consult that
15:18:56 <rmustacc> If an NMI doesn't work the alternate break is not going to get you out of it.
15:19:18 <rmustacc> You can verify NMI functionality by booting to your old BE and confirming.
15:29:23 <KungFuJesus> I have the displeasure of a BIOS that has a bug that post 2020, will not let you enter it.  So to get into it, you have to set the system clock from the OS to something prior to december 2020 and then reboot
15:29:59 <Agnar> KungFuJesus: oh Jesus, you really have a hard time:/
15:30:13 <KungFuJesus> Yeah, thanks AMI
15:31:07 <KungFuJesus> I'm working with one of the IT people on prem, he may be able to update the BIOS in order to see if NMI is disabled (which from what I can gather, probably is, due to supposedly an issue in the FBSD boot process that hangs with an NMI watchdog).  Which you know, is Illumos' loader too, so, that could fun
15:31:23 <rmustacc> Those are different things.
15:31:30 <rmustacc> If you want to test the nmi, just go to your old BE and inject it.
15:31:47 <rmustacc> We can definitely hang in ways that an NMI won't matter.
15:31:51 <KungFuJesus> via ipmitool once booted?  I can give it a shot
15:31:55 <rmustacc> Yes.
15:40:38 <KungFuJesus> yeah, ipmitool NMI injection doesn't do anything in the old BE
15:41:24 <KungFuJesus> it really does seem like the "configuring devices" bit is the reason it hangs.  I also have a disk in there that it seems to complain about with a missing GUID
15:43:35 <KungFuJesus> Should I try "reconfigure" from the boot loader for the old boot environment, or is that a recipe for not being able to boot into that?
15:46:35 <KungFuJesus> ok, this is interesting.  I was able to send it from the localhost
15:47:22 <KungFuJesus> it's doing something but I'm not dropping to the mdb prompt, I just keep getting a bunch of diagnostic messages about log info 0x31111000 from a disk on the SAS controller
17:45:44 <KungFuJesus> hmm, got it booting.  Updated the 9211 firmware to something not ancient and relaunched the update process into a new boot environment
17:48:44 <jbk> LSI?
18:02:24 <sjorge> jclulow: i messed with it briefly in a vm, it at least seemed to probe one of my current driver not supported devices correctly, didn't actrually try and run the software that talks to it though
18:02:45 <sjorge> so at a quick glance it seems better than the current
18:04:23 <KungFuJesus> jbk: yes, it was in P15, I updated to P20
18:04:42 <KungFuJesus> hard to say _exactly_ if that's what fixed it, but there was the perfect storm of a disk on the controller there that didn't present a WWN
18:04:58 <jbk> i swear there's a bug in the driver where on resets/timeouts its losing I/Os somehow, but haven't had time to dig into it
18:05:53 <KungFuJesus> and another thing that sorted of hinted at it is when I sent an NMI locally (remote doesn't work for some reason?) it just spammed the console about log info 0x31111000 on one of the targets on one of the SAS controllers
18:06:22 <KungFuJesus> that disk should probably be removed to eliminate future headaches, I don't know why it's such a problem child
18:07:16 <KungFuJesus> It'd have likely booted if I removed that disk, is my guess.  I also removed the nvidia driver so that it'd stop complaining about it at boot, but I suspect that was a red herring and harmless