00:47:48 <neirac> danmcd I see you made the last change on ipxe, I was trying to build it just running make on the src directory, how do I generate the smartos.ipxe ? my intention is to see where the pxe boot is failing on my side by compiling with debug flags
01:17:33 <neirac>  I was just missing 2 node packages to begin building 
04:47:06 <danmcd> SOrry I missed this neirac, but you got it, it seems.
11:57:03 <neirac> danmcd yes the setup for the build image took time but things were pretty documented to follow without problemss
13:38:40 <Smithx10> bahamat:  danmcd  looks like 1 of my CNs is reporting a nvme device missing,  but I think it's definitely there.... not sure what happened
13:38:41 <Smithx10> https://gist.github.com/Smithx10/fa946cc1c208af4f362bcd918945df0c
13:38:51 <Smithx10> Trying to figure out how to get it back 
14:02:44 <jfqd1> jperkin: whould it be possible to bring back the jemalloc package in trunk? It is in 2021Q4 but not in trunk.
14:12:54 <jperkin> jfqd1: possibly, could you create a ticket? generally you really want to just use libumem on illumos, you'll get better performance and debugging
14:13:18 <Smithx10> Looks like IPMI shows 10,  but slot1  SN BTLN9050022R3P2BGN isn't showing in the OS
14:19:47 <jfqd1> jperkin: done https://github.com/TritonDataCenter/pkgsrc/issues/347
14:20:16 <jfqd1> I also reopend the following ticket: https://github.com/TritonDataCenter/pkgsrc/issues/340
14:21:23 <jfqd1> And did you saw this comment related to dnsdist? https://github.com/TritonDataCenter/pkgsrc/issues/344#issuecomment-1320928341
14:22:02 <Smithx10> Looks like cfgadm shows 10 nvme connected devices,  but diskinfo and nvmeadm shows 9
14:27:32 <jperkin> I fixed 340, it'll be in the next build
14:33:31 <Smithx10> Ok,  so I've bumped the gist,  looks like cfgadm shows that "Jan  1  1970 nvme/hp      n        /devices/pci@36,0/pci8086,2031@1:Slot238"  doesn't have any links to /dev/rdsk  but shows connected and configured.  Not really sure what the next steps would be 
14:46:01 <jesse_> try unconfiguring and configuring it again?
15:21:44 <danmcd> Smithx10: fmadm faulty shows the one device is faulted.
15:23:15 <danmcd> You *could* try and acquit the fault and see if it reappears.
15:23:19 <danmcd> `fmadm acquit 0180fc24-f033-e01e-e422-c49ee0b99996`
15:23:30 <danmcd> but if it's really faulty it'll come right back as faulty.
15:29:31 <Smithx10> still missing,  https://gist.github.com/Smithx10/fa946cc1c208af4f362bcd918945df0c#file-gistfile1-txt-L14  still not  showed up,  maybe I should bounce it from configured to unconfgiured to configured?
15:30:29 <danmcd> Did it come back as faulty?
15:30:40 <danmcd> (is there a new fmadm faulty report for this device?)
15:30:41 <Smithx10> no
15:31:26 <Smithx10> https://gist.github.com/Smithx10/8fa88958239be4f04f6a171b76e1c29a
15:31:30 <danmcd> Then try the bump.  (You may induce a fault again...)
15:32:14 <Smithx10> cfgadm -v -c unconfigure pcieb4:Slot238 && cfgadm -v -c configure pcieb4:Slot238 ? 
15:32:21 <danmcd> BTW that overlay fault is caused when portolan service  goes down (reboot of HN does this all the time), you can acquit that one too unless your fabrics on that node are faulty.
15:32:56 <danmcd> Smithx10: I think so?  I've not used cfgadm a lot save for USB.  Should work the same however.
15:39:38 <rmustacc> All that will do is attach / detach the device driver.
15:40:02 <rmustacc> Not knowing more about what's going on, that may be sufficient.
15:40:43 <Smithx10> rmustacc:  a nvme device is showing connected and configured in cfgadm https://gist.github.com/Smithx10/8fa88958239be4f04f6a171b76e1c29a#file-gistfile1-txt-L53
15:41:15 <Smithx10> but the disk doesn't show up in nvmeadm list or diskinfo.  Doesn't have a ln in /dev/rdsk either https://gist.github.com/Smithx10/8fa88958239be4f04f6a171b76e1c29a#file-gistfile1-txt-L41
15:41:41 <Smithx10> I noticed this why looking at zpool status and seeing that device as UNAVAIL.  
15:41:44 <rmustacc> Go into mdb, and figure out if it set the n_dead flag.
15:41:59 <rmustacc> (I'm sparsely avaiable so don't have the time right now to guide that)
15:42:11 <Smithx10> Thanks for the pointer
15:43:56 <danmcd> (I can help you find the n_dead.)
15:44:19 <Smithx10> I'm giving the unconfigure configure a shot
15:44:22 <rmustacc> if you don't see a retired flag in prtconf (not -v, iirc) it won't be there.
15:44:24 <rmustacc> Wait.
15:44:29 <rmustacc> If you do that you will destroy the state.
15:44:45 <rmustacc> It may be the path to recovery but it won't let you see that it was marked dead.
15:44:47 <Smithx10> I've resilvered that device with a spare, and detached it 
15:45:03 <Smithx10> ahhhh :( I ran that command alright sorry 
15:45:05 <rmustacc> yes, but unconfigure will delete the kernel nvme_t.
15:45:09 <rmustacc> If it worked.
15:45:10 <rmustacc> OK.
15:45:15 <Smithx10> already*  trying to configur enow
15:45:23 <Smithx10> my bad
15:45:40 <rmustacc> It is what it is.
15:45:44 <rmustacc> Hopefully configure works.
15:45:54 <rmustacc> Anyways, gotta go afk. I'll circle back in a few hours.
15:46:11 <danmcd> If it doesn't it SHOULD send a new fault to FMA, AIUI.
15:46:41 <rmustacc> It's not clear there was ever a fault to fma.
15:46:46 <rmustacc> Did you have things in fmump -e?
15:46:56 <rmustacc> (But I came late so hard to follow)
15:47:06 <rmustacc> The n_dead is a driver thing, not an fm thing.
15:48:36 <Smithx10> danmcd:  https://gist.github.com/Smithx10/eb839a3728c6cf118c29b32c33f5e0ad   faulted
15:50:03 <danmcd> Hmmm.  Now you can likely see if the n_dead is set.
15:50:23 <Smithx10> and if n_dead is set,  this device is probably bad?  time to replace?
15:51:18 <danmcd> To my mind the continual throwing of faults says that, but we can/should check for n_dead anyway.
15:52:26 <danmcd> We need to find its ddi_info first.  In `mdb -k` utter  "::prtconf !grep nvme"
15:54:18 <Smithx10> https://gist.github.com/Smithx10/865aad812e90cfa2f3396bced2b09274
15:57:14 <danmcd> So let's make sure we are reaching things properly.  Pick one pointer, say fffffd65700666f0 and utter this...
15:58:08 <danmcd>      0xfffffd65700666f0::print struct dev_info devi_driver_data |::print nvme_t n_dead
16:00:32 <Smithx10> n_dead = 0 (0)
16:00:49 <Smithx10> > fffffd6c3a780c10::print struct dev_info devi_driver_data |::print nvme_t n_dead;  n_dead = 0x1 (B_TRUE)
16:00:53 <danmcd> Aha!
16:01:07 <danmcd> The other ones should also be 0
16:01:19 <Smithx10> yea
16:01:29 <Smithx10> looks  that way,   i didn't loop over them all but the next 3 were
16:01:47 <danmcd> Good.  So that one is n_dead
16:02:01 <Smithx10> What does n_dead signify ? 
16:02:29 <danmcd> Hang on... source link to block-comment explaining coming...
16:03:21 <rmustacc> In my experience it means the driver found something it didn't ike which isn't reasonable.
16:03:33 <rmustacc> The last time we had this it was a driver bug.
16:03:59 <rmustacc> Check logs to see if it at least said why.
16:04:04 <danmcd> Oh!
16:04:35 <danmcd> rmustacc: is it filed?  Fixed?  (And if fixed, what PI are you running Smithx10?)
16:04:49 <Smithx10> SunOS ac-1f-6b-a5-ab-f2 5.11 joyent_20220505T001410Z i86pc i386 i86pc
16:05:13 <andyf> danmcd - yes, illumos 15202 (fenix)
16:05:24 <andyf> and if fenix does not work here, ah well
16:05:49 <danmcd> jinni illumos 15202
16:05:54 <danmcd> jinni illumos#15202
16:05:55 <jinni> https://www.illumos.org/issues/15202
16:06:18 <danmcd> That is only in just-pushed-out 20221201
16:06:44 <andyf> That one is specifically related to running an `nvmeadm format` though.
16:07:02 <danmcd> But Smithx10 isn't running that, presumably.
16:07:08 <danmcd> This is part of an active pool.
16:07:21 <Smithx10> It was,  its been since removed 
16:08:29 <Smithx10> brb, gotta go to target before the misses ends me. lol 
16:08:39 <danmcd> Ack.
16:08:48 <danmcd> When you return:  fmdump -e -u 184b2bc2-a925-40aa-bfc0-be2133e06f62
16:42:57 <neirac> I debugged pxe today and it stucks on executing the image, after record boot attempt it tries to execute the image, core/image.c 
16:48:27 <neirac> takes a long time before it returns again to the efi shell 
16:52:31 <Smithx10> danmcd:  Dec 02 15:44:57.0724 ereport.io.service.lost
16:56:38 <danmcd> Add -v to it?
16:57:09 <danmcd> or -V gives more.
16:57:18 <danmcd> (it'll be gist-able.)
16:57:37 <Smithx10> https://gist.github.com/Smithx10/2c65e2f9812ca54397152d8673140371
16:58:19 <danmcd> (both -vV ?)
16:58:23 <danmcd> (Or is that the same?)
16:58:32 <Smithx10> same
16:58:41 <danmcd> okay.  THanks.
16:58:47 <Smithx10> little -v just added the ENA
16:58:48 <danmcd> lemme check nvme.c
16:59:59 <danmcd> Lots of places that throw Lost in nvme.c.  Does `grep nvme /var/adm/messages ` have anything unusual?
17:02:46 <Smithx10> https://gist.github.com/Smithx10/b3ff04b7552daca2da118a4e4f4b9630
17:03:02 <Smithx10> nvme4 unable to reset controller
17:05:19 <danmcd> Yeah.  init-time failure... link coming
17:06:07 <danmcd> https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/io/nvme/nvme.c#L3222-L3230
17:06:29 <danmcd> nvme_reset() is failing.
17:08:23 <danmcd> So WHY it's failing is beyond me at the moment.  Also, it's lunchtime here, so AFK for now.
17:54:39 <neirac> this is the last step before it returns to the efi shell https://pasteboard.co/VUzmG0r9XJ4d.png 
18:05:39 <Smithx10> danmcd:  no problem,  that pool is healthy.    We also have spare drives on site if you'd like to swap it out at some point
18:38:01 <danmcd> On the off chance rmustacc might have additional clues to dispense don't do anything just yet.  nvme_reset() failing convinces *me* it needs removal and replacement, but I may be missing something.
18:41:54 <Smithx10> danmcd:  sounds good,  will wait for further instruction.  
18:48:08 <danmcd> The first nvme4 reset in your message isn't all that delayed, but the second one has a 1min delay.
18:48:14 <danmcd> Suggesting it's not listening.
19:03:25 <jesse_> rmustacc, whatever the ixgbe/i40e connection problem was for me (months ago), it was either fixed by newer platform or otherwise transient. Upgraded and rebooted the nodes finally and iperf gives healthy 8.21 Gbits/sec with 30 clients
19:23:03 <danmcd> That's good news!
20:24:41 <jesse_> well, it can still be a lurking bug that'll trigger randomly=(
20:25:13 <jesse_> anyway, will get back if it fails again=)
20:25:45 <jesse_> does anybody know what pkgin -y upgrade is supposed to return on failure?
20:26:50 <jesse_> apparently nothing-to-change is 0, and upgraded-something is 1?
20:26:55 <jesse_> (return value, that is)
20:33:18 <danmcd> Don't know if @jperkin is around but his will be the  canonical answer.
20:37:13 <jesse_> yeah, I was assuming he'd be off this late already
20:48:50 <jesse_> actually, irrelevant
20:49:00 <jesse_> I should be using the ansible pkgin module
20:49:54 <jesse_> should always do a basic sanity check first with old playbooks
21:11:48 <pjustice> FWIW, the ansible pkgin module seems to lack the ability to restrict which version of a package is installed.
21:17:18 <jperkin> jesse_: yeh that seems wrong, feel free to raise an issue for that
21:58:04 <Smithx10> Is there a way to block a usb device from being used at boot?
21:58:26 <Smithx10> probably  the same way we  block them for PCIP?
21:58:56 <Smithx10> https://movementarian.org/blog/posts/2018-10-26-pci-pass-through-support-with-bhyve-and-smartos/
21:59:25 <Smithx10> https://docs.tritondatacenter.com/private-cloud/instances/pass-through-config
22:31:00 <rmustacc> danmcd, Smithx10: Actually there is a case where I've seen a reset fail/time out. i'll need to dig a bit and get back to you and see if it's the same.
22:31:55 * danmcd sets aside some popcorn for whenever the digging is done
22:32:11 <danmcd> Thank you, and no rush.
22:32:31 <rmustacc> There was an issue Jordan debugged around a reset that was failing / going out to lunch and i'd need to figure that out.
22:34:31 <danmcd> Again, take your time.  It's almost (friday) dinnertime here so really no big rush.
22:35:30 <neirac> for ipxe, I cannot go further, the last thing executed is get_efi_mmap and stays there 
22:42:21 <danmcd> neirac: so I'm clear, it's still stuck in ipxe.
22:42:30 <danmcd> Yeah, source says so.
22:55:25 <neirac> danmcd I was trying to check to which statement the pxe stopped and was at executing the image and then calling get_efi_mmap, 2 times is called then the third is stuck 
22:55:54 <neirac> on exit_boot_services
22:56:43 <neirac> that's on multiboot2 so now I'm trying to get the last return code from get_efi_mmap
23:39:07 <jbk> gah.. hit that damn usb panic again... forgot i was running a DEBUG kernel