00:14:59 I sent ptribble my quarterly dump of things fixed in our man pages you might also want to address, but if anyone else is interested in cleaning them up, it's at https://pastebin.com/c0k0CmVY for the next month 00:15:21 (shorter list than normal this quarter) 00:20:20 alanc: the extra fopen flags are for the upcoming POSIX things like O_CLOFORK and related? 00:23:53 yes, though we also have "e" for O_CLOEXEC that I think they don't have yet 00:25:34 Do you have confidence with them that we're not going to end up in another draft posix mess like readdir_r or similar back in the day? 00:26:53 I think gnu libc already implemented them, and breaking the most commonly used implementation at this point would not be good 00:27:47 readdir_r is biting me literally currently 00:28:32 people sure did make some interesting decisions about that 00:28:36 Gotcha, thanks alanc. I've been hesitent to dip into any of the issue 8 stuff for folks until it's out because of that. 00:28:49 rmustacc: if I don't remember, remind me to talk to you about that (readdir_r biting me) 00:29:15 I believe they're approaching final draft now, so it may not be too much longer until the standard is finalized 00:29:40 alanc: do you know if the "large time" sort of stuff standardized? 00:29:52 not that I know of 00:30:25 I haven't seen anything of that when I've taken scans. 00:32:45 I'll follow up with you richlowe. 04:37:52 depressing. 04:42:44 so, how long, pray tell, should it take for this gods forsaken boot archive to update? 05:27:07 Not long 06:33:18 jclulow: so that process hanging for more than 2 minutes should be interpreted as the system failing to power itself down? 06:34:59 The VM pinned its processors for several minutes after I issued "halt" at the console. 08:11:19 now it's bootlooping. 08:11:36 I'm not lying when I say that 9front is more reliable, and that's a joke OS> 08:55:31 jclulow ping? 08:56:06 It's 0056 -- what's up! 08:57:28 small question - previous gerrit version did show integration request flag in table view, now there is status column, but IR is well hidden there - might there be a way to reveal it somehow?:) 08:57:55 Molnija: I don't know how you should interpret it. All I can tell you is you're not seeing a problem everyone else is seeing, so you will need to do some digging in your particular environment to find out what's going wrong. 08:58:38 you've got a point. 08:58:44 boot archive creation/update should not take too long, but it really depends upon how loaded the system is. 08:59:21 default should be cpio format and that should not take too much time... 08:59:40 tsoome: The latest Gerrit stuff has changed the labels/voting around a bit. In particular, there is now a distinction between an integration requirement (CR/IA) and and ancillary tag that's not an approval requirement (IR) 08:59:57 Also they're ditching the ridiculously complex prolog rule system 09:00:31 well, yea, it was convinient way to see which ones I have sent RTI :D 09:00:39 It's a bit in flux. We're not really actually looking at the IR flag very much yet so I haven't spent a lot of time on it, but I will definitely take a look 09:00:40 tsoome: All CPUs were pinned at 100%, at which point I force powered-off the system assuming that it had already done it and was just spinwaiting because it couldn't shut down (a bug I previously had with running off a CD). 09:00:49 Just not tonight I need to sleep 09:01:02 Sleep well Josh. 09:01:14 Molnija: Is this a QEMU 6.X VM 09:01:23 jclulow nice:) 09:01:35 jclulow: It might be. Let me go check. 09:01:55 If it is, perhaps you're seeing https://www.illumos.org/issues/14526 09:01:56 → BUG 14526: illumos guest hangs on reboot under QEMU 6.0.0 (New) 09:02:36 [Devin root]/var/log/runsvdir.lrlog # qemu-system-x86_64 -version 09:02:38 QEMU emulator version 7.1.0 09:02:40 Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers 09:02:42 [Devin root]/var/log/runsvdir.lrlog # 09:02:58 Well we've not really reported it to the QEMU people so I imagine it's still broken 09:03:31 It used to work before 6 and they broke something 09:03:46 Probably we're not doing exactly the right thing either but it used to work 09:04:09 maybe I should try the red headband and see if they have the same problem 09:04:11 uh, if it is/was running in BIOS mode, why it did pick efi_reboot ? 09:04:12 I put some notes in that bug which includes inspecting QEMU 09:04:39 tsoome: efi_reset() is our last ditch attempt to triple fault 09:04:52 Which seems now to fuck up QEMU instead of actually resetting 09:04:52 tsoome: If you're asking about the umbrellix situation, I believe myself to be running in EFI mode. 09:05:32 I do acknowledge the possibility/likelihood of the wrongness of my belief. 09:05:58 I don't think it really matters how you boot the guest to be honest 09:06:10 Pretty sure we're going to try to triple fault either way and it's going to get stuck 09:06:27 But I haven't looked at this in 4+ months 09:06:42 ah 09:06:46 There are notes in the ticket about how to look at the QEMU state 09:06:54 You should do that and see where it's stuck 09:07:49 this reminds me to learn the FP context management API in kernel, to create access to EFI RT ... 09:08:43 Mm 09:08:53 The fix for this might not actually be complicated I just haven't had time and I mostly try not to depend on QEMU -- and when I do 09:09:02 I just poweroff and start the guest again 09:09:28 as I have been 09:09:33 I encourage you to dig in though if you want it fixed 09:09:57 It's all just software, surely it can be made to do the right thing 12:13:18 right, so, found my bootloop 12:14:21 https://chatspeed.net./~lightning/20221117-lootboop.txt that's impressively fucked. you would think I did it deliberate 12:15:13 that is definitely a new height of wtf 12:15:40 is this what it feels like to break the whole of us-east-1? 12:20:45 that one is suggesting you have no pci module 12:21:07 indeed 12:21:28 what is inside boot_archive? 12:22:13 [This operation is taking longer than usual. I will tell you when I've found out.] 12:23:05 or rather, if it has no pci module, why it is not there... 12:24:18 might it be that BA update failed to read some files from /platform/i86pc (checksum errors?)? 12:28:57 perhaps 12:31:10 anyway, I reran bootadm from the shell out from kayak, let's see if it'll boot ok now 12:32:56 it did it again. 12:33:34 I give up. 12:33:41 I give up on restoring service on this beast. 12:33:58 * Molnija goes and cries, or something 12:34:26 does it start with boot archive loaded at all? 12:35:04 which bootloader is it btw? 12:45:29 tsoome: it's the BSD loader. 12:45:40 freebsd 12:46:00 ok, so it should not get to boot without boot archive loaded. 12:46:43 if you care to test, get out of menu; enter start and press key to abort autoboot, then use lsmod -vs 12:49:24 start should load kernel and boot archive, lsmod -vs will list what is loaded (verbose + sha1 checksum), then boot -B prom_debug=true will give you verbose early boot. *normally* we should load boot archive with companion checksum file, that should help us to make sure dboot will not continue with checksum mismatch... 12:49:55 if it is so, it must be related to what is inside the boot archive... 12:55:33 tsoome: I cannot interact with the bootloader, because it accepts keyboard input neither on the serial console nor on the visual console. 12:56:05 hmm? 12:56:27 that is interesting... 12:56:30 uefi? 12:56:30 I can't press enter, I can't use the number keys, nothing. I can only wait for it to try to boot. 12:56:32 yes 12:57:30 what hw was it again? 12:58:44 at which layer? it's qemu 7.x on Linux, on a lenovo x3550 m5 12:59:13 you can echo 'boot-args="-B prom_debug=true"' > /boot/conf.d/args 12:59:25 ok 12:59:43 I'm this close to just running the whole thing over with a dump truck 12:59:55 let me get something to eat 13:00:58 you can not input from kbd at all, or is it missing some input? 13:01:44 I can only input from the keyboard or serial if I boot a live CD from the USB CD 13:01:55 mind, this is the system that kpanic'd when I used a SATA CD. 13:02:13 it's cursed. 13:02:43 yea, but loader is not the same as kernel - loader is only using uefi API for communication 13:03:18 that is, it is depending on firmware (ovmf?) 13:10:44 of course, the whole interest about it is what is actually going on and why - so we can understand if we can avoid it happening again:) 13:11:48 I think I drank too much bone hurting juice. All the computers are falling down around me. First this, now my postgres is being weird. 13:12:01 ouch 14:17:20 Any ZFS test folks in the audience... and only off the tops of your heads... is mmp_on_uberblocks a known flaky test? 14:23:35 not on mine, but i haven't used it recently 14:25:11 I'll have to dive into the results a bit more. I just re-ran it by itself and it worked. 14:25:49 what could be happening is a prior test isn't cleaning up properly causing it to fail 14:26:05 I suspect a number of known failures that succeed when you re-run them are this way 14:26:13 but also something i've not had the time to dig into deeply 14:35:04 Yeah... that looks right, plus the "log" output is jumbled a nontrivial amount. 14:35:22 What's odd is that it passed (as part of the big fucking run) last release, but failed now. 14:36:43 Over the past month I've gotten /opt/smartos-test/bin/smartos-test to invoke the ZFS tests if DISKS is set in the environment prior to running. 14:37:29 has there been any changes (new tests) or changes to the runfile between releases (IIRC, tsoome made more of the runfiles common between the various distros at some point, but can't remember when) 15:08:55 yes 15:09:59 14993 and it was followed by 15005 15:23:46 Both of those landed in time for our prior release, which means those two aren't {,finding} the problem. 15:24:24 Re-running it local lends credence to "prior test isn't cleaning up properly" and other transient things. 15:25:04 I'm sure there are tons of those tests 15:26:20 there is also one evasive issue about mkdir -p failing sometimes... 15:27:07 Just checking. I'm rerunning the whole suite now to see if the same ZFS one fails again. I'm guessing it won't fail, but some other one will. (Also, the BHYVE/VMM tests fail sporadically for me but that's because I'm running it on a VMware VM and I don't trust Fusion entirely for in-VMware-guest HVM capacity). 15:29:57 those leftovers and failing cleanups is why I like proper tag support - so we can run single test and verify 21:48:07 hrm.. ofmt_arg_t defines 'ofmt_width', but is in fact never used or initialized (despite being documented in the man page) 21:48:30 So it does. 21:49:19 i'm guessing it's maybe supposed to be the value of of_width for that field? 21:49:28 but i'll at least file a bug 21:49:39 ofmt_index also isn't used. 21:50:00 So at the very least we should update the manual page to reflect the current reality. 21:50:16 the man page at least notes that ofmt_index is unused