I sent ptribble my quarterly dump of things fixed in our man pages you might also want to address, but if anyone else is interested in cleaning them up, it's at pastebin.com/c0k0CmVY for the next month

(shorter list than normal this quarter)

alanc: the extra fopen flags are for the upcoming POSIX things like O_CLOFORK and related?

yes, though we also have "e" for O_CLOEXEC that I think they don't have yet

Do you have confidence with them that we're not going to end up in another draft posix mess like readdir_r or similar back in the day?

I think gnu libc already implemented them, and breaking the most commonly used implementation at this point would not be good

readdir_r is biting me literally currently

people sure did make some interesting decisions about that

Gotcha, thanks alanc. I've been hesitent to dip into any of the issue 8 stuff for folks until it's out because of that.

rmustacc: if I don't remember, remind me to talk to you about that (readdir_r biting me)

I believe they're approaching final draft now, so it may not be too much longer until the standard is finalized

alanc: do you know if the "large time" sort of stuff standardized?

not that I know of

I haven't seen anything of that when I've taken scans.

I'll follow up with you richlowe.

depressing.

so, how long, pray tell, should it take for this gods forsaken boot archive to update?

Not long

jclulow: so that process hanging for more than 2 minutes should be interpreted as the system failing to power itself down?

The VM pinned its processors for several minutes after I issued "halt" at the console.

now it's bootlooping.

I'm not lying when I say that 9front is more reliable, and that's a joke OS>

jclulow ping?

It's 0056 -- what's up!

small question - previous gerrit version did show integration request flag in table view, now there is status column, but IR is well hidden there - might there be a way to reveal it somehow?:)

Molnija: I don't know how you should interpret it. All I can tell you is you're not seeing a problem everyone else is seeing, so you will need to do some digging in your particular environment to find out what's going wrong.

you've got a point.

boot archive creation/update should not take too long, but it really depends upon how loaded the system is.

default should be cpio format and that should not take too much time...

tsoome: The latest Gerrit stuff has changed the labels/voting around a bit. In particular, there is now a distinction between an integration requirement (CR/IA) and and ancillary tag that's not an approval requirement (IR)

Also they're ditching the ridiculously complex prolog rule system

well, yea, it was convinient way to see which ones I have sent RTI :D

It's a bit in flux. We're not really actually looking at the IR flag very much yet so I haven't spent a lot of time on it, but I will definitely take a look

tsoome: All CPUs were pinned at 100%, at which point I force powered-off the system assuming that it had already done it and was just spinwaiting because it couldn't shut down (a bug I previously had with running off a CD).

Just not tonight I need to sleep

Sleep well Josh.

Molnija: Is this a QEMU 6.X VM

jclulow nice:)

jclulow: It might be. Let me go check.

If it is, perhaps you're seeing illumos.org/issues/14526

→ BUG 14526: illumos guest hangs on reboot under QEMU 6.0.0 (New)

[Devin root]/var/log/runsvdir.lrlog # qemu-system-x86_64 -version

QEMU emulator version 7.1.0

Copyright (c) 2003-2022 Fabrice Bellard and the QEMU Project developers

[Devin root]/var/log/runsvdir.lrlog #

Well we've not really reported it to the QEMU people so I imagine it's still broken

It used to work before 6 and they broke something

Probably we're not doing exactly the right thing either but it used to work

maybe I should try the red headband and see if they have the same problem

uh, if it is/was running in BIOS mode, why it did pick efi_reboot ?

I put some notes in that bug which includes inspecting QEMU

tsoome: efi_reset() is our last ditch attempt to triple fault

Which seems now to fuck up QEMU instead of actually resetting

tsoome: If you're asking about the umbrellix situation, I believe myself to be running in EFI mode.

I do acknowledge the possibility/likelihood of the wrongness of my belief.

I don't think it really matters how you boot the guest to be honest

Pretty sure we're going to try to triple fault either way and it's going to get stuck

But I haven't looked at this in 4+ months

ah

There are notes in the ticket about how to look at the QEMU state

You should do that and see where it's stuck

this reminds me to learn the FP context management API in kernel, to create access to EFI RT ...

Mm

The fix for this might not actually be complicated I just haven't had time and I mostly try not to depend on QEMU -- and when I do

I just poweroff and start the guest again

as I have been

I encourage you to dig in though if you want it fixed

It's all just software, surely it can be made to do the right thing

right, so, found my bootloop

chatspeed.net./~lightning/20221117-lootboop.txt that's impressively fucked. you would think I did it deliberate

that is definitely a new height of wtf

is this what it feels like to break the whole of us-east-1?

that one is suggesting you have no pci module

indeed

what is inside boot_archive?

[This operation is taking longer than usual. I will tell you when I've found out.]

or rather, if it has no pci module, why it is not there...

might it be that BA update failed to read some files from /platform/i86pc (checksum errors?)?

perhaps

anyway, I reran bootadm from the shell out from kayak, let's see if it'll boot ok now

it did it again.

I give up.

I give up on restoring service on this beast.

does it start with boot archive loaded at all?

which bootloader is it btw?

tsoome: it's the BSD loader.

freebsd

ok, so it should not get to boot without boot archive loaded.

if you care to test, get out of menu; enter start and press key to abort autoboot, then use lsmod -vs

start should load kernel and boot archive, lsmod -vs will list what is loaded (verbose + sha1 checksum), then boot -B prom_debug=true will give you verbose early boot. *normally* we should load boot archive with companion checksum file, that should help us to make sure dboot will not continue with checksum mismatch...

if it is so, it must be related to what is inside the boot archive...

tsoome: I cannot interact with the bootloader, because it accepts keyboard input neither on the serial console nor on the visual console.

hmm?

that is interesting...

uefi?

I can't press enter, I can't use the number keys, nothing. I can only wait for it to try to boot.

yes

what hw was it again?

at which layer? it's qemu 7.x on Linux, on a lenovo x3550 m5

you can echo 'boot-args="-B prom_debug=true"' > /boot/conf.d/args

ok

I'm this close to just running the whole thing over with a dump truck

let me get something to eat

you can not input from kbd at all, or is it missing some input?

I can only input from the keyboard or serial if I boot a live CD from the USB CD

mind, this is the system that kpanic'd when I used a SATA CD.

it's cursed.

yea, but loader is not the same as kernel - loader is only using uefi API for communication

that is, it is depending on firmware (ovmf?)

of course, the whole interest about it is what is actually going on and why - so we can understand if we can avoid it happening again:)

I think I drank too much bone hurting juice. All the computers are falling down around me. First this, now my postgres is being weird.

ouch

Any ZFS test folks in the audience... and only off the tops of your heads... is mmp_on_uberblocks a known flaky test?

not on mine, but i haven't used it recently

I'll have to dive into the results a bit more. I just re-ran it by itself and it worked.

what could be happening is a prior test isn't cleaning up properly causing it to fail

I suspect a number of known failures that succeed when you re-run them are this way

but also something i've not had the time to dig into deeply

Yeah... that looks right, plus the "log" output is jumbled a nontrivial amount.

What's odd is that it passed (as part of the big fucking run) last release, but failed now.

Over the past month I've gotten /opt/smartos-test/bin/smartos-test to invoke the ZFS tests if DISKS is set in the environment prior to running.

has there been any changes (new tests) or changes to the runfile between releases (IIRC, tsoome made more of the runfiles common between the various distros at some point, but can't remember when)

yes

14993 and it was followed by 15005

Both of those landed in time for our prior release, which means those two aren't {,finding} the problem.

Re-running it local lends credence to "prior test isn't cleaning up properly" and other transient things.

I'm sure there are tons of those tests

there is also one evasive issue about mkdir -p failing sometimes...

Just checking. I'm rerunning the whole suite now to see if the same ZFS one fails again. I'm guessing it won't fail, but some other one will. (Also, the BHYVE/VMM tests fail sporadically for me but that's because I'm running it on a VMware VM and I don't trust Fusion entirely for in-VMware-guest HVM capacity).

those leftovers and failing cleanups is why I like proper tag support - so we can run single test and verify

hrm.. ofmt_arg_t defines 'ofmt_width', but is in fact never used or initialized (despite being documented in the man page)

So it does.

i'm guessing it's maybe supposed to be the value of of_width for that field?

but i'll at least file a bug

ofmt_index also isn't used.

So at the very least we should update the manual page to reflect the current reality.

the man page at least notes that ofmt_index is unused