[illumos-gate] 16137 Update tzdata to 2023d -- Robert Mustacchi <rm⊙fo>

introducing qemu to kernel did not eliminate the hung on multi-core VM startup, so I'm rather out of ideas. The only thing I'm sure of is that it does happen while we are attempting to bring non-boot cpus online:)

the only stack I've seen from you, we have an interrupt pinning the pause thread

oh, that I can do.

here it is: paste.ec/paste/D-q7OOCq#Zj1c60Guk19jmOXJvwcf1xeNmee6j9b9tPbUJLxrQAX

huh

the warnings are worrying, too

apparently we don't know how to save fpu state

and we got preempted starting that cpu?

does everything stay like that?

and does this look like the hang in qemu 6 thing I forgot the details of? :)

but it works some of the time?

I don't think the fpu state warnings would break us yet(?), but from what I remember of rmustacc's xsave stuff, it's not good

also the: WARNING: kmem_alloc(): sleeping allocation with size of 0; see kmem_zerosized_log for details

I'd track that one down and figure out why it's happening. at the very least it would get the noise out of the boot log but it might indicate something is mis-sized.

what do we know about the multi-core startup? I assume you're trying 2 cpus - is cpu 0 hanging waiting for cpu 1 to show signs of life?

the one time I had to debug something like this I ended embedding a bunch of "store constant to global variable" through early cpu startup code to see evidence that cpu 1 was doing stuff.

what system is this? the broken cpuid bits seem strange... I thought some things will key off of data returned from them, but not sure if any of that would be relevant to this

sommerfeld: i admit I assumed, but I think that's probably the save area we got told has a size of 0

jbk: could you re-up the err format review? :)

done

yes, it does work sometimes. I actually did try to use some intel cpus there (to eliminate those amd related warnings), but that did not really change the behavior.

sommerfeld actual setup is 4 cores, but 2 will hung just as well. at this point, we actually do have 2 cpus detected and somewhat configured.

well, apparently it is not just completely hung, thread on cpu0 is processing interrupts - it does look like it is rather busy doing so and maybe at some point of time it will get back to onlining cpu1:D

maybe a stuck/not properly acked interrupt?

::interrupts paste.ec/paste/wBcW-IlD#7sR-6+NpmBOMj9wPjHZRBbcGitRYgFB0sWf+u2+56fv

cpu1 has thread fffffe000fca3c20, but it has nothing in stack. reported by ::stacks is 'fffffe000fca3c20 ONPROC <NONE> 2' and '<no consistent stack found>'

either stuck spinning, or not actually running anything yet...

ok, got the RIP of CPU1 from qemu info regs, it is sitting in fffffffffb8a4f68 is pause_cpus+0x128, it seems to be src.illumos.org/source/xref/illumos….c?r=3df2e8b2&mo=54021&fi=1968#1069

pause_cpus+0x128: movzbl 0xfffffffffbc59640(%rdx),%eax <safe_list>

so it's spinning, waiting for the other cpu(s) to pause..

yep

RAX=0000000000000001 so it is != PAUSE_WAIT (2)

[0]> safe_list::print

[ '\001', '\002', '\002', '\002', '\002', '\002', '\002', '\002', '\002', '\002', '\002', '\002', '\002', '\002

', '\002', '\002', '\002', '\002', '\002', '\002', '\002', '\002', '\002', '\002', '\002', '\002', '\002', '\00

2', '\002', '\002', '\002', '\002', ... ]

and cpu0 is not cooperating.

so the cpu0 does not want to move from PAUSE_READY to PAUSE_WAIT

yep

at least the mechanism is understandable now:D

this is what cpu0 is doing at that moment: paste.ec/paste/nSdvfoOH#xjkzzC9TQYx8WV8at885QZnjKAt4JYrGdUTHV-BXEkW

so we got _interrupt while running cpu_pause

interrupted in the: while (cpi->cp_go == 0) ; ?

before it could splhigh() ?

it seems, in sema_v() just a sec, getting paste...

cpu_pause+0x58, but yes, before spllhigh()

seems like race case

in cpu_pause_info we have cp_go = 0x1

so perhaps looping in the interrupt handler on cpu0, not able to reach the test on cp_go ..

"...we don't want to spl because that might block clock interrupts needed to preempt threads on other CPUs." -- but there are no other cpus yet.

so thats the catch there.

possibly looping in tsc_gethrtime() ?

no, I mean, we only do have 2 cpus set up yet, we are about to online cpu1, but we need to quiesce cpu0 first, but there is small window between rising semafore and going splhigh() during which we are actually interrupted ...

um, well, to put it other way around -- if all other cpus are paused, we do not need to keep that window any more