Is anyone running code on a Xeon Scalable Gen4 ("Sapphire Rapids")? Bonus points if it's

"GenuineIntel 806F8 family 6 model 143 step 8"

MORE bonus points if it's a Xeon Gold 5418Y

I'm seeing some strange behavior, and I want to know if anyone else is seeing anything strange or not before I completely rabbit-hole myself.

I'm not, but you should describe the behaviour

we might have some in our lab, but I'll need to check later

A specific version of one of our node.js binaries craps out on this specific server. Don't wanna go blame bad-HW just yet.

unfortunately, we don't

My BIG and hopefully overblown worry is that Sapphire Rapids has something CPU-indirect-jump-protection wise that's kneecapping us, and I haven't accounted for it. Both pre-and-post latest-ucode misbehave that way.

It's gonna be eating my time all today and into next week if I don't figure it out quickly.

I'm insta-core-dumping with SEGV, and it's likely in an indirect jump inside elf_rtbndr+0x2c

that is us binding the symbols of dependencies

so the indirect jump is expected, where does it go?

another place that might be sad is the FPU saving stuff

This binary works on other non-Sapphire-Rapids machines.

the FPU saving stuff is dynamic

it figures out the size of AVX* stuff at runtime, to avoid the massive hit the maximum would be

Also it's possible we need to compile it with better options. I'm less worried now about illumos-kernel-interventions than I am about out stuff. And yes, that could be a concern re: AVX*.

yeah, I would bet this is the FPU save

the first call *(%r10)

so the difference in CPU model is going to be the info provided for _plt_save_size et al.

danmcd: if mdb is annoying you with `::dis`, you need to specify the object (`ld.so.1`elf_rtbndr::dis`), even though the tab complete works without

Hang on. I haven't gotten that far yet.

Trying to inventory what's going on, and what is common vs. not-common.

And I meant "our stuff" instead of "out stuff" above.

<sigh> THe Sapphire Rapids box kernel panicked with the very current `master` SmartOS build.

if it is weird about the FPU save, it seems reasonable to suspect kfpu will be broken too

I can't remember if we can/how to disable that

Lemme see WHAT the panic is first. Lucky me it's small, in spite of the machine's 512GiB memory.

Ooooh, "bad mutex".

Maybe it IS a hardware problem?

Or something unrelated to my node SEGVs? It's a deep stack starting with hubd_root_hub_cleanup_thread() and going deep from there.

HAH! Use-after-free.

It's fenix illumos#17443

BUG 17443: xhci panics when unable to stop device (New)

↳ illumos.org/issues/17443

Hey @jbk --> want a kernel dump from a Sapphire Rapids machine (that also happens to be a Dell)?

I can even boot a DEBUG kernel on it if you'd like. :)

@richlowe you nailed it:

> <r10=P

ld.so.1`_plt_fp_save

>

> <r10=J

fffff9ffef3fed90

> fffff9ffef3fed90/P

ld.so.1`_plt_fp_save:

ld.so.1`_plt_fp_save: ld.so.1`_elf_rtbndr_fp_xsave

>

Am I gonna have to introduce a new AT_386_FPINFO_ value for Sapphire Rapids ?

related, I think: fenix illumos#9595 illumos#9596

FEATURE 9595: rtld should conditionally save AVX-512 state (Closed)

↳ illumos.org/issues/9595

BUG 9596: Initial xsave xstate_bv should not include all features (Closed)

↳ illumos.org/issues/9596

Those two bugs are supposed to make things easier for future things, if I read it carefully.

danmcd: tur s out we do have one if you still need some testing