nice.. hopefully some illumos people can make it

hrm... this is a potentially interesting state on this machine

it appears to have deadlocked while starting up the cpus...

I think while issuing the cross call to start a cpu

also, ::cpustack on a particular CPU is failing with EINVAL which seems odd

The place this ipv6 fastpath/smartos issue has taken me are fascinating

(that's not the problem, just gave me a laugh)

Given that token ring lost, and we have no way to test it even if it didn't, I imagine we could remove any vestiges of support. I don't even know how TR works or if we even have drivers; and linux pulled their support for it in 2012.

the key thing is to make sure it doesn't fall off the end of the wire :)

it being the token

in theory it's supposed to behave better in that you don't really get collisions with token ring, but I believe in the end between switches becoming a thing (vs. hubs) and moar bandwith, those advantages lost out vs. the complexity (plus wasn't it an IBM thing meaning they charged more for it?)

From what wikipedia says it lost to fast/gig ethernet.

I'd say that's squarely in the dead column.

jbk: some aspects of various token rings had some advantage over original CSMA/CD broadcast 10MB/s ethernet. But ethernet switching (now ubiquitous) firmly beats relaying packets through half the nodes on the network on average..

(there were many token rings. Proteon had one. Apollo had another (with 4K+headers with header/data split which made zero-copy page-flipping efficient). IBM's was big for a while. FDDI finally got clobbered by gig ethernet.

ahh fddi.. i remember briefly dealing with that back in the early 00's

sprint was using it for their backup network at the time

so the latest bits on this HPE server is making me wonder if we're deadlocking the APICs somehow...

i still feel like i'm fumbling a bit in the dark, but at least now i think i'm maybe seeing the outlines of some things :)

i'd rather have a patch that fixes this :)

does ::switch in kmdb take the cpuid, or the address of a cpu_t ?

I think the addr. ::help text seems to indicate that.

`::walk cpu` *should* get you appropriate addresses.

Ahhh so does `::cpuinfo` , @jbk

yeah, didn't help.. still get Errno 22 trying to get at those CPUs registers...

Ouch ouch ouch.

I don't suppose use of `::stacks` can tell you if something in a CPU-initialization function?

i've been trying to gather any data that seems potentially interesting in the ticket (not sure if you've seen it), though my unfamiliarity with the interrupt stuff at this low level means I'm not always sure what is or is not interesting

i see one cpu ONPROC stuck in disp_lock_exit

"stuck" how?

disp locks could cause you trouble

it's been sitting there for 1+ hour

in a call to lock_clear_splx

at least doing a ::findstack on the value of that CPU's cpu_thread value

(which I don't know if that's interesting, or if there might be something else more useful)

this is from kmdb

one thing I'm wondering is that writing to an apic register in apic mode is apparently a serializing instruction, in x2apic mode it is explicitly not

and so now are we maybe tacitly been relying on the apic behavior and have just gotten lucky?

i mean, people have been using x2apic mode for a while, so i kinda discount that... but also given this is a dual CPU (64 core/ea) system... maaaaaaaaybe (a very big maybe) we haven't done much with x2apics and multi-cpus and that could explain why it's worked in the past???

jbk: disp locks can really ruin your day here

(and all of this code is unmodified from upstream)

I can't remember how to tell post-morten if you have taken a disp lock regularly, or while high

i guess as a bit of a shot in the dark i can try inserting mfence; lfence before any x2apic msr writes just as a test...

we do it for cross calls (at least atomic_or_ulong() is used for that... i'm assuming that's ok.. but it's also using the address on the stack)

(x2apic_send_ipi)

but not the other writes

richlowe: we've not made any changes here from stock illumos-gate...

and i've never dealt with disp locks before, so that'd be a whole new thing to dig into...

I have seen the ticket updates @jbk.

you would have to build way more of your own hardware than they did to avoid this

Oh damn.

yeah, it's built into the CPU itself IIUC

according to intel's docs, there's only a small number of differences between the apic and x2apic

the big one being using msrs instead of MMIO as well as supporting 32-bit apic ids

instead of 8-bit

(so there's one register access that's a bit different between the two because apic you need to shift & mask to get the id

there's also one bit in a register that's no longer relevant

so it _shouldn't_ work that different from the apic code

(I guess I should say each core gets it's own apic)

each core or each thread? (haven't looked at apics since before hyperthreading...)

hrm.. the intel docs aren't very clear on that.. it could be every thread

but either way, it's not a discrete component from the cpu package

I don't have access the the PCI or PCIe spec -- but in the apic manual, it describes the MSI message as having an 8-bit destination id... for apic ids < 256, these values match

for the apic values > 256, it seems to be truncating the value...(and maybe relying on the IO-APIC???)

this now sounds familiar to me in another context

and I think you might be absolutely screwed

gimme a sec

when I set ddidebug to 0x401, it hangs pretty early starting the CPUs, with the added lfence and sfence instructions to apix_regops.c, it now gets back to where it was at least with mlxcx

it appears to be using a 32-bit 'address' in the msi-x table for the device..

but i feel like i'm maybe missing a step in here somewhere

[illumos-gate] 15972 ZFS getattr could avoid many kidmap_cache_lookup_X calls -- Gordon Ross <gwr⊙rc>

[illumos-gate] 17677 SMB server on ZFS can avoid many kidmap_getXbyY calls -- Gordon Ross <gwr⊙rc>

[illumos-gate] 17678 zfs could avoid kidmap when there are no subgroups -- Matt Barden <mbarden⊙rc>

could it be related to interrupt remapping/virtualization?

do we need to maybe limit interrupts for a device to cores within a single physical CPU

?

as a side note, I'd really like us to prefer using C99-style initialization for the 'ops' structs (where we collect a bunch of function pointers that are meant to be overridden)

would make it a lot easier to see where they're getting assigned (or at least a lot less of a pain)

since we seem to override specific ones..

stackoverflow.com/questions/6702814…s-there-a-need-to-poll-the-chosen-m has text which suggests that there is something in the iommu which will map the 8-bit cpu id in MSI into a 32-bit x2apic processor id.

rzezeski: I give you permission to sunset token ring :P

jclulow: thank you, it's on my list

sommerfeld: that matches what I'm trying to remember too, but the people involved aren't available at the moment

hrm.. the intel vt for directed i/o document does suggest that in x2apic mode, the upper 24 bits of the apic id go in the upper 40 bits of the address register...

worth a shot, though it does lead to 'is the support of this on the device tied to any specific capabilities?'

(i.e. is it possible to have devices that don't support all 64-bits of the msi address register?)