[illumos-gate] 18179 loader: make sure we do create fresh archive -- Toomas Soome <tsoome⊙mc>

/win 1

what is the separation between illumos and distributions? I assumed it was something like the linux kernel where there is no userland whatsoever in the tree but looking in illumos-gate tree i see a libc folder? is that only kernel internal functions?

seti_: its more components than linux but less than FreeBSD so building does require a distribution but bundles many userland components together

I see, interesting!

seti_: among other differences, the stable ABI in illumos is via its dynamically-linked libc, not the syscall trap interface.

which has a lot of advantages despite statically linking the world being the en vogue thing

a lot of it comes down to "What did AT&T include in SVR4?" historically - the desktop & window system aren't there, but most classic Unix utilities are (with the notable exception of the compiler & related tools)

richlowe: would you think that after startup_vm() returns, kas.a_hat should produce reasonable output for ::htables (for comparison)?

I've managed (through a bit of a binary search) to figure out the vaddr that seems to produce a bad htable_t in clear_boot_mappings() that causes x86pte_inval() to get stuck in a look

sommerfeld: thats nice, I think linux is alone in having stable syscalls among all the OSes

i believe freebsd kinda/sorta does, but it's limited to a given version (so you might as well just link against libc or such and save yourself the headache)

tsoome: with the whole dance we do from UEFI->dboot->startup(), do we reuse the page table (and cr3) from UEFI at all, or do we build our own from the mappings passed in from loader?

or i guess do we reuse/manipulate the page tables UEFI has built before we've setup our own

jbk: yeah ive read it as well that they guarantee ABI stability within major versions

jbk dboot does call dboot_startkern.c:startup_kernel() which does build page tables and in dboot_grub.S we load top_page_table into cr3.

hrm...

so we don't re-use the tables UEFI sets up

no

ok

thats because UEFI tables are identity mapping, but we need to map our kernel (see build_page_tables())

basically by the time we call clear_boot_mappings(), it's hitting a va that causes x86pte_inval() to go into an infinite loop

which suggests the entry is bad

so I'm trying to figure out how it could get in that state by then

1G/dump

oops

you can se ethe mapping with -B map_debug=true (enables call to dump_tables())

i do wish microsoft had detailed release notes... i'd love to know what changed in hyper-v between win2022 and win2025

(the former works fine, the latter does not)

andyf: I imagine you're going to add _mdb_ks_ncpu _mdb_ks_ncpu_p2 etc?

or is there more to it than that?

re: #18181 (fenix?)

BUG 18181: mdb modules must stop hard-coding NCPU (In Progress)

↳ illumos.org/issues/18181

(we're in a situation with arm I believe where technically it's _easy_ to out pace NCPU here in the future)

mdb already reads and populates _mdb_ks_ncpu from the target when it opens it so I've added an accessor and changed the modules that define _MACHDEP and make bad decisions to use it and allocate dynamically - or just do more selective reads from the target memory.

[illumos-gate] 18141 fix ksensor dip and unbind lifetimes -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 18142 ksensor_create can create dup minors -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 18143 ksensor should hold the underlying dip from open to close -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 18144 ksensor devfsadm plugin leaves behind stale devices -- Robert Mustacchi <rm⊙fo>

andyf: MACHDEP is exciting for other stuff in the kernel

andyf: I learned with arm that sometimes we use MMU_PAGESHIT when we should use PAGESHIT

did you miss F on purpose?:P

hrmm.. the map seems reasonable

it's dieing when it's trying to clear va=2c04e000 and i can't see anything particularly special about that address...

so it is MA before kernel text (mapping ma=0x2c400000 va=0xfffffffffb800000 pte=0x2c4005e3 l=1 is kernel text segment) and thats why we try to clear that mapping. it does just hung while clearing attempt?

yeah.. it actually gets to the CAS_PTE() in x86pte_inval() and the VM hangs there (I thought it was maybe stuck in a loop, but after stepping after the lock cmpxchgq instruction, it dies

one we do with hyper-v is there is a page in the kernel called `hypercall_page` it serves the same purprose for hyper-v as it does for XEN

early on, we have to tell hyper-v the physical page of that

we use hat_getpfnum(kas.a_hat, hypercall_page)

we can't change it once set...

is it already set when this issue happens?

and the physical page is near(ish) to that.. but not very near (2c442000)

yes

mostly because the module gets loaded at that point and it initializes all of that...

is that vm using dynamic memory?

shouldn't be

just confirmed, it is not using dynamic memory

hrm.. i could see if I could move things so it's loaded after release_bootstrap() is called... the thing I'd be concerned though is then there's no input device until after the vmbus nexus driver is loaded and enumerates its children

(as the vmbus driver requires the hypercall interface)

if it was working ok on top of older windows and fails with newer, that seems to suggest that we are missing something about setting up hyperv infra. I wonder if its possible to inspect the content of the page 2c04e000 - maybe there is any hint... I did browse learn.microsoft.com/en-us/virtualiz…on-windows/tlfs/hypercalls/overview a bit, but I do not think there is something wrong about that one; however, it may be

worth walking through Implementation Recommendations (learn.microsoft.com/en-us/virtualiz…v-on-windows/tlfs/feature-discovery) ... sorry if its too obvious thing to suggest..

yeah, we note all of those, though we don't use hypercalls for all of those

(really, when it was originally ported by delphix, it probably should have been its own platform like x86xpv because it wants to use custom implementations for address space switches, tlb flushes, apic EOI, etc (though it can still emulate like you're on normal hardware)

trying to add a bunch of 'if (hyperv) { do_hypercall() } else { do_hw_thing() }' would get rather ugly

have you tested linux or freebsd vm on windows 2025? both of those are claimed to be supported, maybe logs will provide hits there?

btw, what address that vm gets for framebuffer? framebuffer get from loader ok prompt.

f8000000

* On Gen2 VMs, Hyper-V provides mmio space for framebuffer.

freebsd boots right up

i believe that's all driven from the ACPI info

there isn't a PCI-LPC bridge (often there's no PCI at all)

but if ACPI enumerates the vmbus, we also create an isa device off the root nexus which acts as the parent to the fb (and serial ports)

hm, that too...

(this all works in 2019 or older)

I am browsing fbsd sys/dev/hyperv/vmbus atm:D

it's not the easiest code to follow

it's like taskq-ception :)

hyperv.c is where the hypercall stuff gets setup

yep.

then vmbus issues hypercalls to pass messages to hyperv

and then messages get written back and interrupts issues

github.com/omniosorg/illumos-omnios…r/usr/src/uts/intel/io/hyperv/vmbus is our equivalent code

i do feel like we could improve the vmbus nexus driver further, but since it's completely undocumented (apparently even inside of microsoft), it's a lot of work to try to go backwards

(not that any of that I don't think is relevant to the clear_boot_mappings() hang)