aha! the console font is at that address (not sure yet what that means)

[illumos-gate] 18119 nvme_print_identify_ctrl_list() needs to verify id count -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 18121 Plumb Micron 7600 nvme log pages -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 18122 add support for OCP Hardware Component log -- Robert Mustacchi <rm⊙fo>

tsoome: is there an explicit point where we stop using the boot fb?

i couldn't find it looking yesterday (but it was late)

jbk basically with gfxp_map_kernel_space()

thats in gfx_private

ok.. do you know offhand if that happens before or after clear_boot_mappings() is called? (I can check here in a bit, but as an experient, I'm trying to have clear_boot_mappings() skip over the mappings for the font boot module

I'm not sure, would need to verify.

hrm.. there's a bit of indirection going on, but it looks like it's not until the driver is attached (as in DDI_ATTACH)

i guess more generally, (today) any boot modules are loaded into < 4GB physical (and virtual by virtue of unity mapping) addresses, right?

are we doing something special for the ramdisk that maybe we're not for other modules (i.e. the console font module) to allow it to live after we clear all the VM mappings under _userlimit ?

we make a copy of console fonts and leit the module get freed

i saw that in some ddi setup function, but are we updating things to use the copy? it looked like at least some of the fb-related boot variables were still referencing the original addresses by the time clear_boot_mappings() was called

hm

unfortunately another meeting coming up, but I can see if i can find the spots where I noticed it last night again once it's over

(given that there's been at least one _very_ similar report booting openindiana on physical hardware, i'm thinking it's not something hyper-v specific, but maybe something about the VM in win2025 that makes it much easier to trigger)

that is good point.

also, i know our boot archive/ramdisks have gotten larger over time

and since it looks like the font boot module is stored right after the archive module in memory

i'm wondering if there's some implicit assumption we're violating about how memory < 4GB looks

how big is your ramdisk btw?

631MiB

(661,921,792 bytes)

uncompressed?

no, I think that's the size on disk

(that's what dboot reports the boot module size)

that should be uncompressed, we uncompress it while loading into memory

(when clear_boot_mappings() hits vaddr==0x2c04e00 in it's loop, that's where it hangs when x86pte_inval() writes 0 to the pte)

[0x0000000001394df8 0]: 661921792 (0x0000000027742000)

yep, thats the uncompressed size.

when console is fully set up, then font data is only used by tem and tems::print ts_font does display the same data as in fonts->stqh_first | ::print struct fontlist font_data | :print bitmap_data_t font | ::print struct font (huh, thats long chain:D)

but, I guess, the question is, if we get something printed by code in i86pc/boot/ after the mapping has changed....

so skipping the font module does allow things to proceed at least a bit, though we end up hanging again a bit later

I think, you did see boot_fb_font.

one other thing is gen1 vms seem to be fine

(except your boot device has to be IDE, and our IDE driver is so terrible that it's effectively unusable for much beyond 'look i can boot')

terrible in the sense that its _insanely_ slow due to the lack of DMA support

like 'a carrier transporting a flash drive could be competitive in performance' slow :)

pio modes for win;)

yeah, which 99% isn't a problem (because no one these days runs physical hardware w/ that anymore)

(I'm not even sure you can get 64-bit hardware w/ IDE -- it's probably all SATA)

i do notice that we start (in gen1) loading things at somewhat higher virtual addresses

2850_0000 instead of 2c04_0000

as a result the kernel ends up at higher physical addresses in the gen1 vm

(i thought there's a way to upload a file as a gist via the git command, if i can figure it out, I'll stick the gen1 vs gen2 boots out there)

what does gen1 memory map tell?

loading at higher address implies unusable chunks in memory map

err i got that backwards.. (too much task switching :P)

the gen1 is lower than gen2

as we load at (0xc00000 - sizeof elf header) + size of unix + MBI + modules and next free address will become first address to be used for allocations. then we allocate space for kernel and copy it, + some for memory maps. but if there are unusable areas in between, those will add up.

therefore start address is always the same but physical location of kernel depends on module sizes and memory map.

the size and specs of the two vms are otherwise identical

(aside from gen1 vs gen2)

(as a side note, I wouldn't mind tweaking some of that debugging output since it often requires you to be able to do math in hex to make sense of things)

well, in gen1 the rootfs is 0xe81000, end 0x285c3000 and in gen2 its 0x490b000, end 0x2c04d000, in gen2 case there must be something occupying ~58MB between loader loaded kernel and rootfs.

do you have gen2 loader memmap output handy?

i can get it -- if i'm not mistaken, it does look like in gen1, the address for module 0, 1, 2 are all increased (i.e. addr1 < addr2 < addr3) while gen2, module 1 is at a _lower_ address than module 0

i updated the gist with it

and gh helpfully put it in the middle of the two files :)

so your bad clear was 2c04e000 - which is the start of console-font module. you did disable memory map clear for it and the issue did move to next address?

yeah.. if we skip the entries for the font, we get further, though it appears we lock up again when we clear some larger addresses, thuough I haven't narrowed down the exact range yet

where do the console font files come from (in the boot module)?

size a620 vs 1d178

the environment is 869 vs 8d3 which seems reasonable given BIOS vs UEFI

console font size depends on selected font (based on resolution).

fonts are in /boot/fonts

ah.. i wonder if hv defaults to a larger size for gen1 vs gen2

the font and screen sizes are listed in loader env.

btw, did you try to disable clear_boot_mappings() like in that AMD Ryzen case?

not yet, though I can try that..

well, I guess its just about "yes, it is happy now" but does not really tell us more:D

i mean.. why would zeroing out that page table entry cause things to hang? the only thing I can think of is we're somehow still trying to use that VA (despite it being below _userlimit) when clear_boot_mappings() is called

we are (unfortunately) multi-threaded at that point, though still single-CPU

that PTE is in the middle of a page of entries, and otherwise doesn't seem special

the page table for 2c04e000 is at gist.github.com/jasonbking/bbe2c3b2…d7cd2f258671d85#file-gen2-txt-L1942

thats the tail of the list of tables... I wonder if "next" bad case was also falling there - which could hint about some sort of range limit.

virtual fffffd002cd67270 mapped to physical 2cd67270

that's the specific pte

so that all looks fine

so skipping that gets us all the way to sched() in main() if we skip clear_boot_mappings

hrm.. actually it gets all the way into userland if we skip that

I guess thats not a surprise.

jbk: to be clear because of my worries about i86mmu.c, have you confirmed (manually, don't trust ::report_map), that those mappings are correct?

if you do (random number here) `fffffd002cd67270/512K` and `2cd67270\512K`

confirm they're truly the same

you might be able to do that with ::dump and intelligence, but I never have

hopefully once i have a new build that lets me easily play with bits I can check again...

there's a sparate hyper-v specific issue because microsoft apparently can't settle on a consistent ACPI id for the vmbus

the bzero of htable bits you filed I'd wondered about, and then wondered if somehow it was a hot path

and then forgot

presumably if done in a kmem cache constructor, shouldn't that let it usually happen outside of a hot path

and when it doesn't, it's probably already in a sitaution where it's going to be taking longer already

not sure, see "somehow" and "forgot"

?

i just wanted to note that in case someone else sees it and freaks out a bit because they can't remember which pattern is used for which (redzone, uninit, etc) :)

yeah, I did the same thing

and then tried to remember how the sharing really works out

prior to kpti, a lot of the deep magic there was all about ism/dism

and pretty boring unless you're a database

oh yeah.. all the bending over backwards for oracle

so all the various processes can share the same memory, and use as few page tables

which is kinda funny all the effort that's gone into making oracle (RDMS) go fast on solaris, and then they royally botched it with I think solaris 11.3

(eventually fixed it, but initially VM2 made what at least externally feels like 'did no one tell you how one of your most critical apps runs on your OS?'

IIRC, using sysv shared memory would live lock the system -- not a question of if, but how many hours or days after boot