02:45:10 aha! the console font is at that address (not sure yet what that means) 03:00:20 [illumos-gate] 18119 nvme_print_identify_ctrl_list() needs to verify id count -- Robert Mustacchi 03:00:21 [illumos-gate] 18121 Plumb Micron 7600 nvme log pages -- Robert Mustacchi 03:00:21 [illumos-gate] 18122 add support for OCP Hardware Component log -- Robert Mustacchi 12:31:14 tsoome: is there an explicit point where we stop using the boot fb? 12:31:36 i couldn't find it looking yesterday (but it was late) 14:32:39 jbk basically with gfxp_map_kernel_space() 14:32:51 thats in gfx_private 14:33:57 ok.. do you know offhand if that happens before or after clear_boot_mappings() is called? (I can check here in a bit, but as an experient, I'm trying to have clear_boot_mappings() skip over the mappings for the font boot module 14:35:14 I'm not sure, would need to verify. 14:40:32 hrm.. there's a bit of indirection going on, but it looks like it's not until the driver is attached (as in DDI_ATTACH) 14:41:21 i guess more generally, (today) any boot modules are loaded into < 4GB physical (and virtual by virtue of unity mapping) addresses, right? 14:42:53 are we doing something special for the ramdisk that maybe we're not for other modules (i.e. the console font module) to allow it to live after we clear all the VM mappings under _userlimit ? 14:43:52 we make a copy of console fonts and leit the module get freed 14:45:05 i saw that in some ddi setup function, but are we updating things to use the copy? it looked like at least some of the fb-related boot variables were still referencing the original addresses by the time clear_boot_mappings() was called 14:45:24 hm 14:50:44 unfortunately another meeting coming up, but I can see if i can find the spots where I noticed it last night again once it's over 14:53:03 (given that there's been at least one _very_ similar report booting openindiana on physical hardware, i'm thinking it's not something hyper-v specific, but maybe something about the VM in win2025 that makes it much easier to trigger) 14:54:08 that is good point. 14:54:42 also, i know our boot archive/ramdisks have gotten larger over time 14:55:29 and since it looks like the font boot module is stored right after the archive module in memory 14:56:18 i'm wondering if there's some implicit assumption we're violating about how memory < 4GB looks 14:56:46 how big is your ramdisk btw? 14:58:51 631MiB 14:59:03 (661,921,792 bytes) 14:59:11 uncompressed? 14:59:30 no, I think that's the size on disk 14:59:40 (that's what dboot reports the boot module size) 15:00:26 that should be uncompressed, we uncompress it while loading into memory 15:01:19 https://pastebin.com/hR0tqi9U 15:02:17 (when clear_boot_mappings() hits vaddr==0x2c04e00 in it's loop, that's where it hangs when x86pte_inval() writes 0 to the pte) 15:03:21 [0x0000000001394df8 0]: 661921792 (0x0000000027742000) 15:03:21 yep, thats the uncompressed size. 15:22:25 when console is fully set up, then font data is only used by tem and tems::print ts_font does display the same data as in fonts->stqh_first | ::print struct fontlist font_data | :print bitmap_data_t font | ::print struct font (huh, thats long chain:D) 15:23:36 but, I guess, the question is, if we get something printed by code in i86pc/boot/ after the mapping has changed.... 15:28:30 so skipping the font module does allow things to proceed at least a bit, though we end up hanging again a bit later 15:34:18 I think, you did see boot_fb_font. 16:06:33 one other thing is gen1 vms seem to be fine 16:07:23 (except your boot device has to be IDE, and our IDE driver is so terrible that it's effectively unusable for much beyond 'look i can boot') 16:08:44 terrible in the sense that its _insanely_ slow due to the lack of DMA support 16:09:14 like 'a carrier transporting a flash drive could be competitive in performance' slow :) 16:09:59 pio modes for win;) 16:11:13 yeah, which 99% isn't a problem (because no one these days runs physical hardware w/ that anymore) 16:11:33 (I'm not even sure you can get 64-bit hardware w/ IDE -- it's probably all SATA) 16:12:05 i do notice that we start (in gen1) loading things at somewhat higher virtual addresses 16:12:22 2850_0000 instead of 2c04_0000 16:16:35 as a result the kernel ends up at higher physical addresses in the gen1 vm 16:17:17 (i thought there's a way to upload a file as a gist via the git command, if i can figure it out, I'll stick the gen1 vs gen2 boots out there) 16:20:12 what does gen1 memory map tell? 16:20:51 loading at higher address implies unusable chunks in memory map 16:22:07 err i got that backwards.. (too much task switching :P) 16:22:11 the gen1 is lower than gen2 16:26:15 as we load at (0xc00000 - sizeof elf header) + size of unix + MBI + modules and next free address will become first address to be used for allocations. then we allocate space for kernel and copy it, + some for memory maps. but if there are unusable areas in between, those will add up. 16:28:21 therefore start address is always the same but physical location of kernel depends on module sizes and memory map. 16:29:37 https://gist.github.com/jasonbking/bbe2c3b286895853ad7cd2f258671d85 16:30:10 the size and specs of the two vms are otherwise identical 16:30:16 (aside from gen1 vs gen2) 16:35:00 (as a side note, I wouldn't mind tweaking some of that debugging output since it often requires you to be able to do math in hex to make sense of things) 16:51:18 well, in gen1 the rootfs is 0xe81000, end 0x285c3000 and in gen2 its 0x490b000, end 0x2c04d000, in gen2 case there must be something occupying ~58MB between loader loaded kernel and rootfs. 16:56:27 do you have gen2 loader memmap output handy? 17:00:31 i can get it -- if i'm not mistaken, it does look like in gen1, the address for module 0, 1, 2 are all increased (i.e. addr1 < addr2 < addr3) while gen2, module 1 is at a _lower_ address than module 0 17:02:43 i updated the gist with it 17:02:53 and gh helpfully put it in the middle of the two files :) 17:13:44 so your bad clear was 2c04e000 - which is the start of console-font module. you did disable memory map clear for it and the issue did move to next address? 17:16:10 yeah.. if we skip the entries for the font, we get further, though it appears we lock up again when we clear some larger addresses, thuough I haven't narrowed down the exact range yet 17:19:24 where do the console font files come from (in the boot module)? 17:19:47 size a620 vs 1d178 17:21:05 the environment is 869 vs 8d3 which seems reasonable given BIOS vs UEFI 17:25:17 console font size depends on selected font (based on resolution). 17:25:39 fonts are in /boot/fonts 17:33:36 ah.. i wonder if hv defaults to a larger size for gen1 vs gen2 17:37:50 the font and screen sizes are listed in loader env. 17:38:32 btw, did you try to disable clear_boot_mappings() like in that AMD Ryzen case? 17:53:45 not yet, though I can try that.. 17:56:03 well, I guess its just about "yes, it is happy now" but does not really tell us more:D 17:59:26 i mean.. why would zeroing out that page table entry cause things to hang? the only thing I can think of is we're somehow still trying to use that VA (despite it being below _userlimit) when clear_boot_mappings() is called 17:59:38 we are (unfortunately) multi-threaded at that point, though still single-CPU 18:00:16 that PTE is in the middle of a page of entries, and otherwise doesn't seem special 18:00:29 the page table for 2c04e000 is at https://gist.github.com/jasonbking/bbe2c3b286895853ad7cd2f258671d85#file-gen2-txt-L1942 18:03:25 thats the tail of the list of tables... I wonder if "next" bad case was also falling there - which could hint about some sort of range limit. 18:06:41 virtual fffffd002cd67270 mapped to physical 2cd67270 18:07:00 that's the specific pte 18:08:40 so that all looks fine 18:37:04 so skipping that gets us all the way to sched() in main() if we skip clear_boot_mappings 19:05:29 hrm.. actually it gets all the way into userland if we skip that 19:48:52 I guess thats not a surprise. 19:51:55 jbk: to be clear because of my worries about i86mmu.c, have you confirmed (manually, don't trust ::report_map), that those mappings are correct? 19:52:31 if you do (random number here) `fffffd002cd67270/512K` and `2cd67270\512K` 19:52:38 confirm they're truly the same 19:53:13 you might be able to do that with ::dump and intelligence, but I never have 20:37:40 hopefully once i have a new build that lets me easily play with bits I can check again... 20:38:08 there's a sparate hyper-v specific issue because microsoft apparently can't settle on a consistent ACPI id for the vmbus 20:39:50 the bzero of htable bits you filed I'd wondered about, and then wondered if somehow it was a hot path 20:39:55 and then forgot 20:40:36 presumably if done in a kmem cache constructor, shouldn't that let it usually happen outside of a hot path 20:41:04 and when it doesn't, it's probably already in a sitaution where it's going to be taking longer already 20:41:06 not sure, see "somehow" and "forgot" 20:41:07 ? 20:41:43 i just wanted to note that in case someone else sees it and freaks out a bit because they can't remember which pattern is used for which (redzone, uninit, etc) :) 20:55:15 yeah, I did the same thing 20:55:40 and then tried to remember how the sharing really works out 20:55:52 prior to kpti, a lot of the deep magic there was all about ism/dism 20:55:56 and pretty boring unless you're a database 20:57:07 oh yeah.. all the bending over backwards for oracle 20:57:26 so all the various processes can share the same memory, and use as few page tables 21:04:50 which is kinda funny all the effort that's gone into making oracle (RDMS) go fast on solaris, and then they royally botched it with I think solaris 11.3 21:05:28 (eventually fixed it, but initially VM2 made what at least externally feels like 'did no one tell you how one of your most critical apps runs on your OS?' 21:06:38 IIRC, using sysv shared memory would live lock the system -- not a question of if, but how many hours or days after boot