i like how there's an htable cache for the htable cache

as well as a reserve

(which all makes it hard to determine if all the fields are always getting properly initialized)

since in at least some cases ht_shares it is not

(though it's unclear if it ever actually causes a problem)

hrm..

if you have two consecutive pages (by vaddr), the _address_ of the page table entries that map those va's to pa's should be off by 8 bytes (same level) i.e. adjacent entries in an array, right?

jbk: yes unless the pte for the first page is at the end of a page-table page -- then it depends on what's in the next level up)

Hey folks!

I've been mapping out what confidential computing (mainly on AMD, e.g. AMD SEV-SNP) support would look like in the bhyve/illumos stack, and an Oxide engineer suggested I reach out here to see if there's prior art or existing interest from the OS/bhyve side. The bulk of the kernel work would probably live in vmm.ko (RMP management, memory pinning, and a driver interface to the AMD Secure Processor firmware, etc.), while the userland (e.g. Propolis) side seems

comparatively cleaner.

Has this come up in illumos before? I'm thinking about drafting a design proposal for the kernel plumbing and wanted to get a sense of the landscape before going too far down that road.

Thanks, MJ

Apologies for the poorly formatted message, that was literally my first message on IRC

[illumos-gate] 18157 NFSv4.1 backchannel slot negotiation bug -- Gordon Ross <gordon.w.ross⊙gc>

[illumos-gate] 18159 NFSv4.1 panic in rfs4x_do_cb_recall -- Gordon Ross <gordon.w.ross⊙gc>

(for the logs, since he left) mj_p: I haven't seen prior discussion of enabling confidential computing hardware for bhyve/illumos either here or on the mailing lists. memory pinning of vms seems like it could be useful even without the confidential computing work.

oh ffs

you can't 0-pad a %p pointer in the kernel

so this looks _very, very_ similar: illumos.topicbox.com/groups/discuss/Tbc4d081ad58aef80

(and not on hyper-v)

it's hanging on the exact same instruction

in the exact same function

jbk: so, one thought here regarding clear_boot_mappings() is that you don't really need to do the locked CAS_PTE at all in this case as you don't care about mod/ref bits on the mappings you're removing.

so having clear_boot_mappings use an alternate x86pte_inval() call which dispenses with the CAS_PTE might get you past this point in boot.

that at least would distinguish between it being the CAS that's broken vs. the write to that particular PTE that's causing trouble.

been in meetings

but so I had tried writing to the same PTE (the value that was there)

that's fine

writing 0 from kmdb also hangs just like the cas

so that appears to be the problem

so i guess now the question is why would clearing that mapping hang a system

writing the existing value back seems to be ok though

also skipping that page and trying the next one also hangs

so it seems like something starts there

but it doesn't appear to be on any boundary

(why do i get the feeling that once I figure out the right place to look, it'll become very obvious :P)

hrm..

jbk: okay, some obvious things to check: any registers pointing at the mapped VA? (stack pointer? frame pointer? program counter?)

[illumos-gate] 15725 Replace uselocale(NULL) with __curlocale() in libc -- Bill Sommerfeld <sommerfeld⊙ho>

not that i can tell...

richlowe: the elftest fails because the assembler now refuses to assemble "leaq foo@GOTTPOFF(%rip), %rsi".

so you may need to check in a .o file ...

the fact a very similar issue has been seen outside of hyper-v does worry me a bit...

it makes me think there's either some edge case we're screwing up in early boot

or we're touching memory that isn't identified by UEFI yet we should still be avoiding

(I did look at the memory maps in UEFI -- the addresses are labeled as conventional memory and aren't anywhere near any boundaries

sommerfeld: Its mj_p, continuing the discussion on confidential computing: Thanks that’s helpful to know. The clean slate is actually encouraging. The memory pinning point is interesting too! I could see that being worth scoping as a standalone piece even in the near term. Is there a preferred place to share early design thinking? should I aim

for a formal IPD/RFD, the mailing list, or just bounce ideas here first?

pooladkhay: any of those can work; which one is best depends on how far along you are. Design doc -> IPD or mailing list, then open an issue or three on illumos.org/issues; random half-based musings: here or mailing list.

sommerfeld: Makes sense, thanks. Right now I'm somewhere between half-baked and a concrete plan. Specifically looking at scoping out generic memory pinning first as a foundational piece based on your suggestion. I'll sketch out a summary of my thoughts on that and will send it here to get some early feedback before starting an email thread.

Yeah. I'll say figuring out what problems we're trying to solve and how excited folks are for what level of invasiveness, and where is it that you have the hardware to complete this versus are just trying to come up with a design are important other pieces here.

rmustacc: fair points! On scope and invasiveness: that’s mainly why sommerfeld’s memory pinning observation caught my attention. It seems like a step that could deliver standalone value with a more bounded blast radius than jumping straight into the full SNP stack. On hardware: I don’t have an EPYC node running illumos today, but I’m

looking into sourcing testing options. I know blind kernel development is a dead end, so I want any foundational work validated before stepping into the actual SNP hardware phase. I believe pinning can be developed on commodity hardware which is another reason why it’s a good idea.

pooladkhay: Memory is already pinned basically in the illumos implementations of bhyve (and kvm before it).

rmustacc: good to know, thanks for the correction. That’s exactly why I’m asking before writing anything. Let’s take a step back first: is supporting confidential workloads something the illumos/bhyve community actually wants to have? I know KVM has been supporting that which was probably backed by major cloud providers.

If yes, what would you say are the real missing foundational pieces between where illumos bhyve sits today and SNP support? RMP management and the AMD SP interface seem like the obvious gaps from the outside, but I’d rather hear your opinion.

oh hrm... i just noticed that the first VA (unity mapped) that hangs is ramdisk_end

that seems suspicious