pmooney: I've got SmartOS running under KVM on ubuntu 22.04, all of my Linux bhyve guests are getting a panic with general protection fault. I was wondering if you had any tips, or if you'd like to look into it.

I initially thought I was hitting OS-7738, which is a bhyve triple fault under kvm on ubuntu 18.04 closed as wontfix, but the comments say that ubuntu 19 works fine.

strictly speaking, running bhyve under nested virt isn't supported

it's the bhyve-guest kernel which is panicking with the #GP?

Yeah

the dmesg from that would probably be useful for tracking it down

it's potentially something we can fix

OK, I'll capture that and send it over.

(unlike the ubuntu 18 case, which was a legit nested-virt bug)

This same json payload to create a guest worked with vmware fusion, so it's at least *mostly* sane :-)

SmartOS itself is being given 2G, and the bhyve is being given 512M.

Does oxide have a coal equivalent? I'm curious what you might be using for this kind of thing.

I tried vmware workstation on ubuntu, but it's not really working out for me because I want it to be headless, and it breaks every time the kernel changes (which is a lot for an LTS release).

But using kvm has worked out well so far, with this exception.

[illumos-gate] 15212 WPTS SMB 3.1.1 tests fail when encryption enabled -- Gordon Ross <gwr⊙rc>

bahamat: pkg.oxide.computer/install and pkg.oxide.computer/helios-dev/en/index.shtml are out there...

I don't think it includes their control plane software at this point, but I defer to jclulow on that.

nahamu: Right, but I mean where does that get run?

We don't have a magic solution to nested virt.

^ I guess that's the answer.

I'm on the verge of deprecating the SmartOS vmware images anyway...now vmware fusion is non viable, I don't even think anyone is using them anymore.

rmustacc: yeah, that answers my question.

You can use the ISO installer on VMware fusion for either SmartOS or Triton. And I know nested-virt there is iffy, given the jitter in pass/fail of certain bhyve/VMM tests every time I test a release.

[illumos-gate] 15213 SMB should expose encrypt_ciphers to sharectl -- Gordon Ross <gwr⊙rc>

danmcd: Yeah, the ISO should work fine. The main issue with fusion these days is macs with MX processor.

Oh yeah that.

bahamat: if that nested instance is not rebooting after the #GP, it might be worth trying to attach with mdb-bhyve to verify the instruction(s) in question which triggered it

what I'm seeing from the decoded bytes doesn't make a ton of sense

pmooney: Yeah, it's not rebooting. What's in the gist is the last output from it and it never moves after that.

I've never used mdb-bhyve...do you have a primer I can use?

from the gz: mdb -b <bhyve_instance_name>

you can then poke at stuff in the instance

of, of course it doesn't include the numeric %rip

*oh

but you might be able to find where you are with $C

mdb: failed to open 37be9c8b-c32c-6462-eb5e-cbcc7a35195c: No such file or directory

Do I need the full path to something?

no, you shouldn't

I don't recall if the smartos bhyve plumbing uses the uuid as the name, or something else

IIRC it might use some string derived from the zoneid

pargs-ing the process in question should get you a hint at the name

can I find that out from mdb?

maybe SYSbhyve-7?

That's the last arg

yeah

that'll be the name

SYSbhyve-7

OK, yeah, that's doing something.

(which I would expect to see in /dev/vmm

No prompt yet

oh, what kind of machine is this?

AMD or intel?

It's an Intel NUC

kk

truth be told, I haven't tried live debugging under nested either

typically I'm doing it on a dead instance

On the ubuntu (host) side, lscpu tells me that it's "Intel(R) Core(TM) i3-10110U CPU @ 2.10GHz"

psrinfo tells me it's "Intel(r) Core(tm) i3-10110U CPU @ 2.10GHz"

pstack of mdb says it did vmm_open_vm, then usleep

Oh! it returned. I have a prompt now.

Nice.

Stack trace is all hex, I'm not sure if that'll be useful to you

that's the challenge here - we don't have tooling to go dig up the kernel symbols

there should be a word for making merging with yourself suck

bahamat: is that vmware fusion with linux as host?

toasterson: No, it's linux-kvm

hmmm, I thought I tried that some time ago but maybe not? Curious on the output then :)

The whole chain here is linux-kvm -> smartos-bhyve -> linux

And it's that last linux that's blowing up.

smartos itself runs fine as a kvm guest with a linux host

nested bhyve is the only thing not working so far. Nested kvm seems to be ok.

you know you've dug deep when the answer to a "what the hell is that?" question about one of our man pages is found in github.com/dspinellis/unix-history-…1_snap-Snapshot-Development/usr/old

how many people knew illumos includes a command named "error" to make it easier to handle compiler errors on your ancient terminal by inserting them into the source code at the right line?

though strangely, you seem to have lost the error(1) man page but still have the code in usr/src/cmd/sgs/error