17:09:11 <bahamat> pmooney: I've got SmartOS running under KVM on ubuntu 22.04, all of my Linux bhyve guests are getting a panic with general protection fault. I was wondering if you had any tips, or if you'd like to look into it.
17:09:56 <bahamat> I initially thought I was hitting OS-7738, which is a bhyve triple fault under kvm on ubuntu 18.04 closed as wontfix, but the comments say that ubuntu 19 works fine.
17:11:56 <pmooney> strictly speaking, running bhyve under nested virt isn't supported
17:12:11 <pmooney> it's the bhyve-guest kernel which is panicking with the #GP?
17:12:18 <bahamat> Yeah
17:12:35 <pmooney> the dmesg from that would probably be useful for tracking it down
17:12:42 <pmooney> it's potentially something we can fix
17:12:45 <bahamat> OK, I'll capture that and send it over.
17:12:59 <pmooney> (unlike the ubuntu 18 case, which was a legit nested-virt bug)
17:15:22 <bahamat> https://gist.github.com/bahamat/484f78f78535856ae1e014771877fcf4
17:16:01 <bahamat> This same json payload to create a guest worked with vmware fusion, so it's at least *mostly* sane :-)
17:17:51 <bahamat> SmartOS itself is being given 2G, and the bhyve is being given 512M.
17:20:26 <bahamat> Does oxide have a coal equivalent? I'm curious what you might be using for this kind of thing.
17:21:15 <bahamat> I tried vmware workstation on ubuntu, but it's not really working out for me because I want it to be headless, and it breaks every time the kernel changes (which is a lot for an LTS release).
17:22:01 <bahamat> But using kvm has worked out well so far, with this exception.
17:22:41 <gitomat> [illumos-gate] 15212 WPTS SMB 3.1.1 tests fail when encryption enabled -- Gordon Ross <gwr⊙rc>
17:22:54 <nahamu> bahamat: https://pkg.oxide.computer/install/ and https://pkg.oxide.computer/helios-dev/en/index.shtml are out there...
17:23:35 <nahamu> I don't think it includes their control plane software at this point, but I defer to jclulow on that.
17:23:56 <bahamat> nahamu: Right, but I mean where does that get run?
17:24:12 <rmustacc> We don't have a magic solution to nested virt.
17:24:24 <nahamu> ^ I guess that's the answer.
17:24:41 <bahamat> I'm on the verge of deprecating the SmartOS vmware images anyway...now vmware fusion is non viable, I don't even think anyone is using them anymore.
17:25:02 <bahamat> rmustacc: yeah, that answers my question.
17:35:03 <danmcd> You can use the ISO installer on VMware fusion for either SmartOS or Triton.  And I know nested-virt there is iffy, given the jitter in pass/fail of certain bhyve/VMM tests every time I test a release.
17:58:02 <gitomat> [illumos-gate] 15213 SMB should expose encrypt_ciphers to sharectl -- Gordon Ross <gwr⊙rc>
17:59:21 <bahamat> danmcd: Yeah, the ISO should work fine. The main issue with fusion these days is macs with MX processor.
17:59:32 <danmcd> Oh yeah that.
18:20:03 <pmooney> bahamat: if that nested instance is not rebooting after the #GP, it might be worth trying to attach with mdb-bhyve to verify the instruction(s) in question which triggered it
18:20:13 <pmooney> what I'm seeing from the decoded bytes doesn't make a ton of sense
18:53:59 <bahamat> pmooney: Yeah, it's not rebooting. What's in the gist is the last output from it and it never moves after that.
18:54:40 <bahamat> I've never used mdb-bhyve...do you have a primer I can use?
18:55:38 <pmooney> from the gz: mdb -b <bhyve_instance_name>
18:55:53 <pmooney> you can then poke at stuff in the instance
18:57:03 <pmooney> of, of course it doesn't include the numeric %rip
18:57:06 <pmooney> *oh
18:57:47 <pmooney> but you might be able to find where you are with $C
18:58:11 <bahamat> mdb: failed to open 37be9c8b-c32c-6462-eb5e-cbcc7a35195c: No such file or directory
18:58:16 <bahamat> Do I need the full path to something?
19:07:09 <pmooney> no, you shouldn't
19:07:28 <pmooney> I don't recall if the smartos bhyve plumbing uses the uuid as the name, or something else
19:07:38 <pmooney> IIRC it might use some string derived from the zoneid
19:08:05 <pmooney> pargs-ing the process in question should get you a hint at the name
19:08:05 <bahamat> can I find that out from mdb?
19:08:52 <bahamat> maybe SYSbhyve-7?
19:08:55 <bahamat> That's the last arg
19:09:00 <pmooney> yeah
19:09:02 <pmooney> that'll be the name
19:09:06 <pmooney> SYSbhyve-7
19:09:15 <bahamat> OK, yeah, that's doing something.
19:09:17 <pmooney> (which I would expect to see in /dev/vmm
19:09:20 <bahamat> No prompt yet
19:09:59 <pmooney> oh, what kind of machine is this?
19:10:11 <pmooney> AMD or intel?
19:10:13 <bahamat> It's an Intel NUC
19:10:25 <pmooney> kk
19:10:56 <pmooney> truth be told, I haven't tried live debugging under nested either
19:11:05 <pmooney> typically I'm doing it on a dead instance
19:11:06 <bahamat> On the ubuntu (host) side, lscpu tells me that it's "Intel(R) Core(TM) i3-10110U CPU @ 2.10GHz"
19:11:27 <bahamat> psrinfo tells me it's "Intel(r) Core(tm) i3-10110U CPU @ 2.10GHz"
19:12:50 <bahamat> pstack of mdb says it did vmm_open_vm, then usleep
19:12:57 <bahamat> Oh! it returned. I have a prompt now.
19:12:59 <bahamat> Nice.
19:13:21 <bahamat> Stack trace is all hex, I'm not sure if that'll be useful to you
19:13:59 <pmooney> that's the challenge here - we don't have tooling to go dig up the kernel symbols
19:17:52 <richlowe> there should be a word for making merging with yourself suck
19:41:15 <toasterson> bahamat: is that vmware fusion with linux as host?
19:42:51 <bahamat> toasterson: No, it's linux-kvm
19:44:00 <toasterson> hmmm, I thought I tried that some time ago but maybe not? Curious on the output then :)
19:45:48 <bahamat> The whole chain here is linux-kvm -> smartos-bhyve -> linux
19:45:59 <bahamat> And it's that last linux that's blowing up.
19:46:14 <bahamat> smartos itself runs fine as a kvm guest with a linux host
19:46:55 <bahamat> nested bhyve is the only thing not working so far. Nested kvm seems to be ok.
23:39:58 <alanc> you know you've dug deep when the answer to a "what the hell is that?" question about one of our man pages is found in https://github.com/dspinellis/unix-history-repo/blob/BSD-4_1_snap-Snapshot-Development/usr/old
23:46:59 <alanc> how many people knew illumos includes a command named "error" to make it easier to handle compiler errors on your ancient terminal by inserting them into the source code at the right line?
23:47:35 <alanc> though strangely, you seem to have lost the error(1) man page but still have the code in usr/src/cmd/sgs/error