Had a bhyve VM displaying OOM Killer in the console logs I ran `vmadm shut <UUID> -F`. In another terminal I had a bhyve console connect and I saw the `zone rebooting` message pop-up. Then never go any further. I finally `ctl-c`ed my `vmadm stop` command. I was left with a Zombie process that cannot be reaped with `preap`

john: What platform version are you on?

20221225T144337Z

This is barfield btw, not sure why my nick is wrong I did set it on login.

We had a major bhyve upgrade starting with 20230223

I remember seeing Dan's notifications about upstream merges

Ultimately, what you're describing is a bug in bhyve, and the update might have fixed that already.

Aha, that explains it. Is there a work around to kill the VM in the future? like bhyvectl --destroy or something rather than vmadm stop -F?

Unfortunately things like that aren't just easily reproducible to test if it's fixed, but if you're using bhyve you almost certainly want to be on 20230223 or later.

I will start a platform upgrade then ASAP. All of these systems are bhyve

With that being said, does Triton and Manta need to be updated for any reason as well? I've been considering going through that as well.

Maybe? But also remember that bhyve (as well as qemu) runs inside a zone so take that into account.

It's best to review the release announcements for highlights. I don't know what release you're on so I can't say what you're missing out on.

Yeah, I knew that, because the actual zombie is is coming from zone init

I'm only a couple of months in arrears on triton

The big things from the most recent release is that pkgsrc-tools can be set up during headnode install, rather than having (and potentially forgetting) to do it later, and the shared/support shell (triton-scry) utility.

Cool. I saw the new shell, that is pretty awesome. We could have used that back when I worked at Computer Associates lol

Yeah, it's super handy.

20230406T001127Z

looks like that is the latest available

Theres no way to hotpatch bhyve without bouncing every node that you're aware of is there?

bahamat: is the linuxCN project still under development?

john: vmm and viona are separate modules. You _might_ be able to do some funky stuff to "hotpatch" bhyve. It's been awhile since I tried doing that. And iirc you needed mdb -kw to actually unload vmm etc. pmooney might know if it's even worth a try these days

and of course that means you would also need the right bhyve executable that works with those kernel modules

I would not recommend loading a newer bhyve or viona module on an older kernel

there is so much which could go wrong

and yes, you might need an updated libvmmapi or bhyve binary in userspace as well

that isn't to say that doing such a thing is necessarily impossible

but it's fraught, and would require careful research

and is likely to put you in an even more tenuous position when it comes to correct behavior

papertigers: Sounds like more work than I am willing to even attempt! Hows your new(ish) gig anyway? Been a long time!

pmooney: Thank you for the insight definitely sounds risky. I'll just deal with the reboots. But I appreciate the architectural breakdown at the very least. Good information.

john: going well! still doing all the illumos things in my spare time haha. How are you?

I will DM you sometime and give you all the updates :) ... I'm still doing it too...illumos4life

I always have to remember how to even do a DM in irssi/irc lol

john: about linuxcn: Depends on your definition of "still". It's still a project that we want to pursue, but there hasn't been much active development on it lately (largely due to the multiple truckloads of work moving all of Triton from one owner to another). But it's also being rekindled and hopefully we'll have a new "blessed" release soon-ish.

bahamat: thank you, I am looking forward to playing around with it. Though I will always be a smartOS user first.