bahamat: I did some experimenting, and that problem with the DNS servers being tied to the global zone configuration (8.8.8.8/8.8.4.4) does seem to be exlusive to the debian-12 image. The ubuntu-22.04 HVM image seems to resolve against my internal resolvers just fine. The base-64-lts zone does as well.

In general, LX images aren't working well for me at the moment, so I can't really test that (lots of angry errors like: "netmgr/udp.c:1019:isc_nm_udpconnect(): fatal error: RUNTIME_CHECK(result == ISC_R_SUCCESS || result == ISC_R_NOTIMPLEMENTED) failed").

I did try the debian-12 HVM image on both KVM and Bhyve and they both exhibit the same problem behavior.

jdt: I found the cause of the issue with the deb12 images. FWIW, it also affects deb11.

I'll be publishing new images soon (tomorrow, or early next week) that will address it.

What application are you running that's producing those errors? I'd like to dig more into that.

just dig and nslookup

here's a quick (multi-line) example:

root@varnish-test:~# dig cnn.com

netmgr/udp.c:1019:isc_nm_udpconnect(): fatal error: RUNTIME_CHECK(result == ISC_R_SUCCESS || result == ISC_R_NOTIMPLEMENTED) failed

Aborted (core dumped)

root@varnish-test:~# nslookup

> cnn.com

netmgr/udp.c:1019:isc_nm_udpconnect(): fatal error: RUNTIME_CHECK(result == ISC_R_SUCCESS || result == ISC_R_NOTIMPLEMENTED) failed

Aborted (core dumped)

And that's the deb11/12 LX image?

LX Instance (Debian GNU/Linux 12 (bookworm) 20230721)

Fwiw, I couldn't get the latest void LX instance to provision at all. It tries to, spins for 5 minutes in "provisioning" and then fails.

Yeah, I've had some issues with the void image which is why I haven't published a new one recently.

I figured that was probably a known issue.

Random question - I'm thinking about adding another CN to my cluster and was thinking mayby Linux CN. Probably obvious, but I wouldn't be able to run SmartOS zones on that, right?

*maybe

definitely not

Linux doesn't do illumos syscall translation in any form.

Makes sense. I assume it does bhyve and kvm, though?

No.

Oh, okay.

Currently it's LXC only.

Got it.

Thanks for the clarification.

Adding KVM is possible, and not even really that hard. But it would take some additional work in imgapi/imgadm and vmapi/vmadm.

It just remains to be done.

But we're not porting bhyve to Linux.

Ah yeah, I see that now in the "Importing Supported Images".

A prerequisite to even considering bhyve on linux is someone from the linux and/or freebsd communit(y|ies) does that work first.

Until then, there's zero chance.

Odds of that even happening, I think are roughly zero.

Still, LXC is useful in its own right.

It is. It's super useful in some cases.

Even getting KVM on Linux working with the rest of our stuff, whether we can use the same images as SmartOS remains to be seen.

LXC provides KVM images, and it uses a model similar to SmartOS where they run qemu inside a container.

So we could just get those, but those images probably wouldn't be usable on SmartOS, and the images we produce now probably wouldn't be usable by LXC.

At some point the cost-benefit ratio has to be considered, I suppose. Do you think there's a desire to move toward Linux with the platform as a whole?

Whether the strategy is "SmartOS for the future" or "Linux for the future" is probably the most important decision.

Well, SmartOS zones alone can fill the vast majority of needs. But we're not going to dictate that people use SmartOS zones if they don't want to. People have preferences, existing tooling, existing workflows, etc. It can be extremely disruptive to change all of that. And I get it.

LX also satisfies a huge set of use cases. There are some bugs and incompatibilities with it. Some of those are easier to address than others.

Where there are issues with LX, bhyve is almost always sufficient. Certainly at least as sufficient as any other VM platform from any other provider.

There's an extremely narrow set of cases where someone wants container level performance (i.e., a vm isn't good enough), but LX also won't work.

In those cases, LXC solves that.

Personally, I like using SmartOS zones when I can, but I do find myself using Debian more frequently (mostly because the company I do the most work with prefers it and is currently using Ganeti to manage VMs - I'd love to move them to Triton at some point, though).

The use cases that I know of where LXC is definitely more suited are running a firewall/router/packet filter in a continer (i.e., you actually want iptables)

The other is like running Oracle DB (or something similar..maybe SAP?)

Interesting: github.com/jack9603301/vyos-on-lxd - that's something to experiment with.

(For me. :) )

Yeah, that's the kind of thing where LXC is the best place to run it.

We're never going to provide netfilter compatibility, and for something like that you want higher performance than you can get out of a VM.