voy4g3r2: The joke is I don't like fonts with ligatures.

You never know when typing in hieroglyphs will be required.

nullfs looks like just a symlink, but i'm sure i'm wrong so how pls?

kerneldove: it is _similar_ to a symlink. There are use cases where a symlink wouldn't work, like creating a jail and sharing a host path into the jail.

so it's like symlink + fs emulation in a way

does it pass through quota stuff? like if i root a nullfs in a dir on the host system and it has a quota limiting it to 1GB, will inside the jail see that as a 1GB disk?

realpath on a symlink'd path will resolve to the full path while a nullfs mount won't, which has uses

disk/fs

a UFS fs?

*shrug*

I expect quotas to still be enforced but I don't know that it could be seen and it probably depends on other things

voy4g3r2, how'd you get sound to work through rdp? using xrdp or?

hm ok

kerneldove: I have sound working with xrdp.

how?

in docs.freebsd.org/en/books/handbook/…reating-thin-jail-openzfs-snapshots it says `# tar -xf /usr/local/jails/media/14.2-RELEASE-base.txz -C /usr/local/jails/templates/14.2-RELEASE --unlink` but what's --unlink needed for? i read its doc in man tar but still don't get it

kerneldove: I installed audio/pulseaudio-module-xrdp

on host or vm?

kerneldove: VM

ek any config required or just install pkg?

Just install the package.

man that's cool

Yep. I rarely use it, but last it did it seemed to work pretty solidly.

Apparently, there's a Pipewire xrdp module as well (audio/pipewire-module-xrdp). I haven't used this one, though.

kerneldove: i installed the pulseaudio module

sorry for delay.. was at the black crowes concert

and it just "worked" unlike the linux-browser-installer

im experimenting with the most minimal thin jail i can make to learn how stuff works. i have ip4 set to inherit and now i can resolve domains to ip, but how? i didn't make an /etc/resolv.conf yet

your jails keeps the hosts /etc/resolve.conf

that's empty tho

i verify that with sudo jexec -l test vi /etc/resolv.conf

jail conf has host.hostname, path, exec.clean, exec.consolelog, exec.start, exec.stop, ip4 (inherit), mount.devfs (zfs vol)

untar'd base into it. didn't even set timezone because it defaults to host (UTC)

so how tf can the jail resolv domains? sudo jexec -l test host www.freebsd.org

anyone know?

without an /etc/resolv.conf (inside the jail) the stub resolver will attempt to query 127.0.0.1 (the ipv4 loopback)

so dns will only work if you have a local dns resolver bound to the loopback address

ah it 'inherits' the host system, just like tz. ty!

the ip4=inherit setting makes the jail inherit the host network stack

i would recommend adding ip6=inherit as well for IPv6

even if the host system doesn't have ipv6 setup or used?

yes, because it's not really disabled it's just not configured by default

ok i'll add that ty

so a jail can have several different basic networking configuration types

a jail can have its network access disabled, inherit the host network access (minus raw and bpf sockets), or restricted to a subset of the host ip addresses (called alias networking because the ip addresses given to the jail are normally aliases)

or the jail can have it's own network stack call vnet jails or VIMAGE after the kernel configuration to enable it

the easiest to use is ip4=inherit and ip6=inherit

then there's also netgraph networking right?

netgraph is it's own thing

it can be combined with jails but it's not directly tied into jails

if you use inherited networking for jails the network services must be configured not to clash (don't bind tcp port 22 on the any address for ssh)

the traditional way to do sane secure jail deployments is alias jails

you configure the host to explicitly bind only on addresses not exposed to the jails

e.g. don't bind services like ssh and syslog to the any address on the host

then you add alias ips to your network interface (with /32 or /128 prefix length)

and allow the jail to use these alias ips

you need a /etc/resolv.conf inside the jail for dns to work

if you want to you can copy with an exec.created = "cat /etc/resolv.conf | jexec $name dd of=/etc/resolv.conf";

i wouldn't recommend just using cp because in theory the jail could contain a dangerous symlink that the hook running on the host would follow overwriting a file outside of the jail root

or if you network is static (enough) you can copy the resolv.conf once during jail creation out of trusted sources

ok that's really cool, giving an ip to the host but reserving aliases for jails. iirc that's covered in the chapter on jails but it's not explicit. docs.freebsd.org/en/books/handbook/jails/#jails-networking ya it's not. imho that would be a great addition. host network + ip aliases

i'll use inherit host networking when the box only has 1 ip, and ip alias host networking when it has >1 ip

tyvm crest really helpful

with alias jails, can the jail bind to any port even tho it's not using its own net stack, because it has its own ip?

yes

you can bind ports < 1024 as long as you're root inside the jail or if the mac_portacl(4) kernel module has been loaded and configured

e.g. to allow ntpd to bind port 123 without starting as root

or keeping a process running as root around to bind new sockets

how can i see everything running in a jail? just jexec -l test ps -aux? i know i can't use service -e because that's not necessarily all that's running

do i have thin jail user mgmt right? host system gets user added to it for each jail. then to manage jail, user ssh into host and somehow has access to jail's files?

on my thick jails where jail runs sshd and has its own user that i ssh into and manage files, how do i do that with thin jails? ssh user for jail host, but user ssh in then how manages files? files are in /var/jail/jails/test but how do i bridge the divide?

i guess for thin jails it might be ok to have less jail isolation, and use 1 host ssh user to manage files of all thin jails?

pgrep can filter by jail

ok and what's the right way to create users for thin jails and allow them to manage the jail? (its files mostly but also service(s))

I would think you would use adduser(8) inside the thin jail to add the users you want in the jail

but what good is that? do you ssh into that?

ssh/sftp is how i transfer app files into servers and it's always through a user account

Maybe I'm misunderstanding what you're asking for. You want users inside the thin jail who can then manage the jail? Or you want users on the host that can manage the thin jails from the host?

My needs are different and I don't ssh into the jails at the moment.

that's what i'm asking. like i said, my thick jail model is simple: jails have their own vnet, their own ip, run their own sshd, have their own internal jail user that you ssh into to transfer files into the jail and manage the service. but thin jails, using host networking, i'm not sure how to transfer files into jail

you can use thin jails with vnet and their own ip, run sshd inside and so on.

ah ok i'll say it another way. with inherit networking jails, how to handle transferring files into them and managing them?

inherit networking meaning jail shares host's net stack, ip, and sshd only runs on host

you could also use nullfs mounts inside the thinjail, then you can "transfer" the files by just sftp/scp/ssh to the right location on the host, no need to be inside the jail

ok so i ssh into either 1 user on jail host, or 1 user per jail. regardless, i'm connecting to user on jail host. then what, i have a dir from within that user account that's mapped into the jail or?

with inherited networking, based on my limited knowledge, nullfs mounts seems to be the best way to transfer files to the thinjail. I use that right now for my web service.

do you run zfs on jail host and in the jail?

kerneldove: yes. I have /usr/local/www mounted inside the thinjail as a nullfs mount. All my users on the jail host write to /usr/local/www which is available to the web server running inside the jail. No one needs to ssh into the jail at all. Don't need to do user management or have duplicates of files, etc.

kerneldove: yes, I use ZFS.

I don't have any other filesystem being ussed on my system.

ok sounds like what i want. do you put jails in /var/jail or what's your jail base dir

I put them in /usr/local/jail but I have /jail and /usr/jail symlinked to /usr/local/jail so if I'm copying and pasting in commands, they'll still work :-)

I followed the Handbook chapter on Jails and set things up accordingly.

mind if i ask what your config is to set up the nullfs map from /usr/local/www into the /usr/local/jail/... dir?

kerneldove: docs.freebsd.org/en/books/handbook/jails/#thin-jail section 17.5.2, at the end has the config that I based mine from

that section talks about "skeleton" but wtf is that? it's not explained

i already have the zfs dirs made like media, templates, jails

What you are interested in is the mount.fstab entry that is there in the config file and the the fstab file shown after the jail.conf file at the end of the section.

I haven't done the skeleton setup. I'll get to that at some point, but don't need it at the moment. Right now, just using simple thin jails with nullfs and ZFS

good to know. that skeleton stuff should be broken out into its own section it makes nullfs setup look like a messy pain

should be in a 'best practices' kinda section or smth dunno

Maybe

all the fstab stuff uses /usr/local/jails/ so i can't make out what's what side of the mapping

kerneldove: you should use jailed = on for the dataset and use exec.created = zfs jail ... in jail.conf

the you can manage the dataset in the jail

what's the benefit to that?

well, you asked for "user to manage zfs in the jail ..?"

files

ah

that's the nullfs stuff im trying to figure out

2 pages down from docs.freebsd.org/en/books/handbook/jails/#creating-thin-jail-nullfs

yeah, I'm using plenty of nullfs mounts at $work

kerneldove: the first entry is the filepath on the host, the second is the jail path from the perspective of the host

so path = "/usr/local/jails/${name}-nullfs-base"; says that's where the jail fs is based? what's the .fstab file there saying? i can't make it out

I suggest to avoid creating jails with nullfs-based mounts

why?

if you have dozen of jails it's a hell to maintain

ok so then how do i manage the files and services inside each jail?

(just my humble opinion)

I'm managing everything in Saltstack

ok but what's the actual layout of the freebsd system

kerneldove: path says that is where the jail is located. The mount.fstab entry points to where the fstab file is that will be used to process the nullfs mounts.

the .fstab file is shown right below the jail.conf file

the best way to have something maintainable is to create a separate data partition for each jail and a reproductible installation with a CMS like Ansible or Salstack

ya, i see it, i just can't decipher it and especially in the context of this infinite sequence of base dir, template dir, skeleton dir, then fstabs referencing all those

fucking convoluted

IMHO thick jails > thin jails, especially with ZFS

why

because of independance and isolation

ya well i got a working thick jail setup. i'm trying now to figure out thin jails

mage: if you have several jails, aren't thin jails easier to manage and maintain than thick jails?

i think they can have a purpose

mns: yes if you don't use a CMS, but with a CMS it's about the same

especially with a separate data-like partition

kerneldove: the handbook has the information you need, you don't have to use the example as given, you can fine tune it the way you want. to keep it simple, all you need is the mount.fstab entry in your jail.conf and then provide the fstab file in the location you specify for the mount.fstab entry.

(that's my point of view and years of experience with FreeBSD and many jails at $work)

mage: even when you have a single host at home with a few jails?

of course not :)

mns well ya but i'm wondering, why even both using the nullfs map instead of managing files directly through the jail's dir?

mine is more limited experience and at $home.

kerneldove: both?

like why bother with /usr/local/jails/thinjail-nullfs-base/skeleton when it just maps directly to /usr/local/jails/containers/thinjail ?

bother*

Why is freebsd "Free" if it has built-in "jails"?

the nullfs mounts are for you to "transfer" files into your jail. skeleton is more about how to setup the jail

having jails doe not make anyone closed;)

like are you saying change fstab to "/usr/local/jails/containers/thinjail /home/jailadmin/jailroot nullfs rw 0 0"? then use jailadmin user to sftp files into the jail?

also, now that PKGBASE will soon be the default upgrade path (I hope) it would be even faster to update thick jails and don't bother with freebsd-update

as freebsd-update is painfully slow with many jails

kerneldove: "/usr/local/fubar /usr/local/jails/webserver/opt/fubar nullfs rw 0 0" will mount the hosts /usr/local/fubar directory inside the jail called "webserver" at its /opt/fubar directory. Users can put files in /usr/local/fubar and they will show up in /opt/fubar inside the jail. No one has to ssh/scp/sftp into the jail.

the mount.fstab entry uses a standard fstab file format, nothing specialised about it.

even if the jail is running, as soon as user on host makes file changes at /usr/local/fubar, from within the jail's perspective the changes at /opt/fubar reflect that?

yes

has anybody been able to escape from a "jail"?

yes

so why not have host users directly modify files via /usr/local/jails/webserver/ ?

scottpedia: there are CVEs for jails, so that would count as escaping, yes?

okay yes I think that counts

kerneldove: that would mean giving them root access, which I'm sure you dont want to do

ahh ok so the nullfs mapping gives you an entity that limited privs can be bound to

that makes sense

it's a permissions pinhole

has anybody here ever spent time in a "jail"?

/usr/local/jails/webserver also has etc/ usr/ var/ boot/ dev/ lib/ libexec/ etc. underneath it, and you wouldn't want anyone modifying those files.

mage: how do you find saltstack compared to ansible, since you've used both

I've not used Ansible, only Saltstack

and I think that if I had to do it again I would probably have choosen Ansible

oh, why?

ya why lol

Saltstack has bugs and the code of some modules/states is really dirty

same with ansible

and, when you have many jails, running a salt-minion in each of them consumes some CPU

kerneldove: really?

ya i've run into a few over the years

The problem with Saltstack is that they've been buyed by Vmware, now owned by Broadcom

ouch

development stalled for several months

broadcom is the more serious bug

haahaa I still have broadcom shudders from 20+ years ago....

and if we know anything about a configuration management company getting bought out... it's not great

hmm so seems like the package to install is py311-ansible to get ansible running. Will be a good learning experience, even if everything is on the same host :-)

rtprio: that depends on the company doing the buying. Ansible -> RedHat -> IBM vs Saltstack -> VMware -> Broadcom vs Puppet -> Perforce

I'll have to look at how to use Ansible to manage jails, will be a good way to learn ansible which we're moving to at work.

I've written a formula for Saltstack if you are interrested github.com/silenius/jails-formula

Nice, I'll take a look it

Thanks mage

mns: If you get into Ansible, you can also think about running your own git instance of some sort (Gitea here) and then driving Ansible with Gitea Actions, or the equivalent for whichever you choose.

"Your own git" to .... run post-receive hooks?

wavefunction: Specifically Actions, but essentially.

I haven't done enough with commit hooks in git to know just what the difference would be. I was a Subversion user up until a couple years ago.

But an example... I want to change something in my local DNS, so I make the change in my local check-out, commit, push, and my Gitea server then communicates with the appropriate Action Runner which pulls the appropriate repo(s) and runs ansible-playbook as needed, updating zone or config files and running the appropriate commands.

mns: well perforce totally pissed on the puppet community

I feel like Red Hat has done okay with Ansible. There's stuff that's been open-sourced that wasn't previously.

Like, github.com/ansible/awx for instance. That was proprietary before the acquisition, if I'm not mistaken.

mason: git-scm.com/docs/githooks the post-receive hook runs <whatever script/command you want> server-side after receiving a push

wavefunction: Cool, thank you.

(I get worried when complicated, third-party solutions are promoted when the native solution does the job, especially at smaller scales) :)

wavefunction: Understood and agreed.

wavefunction: On the other end, lately I've been managing some web stuff with bare denyCurrentBranch = updateInstead

mason: That is one of the goals, to setup a local git instance and use that. But that will be later, since I just have a single mini-pc running FreeBSD. I'll need to get some network storage first, then can work setting up a few services I want.

rtprio: I totally agree, perforce messed up a bunch of their aquisitions.

damn the heimdal to mit kerberos migration causes a lot of fallout right now *sigh*

(only relevant if you're following -current)

crest: What's happening?

well libcurl doesn't work because the kerberos libc in base changed their name

Ooh, that's a bit more dramatic than canonicalization issues.

But more easily fixed.

which means a lot of stuff is unhappy unless you go over the flag date in one step

but the package base repos haven't been updated since 2025-07-21

so ports and base packages for current are right now incompatible

I wonder what the legal situation is that allows this change now.

Is MIT Kerberos no longer export controlled or something?

mason: hasn't been for years

beacuse the crypto wars of the 1990s are over

*because

our base system heimdal has been very out of date

also it's still broken in main because of the libgss thing, D51601 is supposed to fix that

the real mess is that kerberos and openssl have to either both be from base or from ports

D51661 i mean

is that the story where some dude took a plane to sweden just to re-write the code so it's not export controlled and then flew back home?

because kerberos (doesn't matter if MIT or Heimdal) use ciphers from openssl

which is an unholy mess because PAM and NSS load modules via dlopen

so you can end up with a process that attempts to combine openssl from base and ports in a single process -> processes explode unless the versions happen to be compatible at the moment (by pure chance)

oh and for some reason libutils latest bump wasn't correctly tracked by make update-packages?!?

i'll run make packages and pkg upgrade -yf FreeBSD-base from my own packages

dumbbell: sorry to ping you directly, but is drm-kmod doing okay? far as i can tell on gh, you're the only one who worked on it in the last two months. my current gpu seems to be dying so i'm thinking of getting a 9070 and i'm trying to figure out how long it'd take for drm-kmod>=6.11 to happen.

mzar: possibly relevant to you: cgit.freebsd.org/src/commit/?id=7d9…69d40510b627f0790c2b9630b5903e926b9

ivy: thank you!