It would be a whole lot better if it it managed a standard FreeBSD configuration IMO. Then you could choose to use the web UI, or the shell. Even let the web UI manage some things, and the shell other things. Instead it is built in such a way that it doesn't work like a standard install.

alepzi: it's not transparent though. It doesn't use /etc/rc.conf. Neither does it use /etc/pf.conf. So I don't think you'd learn much about how FreeBSD works under the hood. You'd learn how pfSense is implemented at best.

well ya that's what i'm saying. no black boxes, just easy configs of a regular freebsd for the noob to get from 0 to web gui firewall in a jiffy. obviate shit like pfsense and netgate

There's probably a market for that.

it's not reolly obviating it

jst another competitor

not if it's all open and has easy setup and links to hardware that's known to be compatible, ready to go. maybe that's what opensense is?

You can buy routers with pfSense preinstalled and ready to go.

to my knowledge pfs also ship an iso

does freebsd-update -b basedir/ fetch install need to be run as root or smth? it fails with dir does not exist or is not writable: /var/db/freebsd-update

fwiw /var/db/freebsd-update is root:wheel and drwx----------

a non-root update?

ya i'm setting up a jail dir

don't want to do anything to the host system

can't run that inside jail?

that would make no host changes at all

trying to make it to a scripted bsdinstall base

so when i install the host in a vm, its jails are already there and ready to go

hmm

well it has -d

is that it?

that seems to run, but then it says at the end you must be root to run this

so not sure

it says 13.3 p1 is newest and doesn't need to be updated

is that latest?

alepzi: latest release of 13 series?

the answer is yes.

alepzi: you are running 13.3 and want to know if there recent patches available? I believe there is none. maybe you are looking for newer production releases? Please describe your objective.

check the rel engineering for details how the software releases are structured: freebsd.org/releng

Title: Release Engineering Information | The FreeBSD Project

This isn't true. There's 13.3-p1, for freebsd.org/security/advisories/FreeBSD-EN-24:06.wireguard.asc and freebsd.org/security/advisories/FreeBSD-EN-24:07.clang.asc.

V_PauAmma_V: I am confused by what you mean 'isnt true'. He said exactly 13.3 p1 is the newest. I understand this is correct statement. Please clarify if that is latest or any newer.

V_PauAmma_V: Is there a patch over 13.3-p1?

shouldn't be, my wireguard EN just landed like, last week

NO, you're right I just didn't read up after "recent patches" - which I think -p1 is, being 4 or 5 days old.

My fault.

s/after/past/

V_PauAmma_V: np, many thanks for the update!

I'm kinda liking my "inception" setup as a work around to PR 278058 (I'd like no bug better, lol)

278058 – Simultaneous use of Bhyve AND vnet on network PCI devices causes network failure bugs.freebsd.org/bugzilla/show_bug.cgi?id=278058

When going from freebsd13 to 14, we're seeing a ~10% drop in throughput on a simple nginx (tls) server benchmark. I have been warned that some kernel work might do this to us, since we use "wrapped" jails - an outer vnet+epair jail and an inner "classic" jail.

Are there any "glaringly obvious" things we should change in nginx config to reap benefits of 14 and perhaps offset this regression, before I start complaining to kernel devs? :)

Ltning: maybe find out what the benchmak is using as an indicator of throughput? The use of a benchmark to determine performance is very specific to that benchmark and may not reflect reality.

and from what i am reading, youa re doing a freebsd 13 -> freebsd 14 host which contains a jail that has another jail in it?

are those jails still on 13 or they have been ugprade to 14?

The jails are 13, but I was warned that there are kernel things that are already suspects.

But not sure if userland being 13 should have any impact here?

Also well aware of the synthetic nature of benchmarks. That's both their weakness and their strength.

As opposed to them having a natural nature. Man I can't even imagine such a thing.

Ltning: that is possible, unless yyou have a specific situation. that will be a needle in a haystack situation

is this machine in production? and being used to serve customers?

i may be wrong in this approach but jumping from nginx all the way to the kernel being suspect, is a big leap.. in my head

if you need to build virtual machine images with the latest cloud-init, here's my repo: pkg.igalic.co

Title: Repository for net/cloud-init

xulfer: It's called a manned botnet ;)

voy4g3r2: Our production machines are still on 13, we're testing 14 on comparable hardware to see what we can expect. The reason I "blame" the kernel is *very* specific remarks from kernel people after my talk at EuroBSDCon last year where I detailed our "wrapped jails" concept. Apparently some optimisations that might backfire in our scenario.

And since we - despite being warned - didn't manage to get our act together and test this before/during the -BETA phase, I hardly find it fair to complain about that right now. Hence my hoping someone knew low-hanging nginx-optimization-for-14-fruits. :)

On the positive side, nothing seems to *break*, and it's still only ~10%, so it's entirely serviceable. And I suspect other concurrency-related improvements in 14 can help offset even that, since we're running "a handful" of such wrapped jails on each host, and Java code (apache-tomcat) is also known to be significantly faster on 14.

Ltning, If you were to run a benchmark program such as "iperf" to be a specific network I/O benchmark only (removing the rest of nginx) in both the old and the new do you see 10% as well?

And regardless if the problem is throughput then using a throughput specific benchmark would be a useful way to make a reproducer for a benchmark problem report ticket.

rwp: Yes, that is on the list of tests to run.

I could easily see that an I/O benchmark might be more than 10% by a significant amount if it avoids the rest of nginx.

On a not-related-at-all-note - seems like both electron28 and signal-desktop has disappeared (again). I cannot overstate how much I hate everything electron.

Ltning, Almost all of the new-school rolling release rapid turn massive churn projects have routine instability problems.

i'm setting up a jail and part of the process is to retrieve base.txz and expand it. question, if i already have the freebsd git repo cloned, is there any way to use that and skip the fetch, expand, update step pls?

alepzi: man 8 jail -- "Setting up a Jail Directory Tree"; you build and install userland in a specific destination directory. Not how I would do it, but if you want to start from a cloned git repo (which is source code only), then that's one way of going about it.

it is possible!!

tyvm

btw why is that not how you'd do it?

It'll take forever the first time, since it's building the whole system. If you do that, you want to add -j $(sysctl -n hw.ncpu) to the 'make world' step: "nice make -j $(sysctl -n hw.ncpu) world DESTDIR=$D"

...or something like that. (sysctl -n hw.ncpu simply spits out the number of CPUs you have; that number can of course be specified directly instead, and depends on how much of your CPU you want to spend building :)

I'd rather download base.txz and expand it into some template-jail directory, and then "mount_nullfs -o ro <template>/<dir> <jail>/<dir>"

that's thin jail type tho right? i'm doing thick jail atm

Yea, so for thick, just blow up base.txz inside your jail

You'll still have to create some files, resolv.conf and whatnot (it's been a while since I did this manually)

ya, could, but i already have the git repo so i'd rather skip the networking

That's up to you :) Depends what's more expensive, cpu or network

A third option would be to use pkgbase

Would allow for (much!) easier jail upgrades down the road

how would that work?

But that would involve more network

Well you pkg install the base os packages into your jail (don't ask me exactly how), then later you can pkg upgrade the same way

alepzi, Most people just untar /usr/freebsd-dist/base.txz which is all local, no networking, and about as fast as any tradeoff of having a bundled "thing" to untar.

Where did /usr/freebsd-dist come from? :D

Must be a *really* long time since I installed anything with bsdinstall ..

freebsd-update keeps it updated

wat

But it is just a base tar bundle that anyone can create themselves.

i don't have /usr/freebsd-dist

And if I were doing this after a bunch of customization then I would tar up my own customized template.

Well, there you go alepzi -- there are as many ways of doing this as there are non-lurkers in this channel at the moment. At least. :D

very cool :)

i build my systems from the git repo to learn about how releases and stuff work

i never run freebsd-update because i guess it can't handle installs that aren't from a RELEASE

sadly

Yea, that's one of the (many) reasons I got my employer to throw a *lot* of money at pkgbase. It will make it *sooo* much easier to follow snapshot releases, for example.

tyvm for that. pkgbase looks like a great enhancement

pls keep pushing my dude

I am :D

<3

another reason i don't like the typical install path is update (possible security fixes) comes after an install. so a new box can be installed and temporarily vulnerable

Yet another possible way is to keep a live dataset, snapshot it, then zfs send | recv it to your new jail dataset. It's an alternative tar but the template is unpacked if you are routinely changing it.

but if i build from git repo after a pull, i always build and run the latest patch level from 0 moment

nice

Or you could also avoid the tar and zfs send > file, compressed, and then send that captured zfs-send image to recv when creating a new dataset.

Tried to be sneaky about it. pkgbase has been "nearly there" for almost a decade (perhaps exaggerating a bit here..), so I thought if we can push it over the finish line, it'll stick around and people will use it -> people will finish it ;)

yes

can we make it the default starting with 15.x?

I do like the half-thin approach I normally use.. Since the base OS should never be touched, I mount all the binary directories from base into each jail, read-only, but have local /usr/local, /etc, /var, /home

immutable base is a really good idea for sec and keeping systems from clobber right?

Yeah, absoluteyl

fuck ya

absolutely*

bin lib libexec sbin usr/share usr/bin usr/include usr/lib usr/libdata usr/libexec usr/sbin -- those are the dirs you want to "nullfs ro" mount in the jail-specific fstab

I always keep a symlink around pointing to the "current" template dataset, and use that as mount source. Then I can create a new one when there's an upgrade, rewrite the symlink, and restart the jail.

For minor updates, just update the template in-place

If you want real fancy, you can use a zfs snapshot as the source mountpoint

(.../jails/template/.zfs/snapshots/<last-known-good>/...)

mm nice

(or symlink to that, so you can "upgrade" by pointing to a different snapshot and restarting the jail)

Ltning: I recall comments about being able to boot freebsd from a snapshot being made at eurobsdcon

o/

Ltning: in practice, have you have any issues with various packages leaking out of /usr/local/* etc?

for your half-thin jails

dch: That's *extremely* rare.

I can't think of an example right now.

Ltning: thats kinda what I expected in practice...

I mean if that were to happen I'd promptly raise a bug

Well.. that's a lie. There are some packages that mess around in /etc and /var, but at least it doesn't prevent the fed-but-not-full jails

extracting base.txz into a jail dir was 540M. but then building world into the jail dir from the freebsd git repo was 1.5G

so how can i make my own base.txz from the git repo? so i save 1G per thick jail

alepzi: what are you trying to get rid of?

1GB

Building world builds all the stuff that is in the other distfiles, too

Or rather, 'make distribution' does

You might find /usr/src in the destination, I don't know ..

Certainly lots of docs and such that wouldn't be in base.txz

meena: the normal freebsd install process is insecure, so i build my systems from the git repo with a fresh pull off of the x.y branch. trying to get a process for setting up jails the same way. instead of downloading base.txz and expanding then freebsd-update fetch install, i wanna build the base.txz from the git repo

alepzi: if the normal process is insecure, we need to fix it

With pkgbase I presume it would not be.

that would involve releasing a .iso for every patch level

No more so than fetching an iso or sources from git

alepzi: more people more hardware

ya so that isn't something i can solve. so my workaround is building my own systems from the git repo

So it should be like a Debian netinstall process. Where it pulls down all the latest packages within the install process. So you have an up to date (by Debian Standards) system right out of the box?

i guess but what if i have 100 boxes, that's tons of network. or if i'm working offline

i just really like being able to build my own images from the git repo

Then that's what you should be doing

well ya i am

fair enough

but now i wanna figure out how to build base.txz from the git repo, so i can set up jails from that

But the nice thing with pkg is that you can run it through e.g. a caching proxy, which is what we sometimes do even with freebsd-update when we need to upgrade a lot of boxes and don't want to wait (and wait and wait)

alepzi: did you read the handbook?

there's a page on it!?

And for packages, we have a shared pkg cache on the host and share it with all the jails (also nullfs, but read-write)

Title: Chapter 26. Updating and Upgrading FreeBSD | FreeBSD Documentation Portal

alepzi: if we don't have documentation on it, that's a bug

that document, that meena mentioned is solid.. i use it to build current every so many days

which section says how to build base.txz from repo?

make world does that

you sure? i just did a make world DESTDIR=... and searching in that dir for base.txz it's not there

voy4g3r2: i think they mean: which make target, exactly, produces base.txz, specifically

ya

do i really have to go read Makefile.inc1?

did i do something wrong?

no

21:17 <meena> alepzi: if we don't have documentation on it, that's a bug

here's my user feedback: every artifact that users directly interact with, like release .iso/.img/etc, or base.txz (setting up jails) should have step by step guide to build it myself from source. just MY opinion

alepzi: No argument from me.

alepzi: a good opportunity, as you are doing this, to write it down and share :)

according to release(7) it's make packagesystem

but i'm writing it down here as i encounter roadblocks no? not being argumentative just trying to understand the advice

i didn't have to read that terrible, terrible file

alepzi: none of that taken, just saying, i also like to browse thsi: wiki.freebsd.org

Title: FrontPage - FreeBSD Wiki

Title: Jails - FreeBSD Wiki

this has some cool info, i have used myself

alepzi: But as I said before, the trouble and the charm is that there are many ways to achieve what on the surface sounds like a simple thing (creating a jail), and you're already far along in being specific about *your* needs, so there may or may not be a prescription-based way of achieving that.

speaking of building, time to upgrade current bhyve image.. i get to do a round of man pages updates :)

It's among the reasons why I do agree we need better jail management tools in base, but I'm not going to be the one to say what they should do and how because they are unlikely to suit my needs anyway. Does not mean they shouldn't exist, as long as they don't remove the flexibility I've enjoyed for 20+ years.

jail management tools are great, once you understand how things are setup

meena: so to make the base.txz i run sudo make -C release packagesystem from source/ ?

alepzi: honestly, no idea. I've been on the PkgBase train since… 2019?

long time

2021

time has no meaning

well i tried it but it immediately said packagesystem is up to date *shrug*

alepzi: did you build world first?

i'm pretty, and sure, you need to build world before you can build release

They built 'distribution' and 'release', apparently?

ill try sudo make buildworld, then sudo make -C release packagesystem

looks like i gotta cd to release from within source/ to run the 2nd command

ill try that

didn't work

sudo make -C release packagesystem shit out packagesystem is up to date

oh base.txz is in /usr/obj/..../amd64.amd64/release/

so maybe buildworld creates it already. i'm gonna clean and rebuild and see

is something like nfs the best way to share host storage with a bhyve guest?

afaik you can't pass sata drives directly to a guest

ok make buildworld buildkernel doesn't make base.txz. but sudo make -C release packagesystem DOES

meena: ^

ok so now i know how to go from git repo/source to base.txz that i can then expand into a jail's dir and NOT need to freebsd-update fetch install

tyvm

alepzi: cool

what options do you have set?

markmcb: for a Linux guest, use 9pfs. for a FreeBSD guest, NFS is probably the best way. you can passthrough an entire SATA (or SAS) controller or an individual NVMe drive, but not an individual SATA drive

none i don't think? the exact commands i ran meena were: (from freebsd-src/ the source dir) sudo make -j24 KERNCONF=GENERIC buildworld buildkernel; sudo make -C release packagesystem

so how do you expect to get a smaller base?

i guess that's a next step

well base.txz is only 205MB so that's the same as if i network fetched it. that's fine for me

what i didn't want was what man 8 jail recommended, make world; make distribution, which made the jail dir 1.5G instead of 500mb

lw: thanks