what's better, swap on zfs, or no swap?

(both are imperfect i already know)

buying/installing more RAM is best

not what i asked

Unix best practice is to have swap

ek Does Nextcloud depend on Apache?

I'm not sure how it is packaged on FreeBSD, but nexcloud itself can run with Nginx or Apache either one

I've finished installing nextcloud-php82 and the last lines after the installation says something about modifying httpd.conf however it seems Apache is not a dependency.

polyex, Use swap on a gmirror outside of zfs. Since zfs needs dynamic memory if swap is on zfs then there is the possibility of a deadlock. Not guaranteed. But possible.

As far as I know nothing has changed since: schmidp.com/2014/01/05/freebsd-10-does-swap-work-on-a-zvol

Upstream issue on it: openzfs/zfs #7734

Useful general background on swap in FreeBSD: klarasystems.com/articles/exploring-swap-on-freebsd

i'm following docs.freebsd.org/en/books/handbook/…ng-edge/#updating-src-obtaining-src and when i issue `git pull -C /usr/src` it says "error: unknown switch C"

If the output says fatal: not a git repository, the files there are missing or were installed with a different method. A new checkout of the source is required.

pretty sure you need to read down a little further and use the "new checkout" method listed if you don't have a prior pull on /usr/src already

freebsd 14 unattended bsdinstall throws up a "rebooting" gui when it's done. 13 didn't do that. how can disable? because now every bsdinstall takes 10 seconds longer out of nowhere. not documented in bsdinstall man page

polyex: you have taken up HOURS asking about how to shave off 10 seconds. It just seems a bit odd all the way around

just curious why it changed and there's no way to restore the old behavior

imagine all of the bsdinstalls going on adding up 10 wasted seconds

way more than the hours you're whining about

if there is that many, that you need to automate the process, it should just be unattended anyway

think of how many seconds you could shave off with faster drives, double the ram, and Alpine Linux ;)

doing the install to a raid 0 might also shave off some time

it IS unattended

that's what sucks. it used to finish then just stop, and the installerconfig needed to issue the reboot. it happened immediately

now it sits there for 10 wasted seconds

your snark is especially funny since you were wrong. "just use unattended install bro" when i already am

Yeah.. Seems part of the problem is I don't fully understand the situation. It is unattended, and use to reboot immediately. And now it still reboots, but after a 10 second pause, or this pause happens, and then YOU need to reboot it manually?

makes me wonder if there are flags/switches to the bsdinstall program to work through this

you don't even read what i type out you're wasting my time

Add the following lines to the file, replacing placeholders with your desired settings:

DISTRIBUTIONS="kernel.txz base.txz" # Packages to install

# Set root password (replace with a strong password)

ROOTPW="your_strong_password"

# Configure automatic reboot

halt_after_install="YES"

according to that, with FreeBSD 14, there is a flag for the automatic reboot option when the install finishes

i'll try half_after_install="HALT" if it works im surprised it's not documented in bsdinstall

Surprises me that the chat bot found something relevant then ;)

polyex: swap shouldn't be on zfs... it should be its own partition

ffs i know that. but what if your only choice is no swap or swap on zfs

SponiX looks like your "AI" hallucinated. there's no such thing as halt_after_install

;)

lol

damn

It talks so convincingly when giving the misinformation. It might as well be a politician

ya you make a great team

Yeah, the forum source it linked doesn't cover that option either. Completely just pull that out of its ass I guess

polyex: if you got a lot of memory why not do noswap?

polarian lamp or fixture?

Poster: what?

swap file...

you could put a swap file on zfs partition

dd if=/dev/zero of=/root/swap.8G.bin bs=1M count=8192

rennj: That's not yet considered safe, is it? Under memory pressure it's problematic.

echo 'swapfile="/root/swap.8G.bin"' >> /etc/rc.conf

so you swap file on the zfs encrypted partition

im just saying..rather then partition

example

winblows does that stupid does it not

rennj, As far as I know nothing has changed since: schmidp.com/2014/01/05/freebsd-10-does-swap-work-on-a-zvol

The first option is a no-go, because in case of low memory, ZFS needs memory to manage the disk writes to the swap file, but as there is no memory available it needs to write to the swap file, but ZFS needs memory to manage the disk writes to the swap file . . .

not sure of that

so oom killer in the end

out of memory

meh..take your chances i guess. know your limits

i know i can crash my system with just vmware and memory limits.

and i can disable the oom killer if i want..but still crash box.

i use zram cause linux sucks without swap...

a hammer and drill bit to you knee cap..give me the password!

a hammer and drill bit to your knee cap..give me the password!

you want security TPI en.wikipedia.org/wiki/Two-person_rule

2 teams never shelf meet or hang out at bar

2 teams never shell meet or hang out at bar...

team A and team B

half the combo

nuclear launch!

turn the key sir!

Per US Air Force Instruction (AFI) 91-104, "the two-person concept" is designed to prevent accidental or malicious launch of nuclear weapons by a single individual.

you want security build a data diode...

1 way ------>

so process the data incoming and then pass it on to next system

Is it needed to update bootcode and do "zpool upgrade" after upgrading from FreeBSD 13.2 to 13.3 RELEASE?

mage: -C is an option of git, not its pull subcommand

Hi. I'm looking into decomming my current old colo server, which runs ESXi with a boatload of Linux VMs, behind another VM running opnsense. I'm tempted to try moving everything to one physical FreeBSD server with a number of jails instead.

What kind of jails would you recommend for what are rather 'standard' services such as HTTP, SMTP, DNS, etc? A full jail is safest but looks like a lot of overhead; there's thin jails, and service jails too. Any downsides for those that I should be aware of that the docs themselves don't explicitly mention?

Alver iiuc it's more of a spectrum than a rigid "thin mode" vs "thick mode". are you planning on using zfs? if so what i do is just expand base media then clone it for each jail

Also looking at the docs of Bastille which looks nice, but curious what you think or would recommend instead. I have a load of UNIX experience but FreeBSD is minimal at best :)

so each fbsd install ends up taking 600mb or so

polyex: Yes, I would do ZFS (so I can add encryption, mirroring, compression and the likes)

polyex: I'm also curious how you keep those jails up to date - is there some central way of doing that for all jails or is it every jail as if it were an OS of its own?

haven't had to update them yet. tbh i'll prolly just redeploy rather than update in place

I meant for security patches/updates. OS upgrades is something I'll tackle when I get there

I like the idea of isolation in jails but if I would end up having a boatload of badly patched and out of sync jails that would not help :°)

With the VMs now it's mostly unattended, but even there it takes more effort than I would like, especially knowing how little stuff really runs in each

well just make a simple ansible task to run freebsd update on all of your machines (jails)

so ansible will ssh into each, run updates

Mmm. Then you have to keep ansible updated. Been there. :°)

But yeah, I guess it's manageable

a thin jail somehow lets you run multiple different jails from 1 base install but i don't know about that. then you just update 1 base and start all the jails back up

k so describe your ideal perfect setup

then i'll see if i got a suggestion

each on their own, but most update tools have a -j flag to operate on a given jail, so you can do it from the host, well, iirc with thin jails you just change the underlaying base for os/security advisories/erratas

As for network, would you recommend giving each jail its own public IP (which would mean managing pf on each jail) or plugging them on a loopback bridge and doing NAT via the host? Or a combination?

i use vnet and each jail getting its own public ip and pf instance

but describe me your ideal setup

polyex: I don't know yet, really. I just know that my current way of working - a type 1 hypervisor with a load of full VMs on top - is very inefficient, and a pain to manage.

is it just staying on top of updates that's the hassle or what else is the pain?

That indeed - especially keeping in mind my FreeBSD experience is minimal. A bit reluctant to bite off more than I can chew

Giving each jail its own IP could turn out costly - but that I can check.

In my current setup all of security/firewalling/routing is concentrated on the opnsense VM, which is nice, but also creates serious overhead due to the network load.

Alver: FreeBSD got bhyve as well

Alver ok sounds like you should go hard into the thin jail direction. and use 1 ip with 1 nat jail to all the other jails

go super super minimal and cheap

then maintenance will be simpler

Sounds good to me!

Do you use any specific tool to manage your jails - such as Bastion - or just the stock system tools?

no but the guy that made jailer.dev is here sometimes

sounds worth trying

manual is the only true way

then you know what is this about jails

angry_vincent: I can believe that, but I have to take feasibility into account. If such a tool can help speed up deployments by having good defaults and correct commands in 5 minutes, while it'd take me hours/days to read docs and get insight, that's worth something.

The time that I could spend that kind of time freely for my own education and amusement is long gone I'm afraid :°)

then you will not learn as much as doing manual deployment.

because all the guts are hidden in jail manager

I am sick of manual jail management.

got another set of old jails on some almost-to-be-decom server with no idea how to update them or else

Hmm, is there a way to hint to the kernel to be more conservative with ZFS ARC? Yes, I know that it will automatically release pages under memory pressure. In my case though, when memory needs to be freed for applications, the system really slows down to the point that it's getting super annoying now.

_xor: never got zfs to run nice here :( still have vfs.zfs.arc_max=524288000 vfs.zfs.arc_min=104857600 somewhere in /boot/loader.conf...

I have memory stats that I get from vmstat and show on my DE bar. It'll usually hit 90%+ memory utilization (wired + active + inactive + laundry) within a few hours of booting. Last night I rebooted my system and left it with nothing running except my desktop environment and it was showing around 7% total memory utilization. Woke up this morning

and it's showing 97% utilization.

Checked with sysutils/zfs-stats, and sure enough, ARC is taking up 20GB out of 32GB total. Normally that would be a good thing, but my issue is that when I'm using Firefox and opening tabs, the system REALLY starts to lag as it scrambles to release/re-allocate memory.

I know I could set an upper limit on ARC size, but I'm wondering if there are any other better options before I try that.

saper: Ah, so you had to limit it :/

saper: what do you use for management?

vfs.zfs.arc.sys_free looks interesting.

saper: Apparently vfs.zfs.arc_max and vfs.zfs.arc_min are now vfs.zfs.arc.max and vfs.zfs.arc.min

Alver: used to be ezjail, at least to set them up

_xor: thanks for the pointer, looks like my ARC is quite small 36.98% 184.91 MiB

_xor was taking a look at freebsd mastery - advanced zfs book, the cache restriction chapter talks about arc min/max and arc_free_target which specifies the number of pages the arc should leave free for other processes

zeylos: Yup, just noticed that one in the sysctl list. Going to take a look.

For now I just limited ARC to 1gb-4gb on a 32gb system. Yes, it's low, but I don't mind it having to I/O thrash a bit. Will bump up max as necessary, or ideally, utilize those target-setting sysctls.

fwiw, I use Bastille because it is all shell scripts, so it generally Just Works but I can also make modifications or see how it all works manually by looking at the scripts. good learning tool for me

^^ Alver

scoobybejesus: thanks! Bastille does look nice from what I can tell.

I think I'm already convinced about using FreeBSD in the meantime. All alternatives just feel... meh. It's been a good few years since I did something new for infra. :°)

Going to think it over on how to do network and disk layout.

I take it that the installer can do a mirrored boot/root ZFS?

when i started using it, it didn't have the rename subcommand as part of the release. but it was in master on github. i only have the package, but i could still create rename.sh in /usr/local/share/bastille/ or whatever, and add that command in /usr/local/bin/bastille... and then it worked! and things like that

the installer technically only mirrors the root partition, i believe. i don't think it writes to the EFI partition on both disks automatically

Changed from ezjail to bastille recently. Bastille looks like the obvious choice atm. Had consistent good experience with ezjail along the years. I hope the same with bastille

Ah, good to know.

Will have to look up a procedure to do the mirroring for the EFI stuff etc.

Yeah Bastille looks better than ezjail to interact, changing or adding scripts

I have a etcupdate endeavour in Bastille and some /etc 'get version ' dilemmas :)

DanDare: mind a PM? It's about the bot

:)

Sario, it' ok. No problem.

etcupdate in Bastille thin jails is an unsolved problem

i just wanted to refresh my memory about how my system boots, and i ran efibootmgr -v, and apparently after a sysctl -a | grep bootmethod i am now BIOS booting, where previously i was UEFI. it seems manually copying the EFI bootcode has caused something funky to happen

My ThinkPad x270 just refused to suspend to S3 after rebooting into 14.1-RELEASE-p1. Did anyone experience such problems?

Unfortunately I could not spend any time debugging, since I needed to catch a train. :/

zeylos: need to check if ZFS Master is included in the latest dump of e-books offered on Kickstarter for those supporting mwl's new book about running the mail server.

okay, i'm crazy. i was running efibootmgr -v from the vps. after i then saw gpart show say my freebsd-zfs partition was 58G and zpool status showed one disk and no errors, i realized i was being stupid

Hello

I need a libxo expert

I think there's a bug in xo(1)

here's what I'm running: xo -J "{l:ipv4} {l:ipv6}\n" "1.1.1.1" "ff::01"

here's what I'm getting: "ipv4": ["1.1.1.1","ff::01"

instead I should be getting "ipv4": ["1.1.1.1"], ipv6: ["ff::01"

has anyone seen this before?

scoobybejesus, That's the type of thing I would do to myself too. We have all done it at one time or another.

saper: the 2 zfs mastery books are really good imo, but I think the kickstarter thing ended mid-June ? or maybe I got it wrong

anyone update their snmpd lately?

rtprio: Mine is up-to-date.

ek: did you move your snmpd.conf to share or leave it in etc ?

rtprio: My config file is still in the ${LOCALBASE}/etc directory.

However, I specify the config location in /etc/rc.conf.

zeylos: yes, but I have signed up and ordered the book.

oh - sorry misunderstood the first statement. Hope they are in then :)

"Sudo Master", "Ed Mastery", "PAM Mastery", "Networking for System Administrators", "$ git commit murder", "SSH Mastery", "Tarsnap Mastery" and of course "Run Your Own Mail Server" once it is out. Nice set anyway!

this JSON output is going to kill me

What's wrong with json?

I like it too, it's just a nightmare to generate it

Beats the hell out of XML or CSV.

(properly)

I honestly prefer XML

With what language?

at least it's typed

CrtxReavr shell and libxo on FreeBSD

libxo?

yes, FreeBSD has a utility for cross-output and a utility named 'xo'

I've never heard of it, so I'm guessing jq is better documented and easier to work with.

jq is for parsing, not outputting

I've done most of my JSON munging with Python.

yeah, too bad not part of base

which shell?

try this for example : xo --style test "my name is {:name}" CrtxReavr

and then run : xo --style json "my name is {:name}" CrtxReavr

sorry, text, not test. LOL

kona FreeBSD's /bin/sh

jq will absolutely create json.

I had no idea

but still, not part of base

(yet? :P)

Not sure how it's licensed. ..

good question

But you could install a package as part of your process.

Licenses : MIT

xo -J --wrap my/name/is "{:name}" moo

what are you transforming to json?

kona I have a jail manager named Jailer, I'm doing `jailer info -j` for JSON output

and I want it to be proper JSON, like "ipv4": []

if there are no IPv4s

instead of "ipv4": "-"

which is what FreeBSD returns :P

(jls, that is)

so yeah

how is xo not helping?

oh. it already emits json?

well, xo is helping, I just have to do so many conditioning, if jls -j jname ipv4.addrs = "-"; then print [], etc etc

oh jls does emit JSON, just not a "proper" one

like if you have 2 IP addresses, it will gladly do "ipv4": "ip1,ip2"

instead of a list

ok, so it emits "JSON" but you have to find certain patterns and reformat them?

yup.

and then I have to figure out if it's VNET Jail or not

and sometimes a jail might NOT have networking at all

etc etc

ok, you having to do a lot of exceptions and reformat with xo, but I wonder if you could come at it from a different angle and maybe: jailer-info -j | awk -f make-real-json.awk

and then code up your exception patterns in awk?

that... sounds like a great idea

I can also use flua (FreeBSD's lua) as well

could do. some version of awk should be in base. :)

One True AWK™

:)

It seems my x270 fails to suspend with the newest boot environment, but it works with the previous. Am unsure of whether the kernel makes a difference. Will make a few more attempts to verify I've gotten the right picture.

In case it is boot environment dependant, how would one best debug further? Boot environment are purely ZFS snapshots, right?

i guess, depending on the complexity of the diff between the jailer-info -j output and proper json, you might be able to get away with sed instaead, but i feel like awk is likely to be more maintainable

|cos|, Boot Environments are clones of a snapshot. Because snapshots are read-only. So a clone is made so that the clone is read-write.

You should be able to boot into the previous Boot Environment and do whatever testing you want there.

You should also be able to boot the current boot environment with the previous kernel and test that combination too.

I'm familiar with beadm on illumos, and it seems to exist in FreeBSD ports. Is that the tool to choose, or would some alternative be to prefer?

Either beadm or bectl are fine for this purpose. beadm is in ports and was introduced as similar to the Sun tool. bectl is a newer rewrite in base and therefore will always be in base and available. Both are okay to use.

In order to boot to the previous environment you don't need to use either one. Just reboot and select the previous boot environment from the boot menu.

is it normal for the previous boot images to stay mounted?

rtprio, No. They are not mounted for me.

Are you sure they are mounted? They exist so zfs list will show them. But they are not mounted.

oh.... but it's not a snapshot

maybe that's what i was getting confused

snapshots are read-only and not suitable for a root file system.

previous boot image should not be mounted, just the image you are using now should be alive

clone a snapshot into a writable live file system "branch" and then boot it and that's a boot environment

rtprio, Look at the output of this command: zfs list -ro name,canmount,mounted,mountpoint zroot/ROOT

rwp: ah. bectl seems like the more bsd-like name. i should had guessed that.. thanks!

That does sound like evidence of a regression.

Can you verify that the problem is isolated to the kernel? Booting the previous kernel with the current everything else works but just the new kernel fails?

This is the origin of Boot Environments in FreeBSD, based upon those previously in Sun, forums.freebsd.org/threads/howto-freebsd-zfs-madness.31662

The sun utility was called beadm so this one is called beadm. Submitted into ports. Then later it was rewritten to be included in base and named bectl to avoid the name collision. Both continue to be maintained to the present time.

One Sun bit of documentation that can be read about Boot Environments is this: docs.oracle.com/cd/E19963-01/pdf/820-6565.pdf

That's mostly of historical academic interest to use in FreeBSD since that's for Sun and we have FreeBSD's bectl+beadm utilities which are similar but not completely identical.

My first suspect was the kernel, but I realized that using an older bootenvironment appears to be the factor. Have kept booting the older kernel consistently.

That would be surprising!

Indeed! But on the other hand the snapshot/clone is rather huge. Over a GiB.

I can't figure out how to use zfs-diff though. The zfs dataset names appear to not work, not being snapshots. And the snapshots fail due to not being mounted.

Maybe I'll need to catch some sleep and get back to this in the morning, rather than now after coming straight from the pub.

zfs-diff will need to use the underlying snapshots.

zfs list -t snapshot -r zroot/ROOT

Those are the snapshots that were used to clone into the live filesystems.

So at a guess for you: zfs diff zroot/ROOT/default@2024-06-03_19:05:00-0 zroot/ROOT/default@2024-06-20_09:09:22-0

However I am not sure how helpful that by itself will be since there will be many files in that listing.

That's the exact line I tries, but get "Cannot diff an unmounted snapshot: operation not applicable to datasets of this type"

Hmm... For example this exact command works okay for me here (different timestamps of course): zfs diff zroot/ROOT/default@2024-06-21-15:59:20-0 zroot/ROOT/default@2024-06-21-16:00:24-0

And produces 900 lines of diff output of file level changes.

In any case if you have a working case and a not working case later then that sounds like a definite regression failure. Perhaps a good time to open a bug ticket on it?

The error message for zfs diff is mentioned here, but I'm not getting wiser. openzfs/zfs #3325

A good nights sleep and opening a bug ticket in the morning seems like what I should do.

Thanks for answering my confused questions.

I'm finally happy with Jailer's output: antranigv.am/misc/jailer-0.1.3-dev.png

looking good

antranigv jailer.dev should have something in the top section about why someone would use it instead of base freebsd commands

polyex because people will end up creating jailer for their own anyway :P

don't tell me, add a section to site

I'll do the marketing thingies as soon as I reach v0.1.5, but yes, you are right

and be detailed about the advantages