crest: Heh, funny you mention that. I'm literally working on some code right now for my nomad plugin.

what will it do xor?

_xor: which plugin is that?

crest: I wanted to originally use OCI to package and distribute jail images, but OCI isn't very ZFS friendly (yet). It expects tarballed layers with whiteouts.

did you write the jail task driver or the pot one?

crest btw which firewall works best with fbsd bhyve and jails? pf or ipfw?

crest: Neither, both of those are written in Go. Mine is written in Rust.

_xor: github.com/michael-yuji/xc/tree/main/ocitar ?

Title: xc/ocitar at main · michael-yuji/xc · GitHub

is that you?

Also, from what I saw, those drivers didn't support netgraph.

Nope, but I did recently see that project. In fact, the one and only issue on the repo is from me heh.

why isnt oci zfs friendly?

That's me.

Title: Questions · Issue #1 · michael-yuji/xc · GitHub

1 – [aha] [scsi] Toshiba MK156FB scsi drive does not work with 2.0 kernel bugs.freebsd.org/bugzilla/show_bug.cgi?id=1

depends on your definition of zfs friendly

OCI is fairly Linux-centric (though has generic mechanisms like annotations) and expects tarballs for its layers.

Would have to figure out how to specify a base image + incremental snapshots to apply, though that shouldn't be too terribly difficult.

"in fact ipfw seems to be a better choice than pf for a lot of the things I'm interested to do"

wow just what i was thinking about

i've been a pf user and it's fine but i wondered if ipfw is better now that i'm getting deeper into fbsd

How come?

imo ipfw is better suited to automation and its more expressive tables can vastly reduce the number of rules

but pf has the more human friendly syntax and pfsync in its favour

What are the trade-offs?

ah

and ipfw tends to scale better to higher core counts

I've always heavily used the tables, which is probably why I prefer it

HashiCorp just went BSL for a lot of their projects. Doesn't impact that much...yet, but it might dim the outlook a bit :/

BSL?

Really? Because it's userland?

RhodiumToad: Business Source License

i like to think of PF as a ready to use firewall and IPFW as firewall construction kit

Title: HashiCorp Switches to BSL License | Hacker News

Whatever happened to ipf?

_xor: died over a decade ago, but nobody buried the corpse

lol

why? they're kernel modules. keeping the modules doesn't hurt

those options remove the userland programs etc.

Man, that was annoying.

Ordered a bundle of CAT6 + RJ45 ends. Turns out, the CAT6 I got is 23 gauge while the RJ45 heads were 24+ gauge only. Had to order new RJ45 ends for 23 gauge.

ya i really wanna see pkgbase and other ways to cut down fbsd to only what's needed

i might try ipfw too

pkgbase will do that. It does for me. You can thank meena.

crest what does ipfw do better for automation?

ty meena!!

you can put the structure into the ruleset and later populate the tables

I've done a fair bit of stuff where ips are automatically added and removed from tables

to express almost all things you want to automate

pf has tables like that too no? and anchors i think they're called?

much easier to manage those than mess directly with the ruleset

pf tables are just sets of ip addresses

ya

these are more?

ipfw tables can match more fields in the packet (protocol, port, etc.) and map them to values

with up to one value per type per entry

e.g. a rule number and an interface

That's one of my biggest gripes with pf, handling dynamic management of rules.

what pf does have are (nested) anchors

Yeah, I use those with nomad (kind of).

does ipfw have something comparable to nested anchors?

and you can feed it the rules for anchors from a pipe

polyex: often times it doesn't need them to accomplish the same

and if your do need something there are sets of rules

(iirc up to 31 sets)

I run nomad as an unprivileged user and so my driver needs to add a mapping for ports <1024 (assuming it's running on the host and not within a jail).

ah ok. just wanted to make sure i could SWITCH to ipfw, not start having to use ipfw AND pf

I do that with nested anchors + rdr rules.

_xor: there is a mac module to allow this

mac_portacl

is there anything pf can do that ipfw can't do and do well?

you also have change the low port range for it to take effect

but if you want to you set it to stop at e.g. port 22

mac_portacl(4) I mean

Well, I mean I'd rather not have any of that at all and just let it just use random unprivileged ports within a specific range.

But for a narrow set of services that I'm still using nomad to deploy, I need to be able to bind those to privileged ports (e.g. Traefik for tcp/80 + tcp/443, etc).

I mean I guess I could also set it up with ng_nat, but all of that is a hassle.

why go through netgraph?!?

Because I'm using it currently for my jails?

ipfw can do local port mappings without nat

Lol, on a side note, I just noticed this while looking at CAD packages in ports...

"The BRL-CAD source code repository is the oldest known public version-controlled codebase in the world that's still under active development, dating back to 1983-12-16 00:10:31 UTC."

if they mean that they still have a repo going back that far, they could well be right

is there anything pf can do that ipfw can't do and do well?

New to Pouderie... after some tinkering and learning, I built a dedicated box (VM) and gave it a lot of resources... but it doesnt seem to be using them, I get slow builds and low resource utilization

Any tips on optimizing a box for pouderirie?

(I speak French and for the life of me I can't spell that word)

haha

sorry, can't answer that question

(The spelling or the optimization? Haha)

what settings have you changed from defaults?

and how much RAM do you have?

and how many CPUs?

RhodiumToad, I didn't change much, just the required stuff, ZFS pool

it's worth noting that poudriere can often be very disk-bound, unless you use tmpfs for everything, which requires a shedload of ram

12 CPU cores, 24gb

have you looked at what is going on during a build?

specifically disk i/o bottlenecks as well as cpu or memory

I'll check disk IO now

Unrelated, does 14.0 not have a pam.d/auth or pam.d/account?

SpaceBass: mine doesn't

Thanks for checking - just wanted to make sure

Next challenge... install python 11

3.11

there's a package (or port) for that

Yeah installed it... but 3.9 was default and now I have to get it all sorted

Trying to get certbot to work in 14.0-CURRENT and it wants 3.9... but it needs a newer version (3.10 +) of cryptotography

even if you install 3.11 first?

Hello there

I'm replacing disk in a raid pool, it starts very quickly but now the system response very slowly

zfs?

Forget to notice ZFS 2way mirror

yes

211G resilvered, 63.99% done, 01:39:04 to go

the progress is about 1% in 15-20 minutes

What will happen if I restart machine during resilver / replace

the strange thing is that the new drive ( seagate barracuda ) is slow to response for smartctl queries

it should pick up where it left off

gstat ops is almost 0

is it an SMR drive?

queue - 2 .. 16 ...

SMR?

shingled magnetic recording

gives a significant increase in storage density at the expense of having the write performance go completely to shit on a regular basis

Seagate BarraCuda 3.5 (SMR)

smr...

yeh

basically, large portions of the disk area can be written only in large sequential blocks

so writes accumulate on temporary areas of the disk and then have to be rewritten to the shingled zones, which takes a long time

So the only option is to wait resilver to complete

yup

:(

if i script bsdinstall, what does bsdinstall do with that script to make a bootable OS? and can we just do whatever bsdinstall does and make images that can be preconfigured then directly booted and run? like skipping any manual install or setup

RhodiumToad, thank you for quick reply

polyex, create partition, write bootcode

RhodiumToad, unfortunately this is a first experience to me

polyex: sure, making a bootable system without bsdinstall isn't hard, especially if you know the disk setup in advance

never had SMR drive

what's the basic process like?

starting from a fresh disk: gpart create, gpart add (possibly several times), gpart bootcode

gpart destroy -F /dev/disk # cleanup entire disk

then newfs or zpool init

^

if it's EFI, then preparing the ESP is a few more steps if you don't want to do the hackjob that's used for bootable images

(newfs_msdos and a few file copies, nothing too tricky)

then installing the OS can be done by unpacking the distribution txz files, or by make installworld distribution installkernel from a source tree

then set up whatever config files you need

(if not using zfs, remember to create /etc/fstab, that's the one that usually bites me when I do manual image creation)

RhodiumToad, fstab is required anyway, swap partition things

polyex, do not forget to create extra swap partition at least 16GB or better 2xRAM

hmm i wonder how to automate all of that

well, that's a large part of bsdinstall's job :-)

but otherwise it's just a matter of scripting

polyex, here is my old script

Title: dpaste/4JQe (Bash)

this will partition the disk for you

it is very old, dont remember what I did back then

that looks like it supports both EFI and legacy boot, usually not needed these days

(though sometimes is)

polyex: you know you can script bsdinstall, right?

yes, but I did this script basically to restore ZFS images back to the system

ya

universal is good :)

say you name script: "partitiondisk.sh", you can create config file like: "partitiondisk.cfg" in the same location

here is the sample of the config: bsd.to/2GNP

Title: dpaste/2GNP (Bash)

polyex, the installation things is quite easy, just download and extract files into root directory of newly created partition

base.txz, kernel.txz, lib32.txz, src.txz

src is optional, but I do like to have them, need to compile something time to tile

prep disk, extract files, copy in config files, boot?

yep

that easy

I love BSD approach - all the configs are text, all the logs a re text

any way to bundle all of that up, even the disk prep stuff, so you can do it all in like 1 step?

exactly

yes, that's literally bsdinstall :-)

^^^

bsdinstall doing BEADM things, which I do not like

RhodiumToad, Thank you so much! You saved me a day. ANd a hard talk with a sales manager in a shop

Im struggling a lot with pam.d and winbind auth... log says user is successfully authenticated and then pam_winbind(sshd): PAM_ESTABLISH_CRED not implemented

cd

what does BEADM things mean?

Before "beadm" was the precursor of "bectl". So I would say the way that command works or is used to create boot environment

s/Before//

oh and nerozero said bsdinstall does beadm stuff, so it needs to be updated to use bectl?

bectl is the successor to beadm

Cannot say what nerozero actually meant

(oh parv said that already, sorry)

ty!

polyex: vermadem is the author of BEADM, he is active on twitterX and on his blog

... on Mastodon: vermaden⊙bc

how do i get local_unbound to listen on the private network IP, as well as localhost?

it would no longer be local_ if it listed on the network ip

I want jails on the host to use it, but they can't reach it without that

I have a remote machine connected via VPN. Iperf3 between the client and NFS server is about ~250Mbps. But copying (rsync) files to the NFS mount is less than 1Mbps... any tips on troubleshooting?

meena: "interface: ..." in the config?

RhodiumToad: *nod*

SpaceBass: what's the latency?

Ill check in a second... other issue (affecting me currently) is that tab complete locks up the system for a good 90 seconds

180ms

that's not at all good for nfs

Would that cause painfully slow write speeds though?

SpaceBass: 90 s or 180 ms?

depends... which protocol and transport?

188.959 ms

It's v4 (assume UDP?... )

tab complete may well be doing a whole lot of stat() calls, each of which will take a round-trip

v4 is tcp only iirc

Ah, it is tcp

FWIW I'm trying to tab complete in ~ not in /mnt/foo/bar

oh

~ is on local disk?

Tab complete could be unrelated but it started being an issue when I started using this mount

Yeah ~ is local (SSD in fact)

any symlinks involved?

Not yet :) ... rsync /foo/bar /mnt/server/share/foo/bar

tab complete in which shell?

might be instructive to ktrace the shell while trying a tab complete and see what's slow

zsh

Setting aside the tab issue for now... it's the painfully slow writes that are killing me... is this a place to use something like nfs cache? Or cachets?

S/cachets/cachefs

There are various time-outs (see "zshall(1)"); gave up before finding one that would clearly apply to completion

s/clear/obvious/

SpaceBass: is this rsync with or without --whole-file and if without, does the file already exist?

SpaceBass: and is the callback path enabled and does the server do delegations?

Without --whole-file, new files (dont exist yet)

Im unfamiliar with the concept of a callback path... server is FreeBSD too FWIW

is nfscbd_enable="YES" set in rc.conf / is nfscbd running

is the filesystem on the server also accessed locally on the server, or only via nfsv4 clients?

RhodiumToad, its only via nfsv4 clients

is vfs.nfsd.issue_delegations set

(on the server)

(however "Unfortunately, at this time, client use of delegations is limited, so performance gains may not be observed.")

if you look at the rsync writing process in top while this is going on, what wait channel is it usually showing?

Let me check

Wait

Oops... uwait

probably not the right process or thread

uwait is waiting on a umutex, which generally means contention on some thread lock

maybe try top -CSH

ir 0CSHz

*or -CSHz

Checking