00:00:35 <_xor> crest: Heh, funny you mention that. I'm literally working on some code right now for my nomad plugin. 00:00:53 what will it do xor? 00:01:14 _xor: which plugin is that? 00:01:25 <_xor> crest: I wanted to originally use OCI to package and distribute jail images, but OCI isn't very ZFS friendly (yet). It expects tarballed layers with whiteouts. 00:01:26 did you write the jail task driver or the pot one? 00:01:30 crest btw which firewall works best with fbsd bhyve and jails? pf or ipfw? 00:01:41 <_xor> crest: Neither, both of those are written in Go. Mine is written in Rust. 00:01:51 _xor: https://github.com/michael-yuji/xc/tree/main/ocitar ? 00:01:53 Title: xc/ocitar at main · michael-yuji/xc · GitHub 00:01:56 is that you? 00:02:18 <_xor> Also, from what I saw, those drivers didn't support netgraph. 00:02:36 <_xor> Nope, but I did recently see that project. In fact, the one and only issue on the repo is from me heh. 00:02:54 why isnt oci zfs friendly? 00:03:19 <_xor> https://github.com/michael-yuji/xc/issues/1 00:03:21 <_xor> That's me. 00:03:21 Title: Questions · Issue #1 · michael-yuji/xc · GitHub 00:03:22 1 – [aha] [scsi] Toshiba MK156FB scsi drive does not work with 2.0 kernel https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=1 00:03:31 depends on your definition of zfs friendly 00:04:11 <_xor> OCI is fairly Linux-centric (though has generic mechanisms like annotations) and expects tarballs for its layers. 00:04:44 <_xor> Would have to figure out how to specify a base image + incremental snapshots to apply, though that shouldn't be too terribly difficult. 00:05:03 "in fact ipfw seems to be a better choice than pf for a lot of the things I'm interested to do" 00:05:07 wow just what i was thinking about 00:05:26 i've been a pf user and it's fine but i wondered if ipfw is better now that i'm getting deeper into fbsd 00:05:36 * RhodiumToad prefers ipfw 00:05:45 <_xor> How come? 00:05:57 * _xor hasn't used ipfw since early 2000s 00:07:01 imo ipfw is better suited to automation and its more expressive tables can vastly reduce the number of rules 00:07:16 but pf has the more human friendly syntax and pfsync in its favour 00:07:20 <_xor> What are the trade-offs? 00:07:28 <_xor> ah 00:08:03 and ipfw tends to scale better to higher core counts 00:08:03 I've always heavily used the tables, which is probably why I prefer it 00:08:17 <_xor> HashiCorp just went BSL for a lot of their projects. Doesn't impact that much...yet, but it might dim the outlook a bit :/ 00:08:31 BSL? 00:08:36 <_xor> Really? Because it's userland? 00:08:43 <_xor> RhodiumToad: Business Source License 00:08:48 i like to think of PF as a ready to use firewall and IPFW as firewall construction kit 00:09:06 <_xor> https://news.ycombinator.com/item?id=37082506 00:09:07 Title: HashiCorp Switches to BSL License | Hacker News 00:09:40 <_xor> Whatever happened to ipf? 00:10:23 _xor: died over a decade ago, but nobody buried the corpse 00:10:40 <_xor> lol 00:11:11 * RhodiumToad uses WITHOUT_IPFILTER and WITHOUT_PF on builds 00:11:17 * _xor is convinced to check out ipfw again 00:11:35 why? they're kernel modules. keeping the modules doesn't hurt 00:12:20 those options remove the userland programs etc. 00:15:10 <_xor> Man, that was annoying. 00:15:15 * _xor just got his package delivered 00:16:01 <_xor> Ordered a bundle of CAT6 + RJ45 ends. Turns out, the CAT6 I got is 23 gauge while the RJ45 heads were 24+ gauge only. Had to order new RJ45 ends for 23 gauge. 00:16:21 * _xor was mounting and wiring up his PoE security camera when he found this factoid out 00:17:23 ya i really wanna see pkgbase and other ways to cut down fbsd to only what's needed 00:17:39 i might try ipfw too 00:17:50 <_xor> pkgbase will do that. It does for me. You can thank meena. 00:17:51 crest what does ipfw do better for automation? 00:17:59 ty meena!! 00:18:29 you can put the structure into the ruleset and later populate the tables 00:18:39 I've done a fair bit of stuff where ips are automatically added and removed from tables 00:18:45 to express almost all things you want to automate 00:18:52 pf has tables like that too no? and anchors i think they're called? 00:18:54 much easier to manage those than mess directly with the ruleset 00:19:05 pf tables are just sets of ip addresses 00:19:13 ya 00:19:15 these are more? 00:19:31 ipfw tables can match more fields in the packet (protocol, port, etc.) and map them to values 00:19:42 with up to one value per type per entry 00:19:48 e.g. a rule number and an interface 00:19:49 <_xor> That's one of my biggest gripes with pf, handling dynamic management of rules. 00:20:05 what pf does have are (nested) anchors 00:20:18 <_xor> Yeah, I use those with nomad (kind of). 00:20:20 does ipfw have something comparable to nested anchors? 00:20:27 and you can feed it the rules for anchors from a pipe 00:20:42 polyex: often times it doesn't need them to accomplish the same 00:20:55 and if your do need something there are sets of rules 00:21:04 (iirc up to 31 sets) 00:21:05 <_xor> I run nomad as an unprivileged user and so my driver needs to add a mapping for ports <1024 (assuming it's running on the host and not within a jail). 00:21:10 ah ok. just wanted to make sure i could SWITCH to ipfw, not start having to use ipfw AND pf 00:21:21 <_xor> I do that with nested anchors + rdr rules. 00:21:25 _xor: there is a mac module to allow this 00:21:32 mac_portacl 00:21:40 is there anything pf can do that ipfw can't do and do well? 00:21:53 you also have change the low port range for it to take effect 00:22:08 but if you want to you set it to stop at e.g. port 22 00:22:24 * _xor is reading mac_portalacl(4) 00:22:31 <_xor> mac_portacl(4) I mean 00:23:10 <_xor> Well, I mean I'd rather not have any of that at all and just let it just use random unprivileged ports within a specific range. 00:23:54 <_xor> But for a narrow set of services that I'm still using nomad to deploy, I need to be able to bind those to privileged ports (e.g. Traefik for tcp/80 + tcp/443, etc). 00:24:38 <_xor> I mean I guess I could also set it up with ng_nat, but all of that is a hassle. 00:25:54 why go through netgraph?!? 00:26:14 <_xor> Because I'm using it currently for my jails? 00:26:55 ipfw can do local port mappings without nat 00:27:44 <_xor> Lol, on a side note, I just noticed this while looking at CAD packages in ports... 00:27:51 <_xor> "The BRL-CAD source code repository is the oldest known public version-controlled codebase in the world that's still under active development, dating back to 1983-12-16 00:10:31 UTC." 00:32:43 if they mean that they still have a repo going back that far, they could well be right 01:04:37 is there anything pf can do that ipfw can't do and do well? 02:35:58 New to Pouderie... after some tinkering and learning, I built a dedicated box (VM) and gave it a lot of resources... but it doesnt seem to be using them, I get slow builds and low resource utilization 02:36:25 Any tips on optimizing a box for pouderirie? 02:36:37 (I speak French and for the life of me I can't spell that word) 02:39:14 haha 02:39:34 sorry, can't answer that question 02:42:29 (The spelling or the optimization? Haha) 02:49:34 what settings have you changed from defaults? 02:49:46 and how much RAM do you have? 02:49:55 and how many CPUs? 02:51:10 RhodiumToad, I didn't change much, just the required stuff, ZFS pool 02:51:27 it's worth noting that poudriere can often be very disk-bound, unless you use tmpfs for everything, which requires a shedload of ram 02:51:28 12 CPU cores, 24gb 02:53:01 have you looked at what is going on during a build? 02:53:28 specifically disk i/o bottlenecks as well as cpu or memory 03:28:29 I'll check disk IO now 03:31:50 Unrelated, does 14.0 not have a pam.d/auth or pam.d/account? 04:11:32 SpaceBass: mine doesn't 04:11:51 Thanks for checking - just wanted to make sure 04:13:13 Next challenge... install python 11 04:13:15 3.11 04:17:21 there's a package (or port) for that 04:19:04 Yeah installed it... but 3.9 was default and now I have to get it all sorted 04:20:21 Trying to get certbot to work in 14.0-CURRENT and it wants 3.9... but it needs a newer version (3.10 +) of cryptotography 05:07:09 even if you install 3.11 first? 07:41:25 Hello there 07:42:19 I'm replacing disk in a raid pool, it starts very quickly but now the system response very slowly 07:43:18 zfs? 07:43:19 Forget to notice ZFS 2way mirror 07:43:22 yes 07:43:32 211G resilvered, 63.99% done, 01:39:04 to go 07:43:51 the progress is about 1% in 15-20 minutes 07:44:23 What will happen if I restart machine during resilver / replace 07:45:46 the strange thing is that the new drive ( seagate barracuda ) is slow to response for smartctl queries 07:46:09 it should pick up where it left off 07:46:30 gstat ops is almost 0 07:46:47 is it an SMR drive? 07:46:48 queue - 2 .. 16 ... 07:47:01 SMR? 07:47:40 shingled magnetic recording 07:48:34 gives a significant increase in storage density at the expense of having the write performance go completely to shit on a regular basis 07:48:59 Seagate BarraCuda 3.5 (SMR) 07:49:02 smr... 07:49:08 yeh 07:49:22 basically, large portions of the disk area can be written only in large sequential blocks 07:49:55 so writes accumulate on temporary areas of the disk and then have to be rewritten to the shingled zones, which takes a long time 07:50:08 So the only option is to wait resilver to complete 07:50:12 yup 07:50:17 :( 07:50:26 if i script bsdinstall, what does bsdinstall do with that script to make a bootable OS? and can we just do whatever bsdinstall does and make images that can be preconfigured then directly booted and run? like skipping any manual install or setup 07:50:29 RhodiumToad, thank you for quick reply 07:50:59 polyex, create partition, write bootcode 07:51:03 * RhodiumToad has a bunch of smr drives for bulk mostly-read-only storage, so is very familiar with the write slowdowns 07:51:43 RhodiumToad, unfortunately this is a first experience to me 07:51:47 polyex: sure, making a bootable system without bsdinstall isn't hard, especially if you know the disk setup in advance 07:51:49 never had SMR drive 07:52:04 what's the basic process like? 07:52:47 starting from a fresh disk: gpart create, gpart add (possibly several times), gpart bootcode 07:52:52 gpart destroy -F /dev/disk # cleanup entire disk 07:52:59 then newfs or zpool init 07:53:04 ^ 07:53:56 if it's EFI, then preparing the ESP is a few more steps if you don't want to do the hackjob that's used for bootable images 07:54:31 (newfs_msdos and a few file copies, nothing too tricky) 07:56:46 then installing the OS can be done by unpacking the distribution txz files, or by make installworld distribution installkernel from a source tree 07:57:33 then set up whatever config files you need 07:58:11 (if not using zfs, remember to create /etc/fstab, that's the one that usually bites me when I do manual image creation) 07:59:20 RhodiumToad, fstab is required anyway, swap partition things 08:00:04 polyex, do not forget to create extra swap partition at least 16GB or better 2xRAM 08:01:55 hmm i wonder how to automate all of that 08:03:50 well, that's a large part of bsdinstall's job :-) 08:03:58 but otherwise it's just a matter of scripting 08:04:20 polyex, here is my old script 08:04:22 https://bsd.to/4JQe 08:04:23 Title: dpaste/4JQe (Bash) 08:05:03 this will partition the disk for you 08:05:20 it is very old, dont remember what I did back then 08:05:59 that looks like it supports both EFI and legacy boot, usually not needed these days 08:06:14 (though sometimes is) 08:06:21 polyex: you know you can script bsdinstall, right? 08:06:42 yes, but I did this script basically to restore ZFS images back to the system 08:06:46 ya 08:07:00 universal is good :) 08:09:07 say you name script: "partitiondisk.sh", you can create config file like: "partitiondisk.cfg" in the same location 08:09:21 here is the sample of the config: https://bsd.to/2GNP 08:09:22 Title: dpaste/2GNP (Bash) 08:13:28 polyex, the installation things is quite easy, just download and extract files into root directory of newly created partition 08:14:40 ftp://ftp.freebsd.org/pub/FreeBSD/releases/amd64/13.0-RELEASE/ 08:14:40 base.txz, kernel.txz, lib32.txz, src.txz 08:15:09 src is optional, but I do like to have them, need to compile something time to tile 08:15:17 prep disk, extract files, copy in config files, boot? 08:15:25 yep 08:15:27 that easy 08:15:53 I love BSD approach - all the configs are text, all the logs a re text 08:15:56 any way to bundle all of that up, even the disk prep stuff, so you can do it all in like 1 step? 08:16:08 exactly 08:16:09 yes, that's literally bsdinstall :-) 08:16:15 ^^^ 08:16:38 bsdinstall doing BEADM things, which I do not like 08:19:42 RhodiumToad, Thank you so much! You saved me a day. ANd a hard talk with a sales manager in a shop 08:20:11 * nerozero AFK 17:55:17 Im struggling a lot with pam.d and winbind auth... log says user is successfully authenticated and then pam_winbind(sshd): PAM_ESTABLISH_CRED not implemented 18:08:59 cd 18:45:07 what does BEADM things mean? 18:50:16 Before "beadm" was the precursor of "bectl". So I would say the way that command works or is used to create boot environment 18:50:24 s/Before// 18:51:13 oh and nerozero said bsdinstall does beadm stuff, so it needs to be updated to use bectl? 18:52:00 bectl is the successor to beadm 18:52:27 Cannot say what nerozero actually meant 18:52:32 (oh parv said that already, sorry) 18:53:18 ty! 18:54:23 polyex: vermadem is the author of BEADM, he is active on twitterX and on his blog 18:57:39 ... on Mastodon: vermaden⊙bc 19:13:48 how do i get local_unbound to listen on the private network IP, as well as localhost? 19:22:14 it would no longer be local_ if it listed on the network ip 19:45:13 I want jails on the host to use it, but they can't reach it without that 19:59:12 I have a remote machine connected via VPN. Iperf3 between the client and NFS server is about ~250Mbps. But copying (rsync) files to the NFS mount is less than 1Mbps... any tips on troubleshooting? 19:59:33 meena: "interface: ..." in the config? 20:01:12 RhodiumToad: *nod* 20:02:41 SpaceBass: what's the latency? 20:03:11 Ill check in a second... other issue (affecting me currently) is that tab complete locks up the system for a good 90 seconds 20:03:45 180ms 20:03:54 that's not at all good for nfs 20:04:12 Would that cause painfully slow write speeds though? 20:04:19 SpaceBass: 90 s or 180 ms? 20:04:31 depends... which protocol and transport? 20:04:31 188.959 ms 20:04:53 It's v4 (assume UDP?... ) 20:05:03 tab complete may well be doing a whole lot of stat() calls, each of which will take a round-trip 20:05:10 v4 is tcp only iirc 20:05:21 Ah, it is tcp 20:05:41 FWIW I'm trying to tab complete in ~ not in /mnt/foo/bar 20:07:59 oh 20:08:25 ~ is on local disk? 20:08:27 Tab complete could be unrelated but it started being an issue when I started using this mount 20:08:34 Yeah ~ is local (SSD in fact) 20:08:52 any symlinks involved? 20:10:58 Not yet :) ... rsync /foo/bar /mnt/server/share/foo/bar 20:12:01 tab complete in which shell? 20:12:27 might be instructive to ktrace the shell while trying a tab complete and see what's slow 20:13:52 zsh 20:14:27 Setting aside the tab issue for now... it's the painfully slow writes that are killing me... is this a place to use something like nfs cache? Or cachets? 20:14:34 S/cachets/cachefs 20:15:15 There are various time-outs (see "zshall(1)"); gave up before finding one that would clearly apply to completion 20:15:27 s/clear/obvious/ 20:16:51 SpaceBass: is this rsync with or without --whole-file and if without, does the file already exist? 20:17:14 SpaceBass: and is the callback path enabled and does the server do delegations? 20:20:31 Without --whole-file, new files (dont exist yet) 20:21:04 Im unfamiliar with the concept of a callback path... server is FreeBSD too FWIW 20:21:46 is nfscbd_enable="YES" set in rc.conf / is nfscbd running 20:22:19 is the filesystem on the server also accessed locally on the server, or only via nfsv4 clients? 20:22:38 RhodiumToad, its only via nfsv4 clients 20:23:11 is vfs.nfsd.issue_delegations set 20:23:16 (on the server) 20:23:48 (however "Unfortunately, at this time, client use of delegations is limited, so performance gains may not be observed.") 20:24:38 if you look at the rsync writing process in top while this is going on, what wait channel is it usually showing? 20:26:33 Let me check 20:29:07 Wait 20:29:13 Oops... uwait 20:31:31 probably not the right process or thread 20:32:06 uwait is waiting on a umutex, which generally means contention on some thread lock 20:32:43 maybe try top -CSH 20:32:49 ir 0CSHz 20:32:54 *or -CSHz 20:36:54 * RhodiumToad should probably get out of the habit of specifying -C, it makes no difference on SCHED_ULE, only on SCHED_4BSD 21:35:36 Checking