01:01:30 anyone have docker fully running on freebsd? 01:08:29 rwp: whassup dude? Hey, that high CPU load was really related to memory consumption. More precisely with swap activation. 01:08:53 i wanna make a freebsd server that can run docker images so linux schmucks can run crap on my box 01:14:28 polyex: you mean natively ? 01:14:45 dunno, just any way where it's actually usable 01:14:51 semi competitive with a linux host 01:20:04 well, there's already the o'good jails. I'm not sure if docker will work on FreeBSD the same way as linux. I think there's components deeply tied with linux kernel features 01:22:18 so what it's just hit or miss lognull? 01:22:43 there should be a docker testsuite that can fully run on freebsd to show that it's working or not 01:23:45 ironwail quake compiles just fine for anyone that plays quake 1. 01:24:02 in docker? 01:24:03 polyex: have you tried those examples from wiki? ( https://wiki.freebsd.org/Docker ) 01:24:04 Title: Docker - FreeBSD Wiki 01:25:37 While typing with you I made a quick read of this page and I must tell you that I was surprised with the reported progress... :o 01:25:58 it's very exciting 01:26:23 freebsd hosting docker is big 01:27:52 If I had the opportunity to chose, I would stick with jails ( with bastille ). 01:28:12 well ofc but sometimes someone wants to pay us to run a docker img whatever tf it is 01:28:16 s/chose/choose/ 01:28:24 wanna do that with freebsd instead of lowering myself to linux 01:28:51 polyex: hey, you got a point there :o 01:29:08 so you use bastille btw? thoughts? 01:30:13 Not yet. But I can't wait to have a good hardware to test it. 01:30:26 seen runj? 01:30:31 I just have tried bhyve for small labs 01:30:35 another freebsd container thing 01:30:44 ya bhyve virt is tops 01:33:07 bhyve is one of my reasons to stick with FreeBSD. I mean OpenBSD and NetBSD hypervisors seems pretty good, but bhyve seems to have learned with big players. Not just the very basic virt like vbox. 01:33:41 and mixing it with ZFS is really a charm 01:33:43 ya and maybe it can take some inspiration from firecracker vm too 01:33:47 ya it is 01:33:51 and jails too 01:34:03 so you go for pf or ipfw? 01:34:59 I have no experience with ipfw TBH 01:35:13 so then pf 01:35:40 i use pf but ipfw is the native fw so thinking of changing to that 01:36:06 I moved from OpenBSD to FreeBSD because of the improvements of desktop stuff on FreeBSD 01:36:47 I was used to iptables stuff , but pf rules are way to simplier to read, you know ? 01:37:33 ya 01:37:44 if i use ipfw do i gotta use iptables too? 01:46:09 No , no. iptables is linux fw stuff. ( nowadays they have iptables and firewalld ) 01:47:01 ah so you went from linux to bsd 01:49:24 lognull, Memory consumption? Swapping? How much RAM does your x220 have? Mine has 8GB. More is always better! 01:49:45 With 8GB one can surf but one must keep things contained. :-) 01:56:41 rwp: I have 16GB here. 02:01:40 rwp: I still not discovered yet the main reason, but my system was allocating all available memory even with a low value for ZFS ARC ( as mentioned before ). 02:03:38 I'm testing some adjustments in vm.kmem_size and vm.kmem_size_max. After set values to a limit lower than total RAM and higher than vfs.zfs.arc_max things seems under control now. 02:36:31 k maybe this doesn't make sense but every time i wanna add another bhyve vm slot i gotta add another tap# cloned_interfaces entry to rc.conf. any way a wildcard thing could eliminate that? 02:36:44 like cloned_interfaces="tap*" 02:37:09 kinda like how we can set skip tap in pf config and it applies to the whole tap group meaning any tap# 02:42:18 do you have to, though? 02:42:42 i note that tun/tap supports the magical devfs cloning so that just trying to access /dev/tapN will create tapN 02:43:02 oh so maybe i don't even need a cloned_interfaces entry for the tap stuff? 02:43:17 i suspect not 02:43:29 that would be amazing. gonna try it now 02:43:35 leave bridge0 in there tho right? 02:43:39 yes 02:43:48 oh, you probably add it to the bridge in rc, huh? 02:45:37 well i got autobridge interfaces having bridge0, and autobridge bridge0 has igb0 foo and tap* 02:45:56 but then i also got the cloned_interfaces with bridge0 then a ton of tap# 02:45:58 0+ 02:46:03 lognull, I have no idea! Good hunting to find the root cause. 02:46:30 i can still get rid of the tap# entries from cloned interfaces prolly? 02:47:43 maybe 02:47:51 it depends on how exactly autobridge is implemented 02:49:14 i have a suspicion that it will need rc to create the tap to do the necessary autobridging... I don't really see any devd magic that could be handling interfaces spontaneously created 02:49:47 trying now 02:50:07 yeah, it's cheap enough to just try it and see, at least 02:50:27 just commented the line out in rc.conf and rebooting 02:50:33 gonna start vm up and see if networking is normal 02:51:56 ah damn. i see in ifconfig the tap interfaces created lazily on demand, but trying to rdp to vm fails 02:52:03 fails to connect 02:52:08 we were so close 02:52:55 yeah, so autobridge needs the rc framework to actually handle the tap 02:53:53 i wonder how ppl handle unbounded vms on a host and all their taps. they run ifconfig and see 30,000 entries? 02:55:02 rwp: thx! :D 03:35:28 you can ifconfig by class 03:35:43 ? 03:36:15 only show physical interfaces or whatnot 03:36:44 would that help my situation? 03:37:08 it would hide the 29997 tun interfaces 03:37:59 or you could use a framework (like vm-bhyve) which cleans up and clones them for you 03:40:06 how do i do ifconfig by class then? to hide the endless tap# in cloned_interfaces in rc.conf? 03:41:39 ifconfig -a -G tap 03:41:57 can it be done permanently in rc.conf? 03:42:15 how often are you running ifconfig -a and visually acting on it? 03:42:20 and how many tap interfaces do you have? 03:42:36 tbh not often, and i'm up to a few dozen by now 03:43:00 huh, i have 11, and i guess i never considered it to be a problem 03:43:10 every vm i make i have to provision a tap# to assign to the vm 03:43:23 once it scrolls off the terminal, it's kinda all the same 03:43:51 so how can i fix it in rc.conf? 03:44:00 polyex: do you have a shell script that runs `bhyve -c -m 4G ....` dozens of times or what 03:44:18 i have 1 script per vm that starts it up 03:44:25 passes its assigned tap# etc 03:44:36 so a few dozen 03:45:17 i have 3 lines for bhyve in my rc.conf 03:45:58 what are they? 03:46:15 vm_enable="yes" 03:46:24 vm_dir="/vm" 03:46:43 vm_list="dorn sirtis steward wheaton burton" 03:46:47 what's vm_dir for? 03:47:08 i guess where you put all the bhyve vms? 03:47:15 they're all for sysutils/vm-bhyve 03:47:25 are your vms all auto started when the host boots? 03:47:38 there are other vms that do not autostart 03:47:53 interface is cloned and bridged automatically 03:49:29 ah nice. i do everything manually still 03:49:32 might change later not sure 03:50:08 i don't think i have the patience to do it your way 03:50:49 check it out; you could provision one vm on it and see how it feels alongside your scheme 03:55:46 tyvm 03:55:54 what do you think of runj? 03:59:31 i haven't used it 06:49:12 Hello 06:49:53 I'm having problems with portmaster using multiple cores when compiling ports 06:50:17 it's under utilizing my hardware and it takes forever to compile ports 06:51:21 I have MAKE_JOBS_NUMBER=13 in make.conf 06:52:50 I have 6 cores 06:53:01 when I do htop, it looks like I have 12 threads 06:53:40 Only one is being used when I compile ports 07:35:59 Grell: some ports specify MAKE_JOBS_UNSAFE. 07:37:03 I have MAKE_JOBS_UNSAFE=yes in my make.conf as well 07:49:47 hi there 07:50:12 how to test process with id: xxxx is alive or not 07:50:24 for scripting 07:51:06 the exit code "ps -p xxxx" doesn't depends on pid alive or not 07:54:34 never mind it does 10:16:27 what do i gotta put in a installerconfig file for a bhyve vm to set up zfs? the bhyve host runs zfs and makes a dataset or whatever for the bhyve vm to use 10:46:07 sorry i meant volume 10:46:27 so host runs zfs and it creates a zfs volume for a bhyve vm to use 10:54:59 zfs host gives a zfs volume to bhyve vm, bhyve boots bsdinstall, then at partitioning i select auto zfs. the zfs configuration options need to get translated into bsdinstall script at /etc/installerconfig 11:02:12 in the context of the bhyve guest, what value should i use for ZFSBOOT_DISKS? 11:35:07 I've a question. I'm trying to figure out why `jail_set` within a jail is failing with "Operation not permitted" (i.e.: nested jail); I was wondering if there was a way to get some more detail as to *why* it's not permitted? 11:35:18 I know I'm probably missing some setting in the outer jail, but. 11:35:27 FTR, this is `jail_set` being called by podman. 11:35:40 (and I'm on 13.2-REL) 11:36:15 I also have a 2nd question (VNET-related), but one problem at a time :D 11:50:22 Oh right, correction, it's `buildah run` calling the `jail_set`. 12:10:00 If it helps, here's the params to `jail_set` (extracted via gdb): http://codepad.org/ETciFkkq 12:10:02 Title: Plain Text code - 42 lines - codepad 12:10:33 Basically `$iov[$i]`, `(char*)$iov[$i]`, and `*(int*)$iov[$i]`, respectively. Hence 3 lines for each. 12:16:59 the parent jail was created with children.max greater than 0? 13:05:27 i just tried to setup ufs with 2M blocksize and it says the filesystem doesn't support that size? 13:07:12 well duh 13:07:59 which is bullshit because more than half the files on disk are <2M 13:09:05 so I've got double overhead on the disk space 13:09:59 huh? 13:15:42 the limits for ufs are that the fragment size must be a multiple of the disk sector, the block size must be 1,2,4 or 8 frags (should always be 8), the block size must be large enough for a superblock and no larger than MAXBSIZE (default 64k) which must be no larger than MAXPHYS 13:16:11 4k/32k is the current standard 16:08:35 RhodiumToad, yup, 24. 16:09:05 And I've tried setting securelevel to -1, enforce_statfs to 0, all that (out of, frankly, desperation) 16:14:58 what are those set to in the parent jail? 16:15:52 child jail's securelevel can't be lower than parent's, child's enforce_statfs can't be lower than parent's, etc. 16:16:30 child's devfs ruleset must be same as parent 16:16:41 (same ruleset number, that is) 16:17:09 RhodiumToad: those are numbers in the parent jail. 16:17:18 Hm, devfs ruleset in the child is different. 16:18:46 that's one of the code paths that returns EPERM 16:18:52 Unrelated question, as I just ran into this error. I upgrade from 12.4 to 13.2, and I'm having some weird trouble with ports being "semi-installed". E.g. `php82` is installed, but it won't show up in `pkg info`. But it *does* show up as a conflict on `pkg install php83`. 16:19:00 RhodiumToad: must be it, will try that. 16:19:42 that's probably a result of being present but with the wrong ABI. 16:19:58 Yeah. How do I list all such ports, though? (so that I can upgrade them "properly") 16:20:49 (also, my bad: devfs_ruleset *does* match up: it's `4` in both jail and subjail) 16:21:25 does it show up in pkg query -a '%n %q' 16:22:08 is the child jail's children.max <= the parent's? 16:22:24 It does show up in there, it seems. 16:22:36 As for the jail, correct: `1 <= 24`, respectively. 16:23:05 do any allow.* flags differ between parent and child jail? 16:24:08 try pkg query -e '%q ~ FreeBSD:12*' '%n %q' to see what needs updating 16:24:23 No, the child jail doesn't seem to have any `allow.*` flags. At least according to what I could extract via gdb. 16:24:49 The parent does have a bunch of `allow.*`, but I don't imagine that would be a problem, since it's more permissive than the child. 16:25:07 RhodiumToad: interesting, no `php82`. 16:25:15 Even though `php-fpm` is definitely missing. Or at least its rc script. 16:25:22 I only have `vim-console` in there. 16:25:44 (which is a simple enough fix) 16:25:46 what abi did php82 show up with in the pkg query -a output? 16:27:01 Oh. It's gone completely. My bad, I could've sworn I saw it earlier. 16:27:36 are you sure it's installed? 16:27:49 It definitely *was* before the 12->13 upgrade. But it seems like it got removed during it. 16:28:17 it could show up as a conflict on pkg install if some port is trying to install it as a dependency while some other port is trying to install a conflicting version 16:30:50 Who knows. Either way, that part's fixed. 17:26:26 is networking in a Debian jail supposed to work like a regular jail? 17:27:12 I have it configured with its own IP on the same subnet as the host's, sharing the bge0 interface, I can connect to it over ssh from host/local network 17:27:39 but I can't seem to be receiving data through the connexions I initiate 17:42:45 vnet or non-vnet jail? 17:59:19 non-vnet 17:59:30 I can ssh out & ssh in 17:59:56 but I'm trying to use rtorrent, it initiates outbound connexions but nothing is being downloaded 18:02:04 net.inet.ip.forwarding = 1 & security.jail.allow_raw_sockets = 1 18:04:11 sockstat -4 on host shows a bunch of connexions from jail:port but no data transfer happens after the handshake 18:20:25 https://vez.mrsk.me/freebsd-defaults.html 18:20:26 Title: FreeBSD - a lesson in poor defaults 18:30:44 making a scripted bsdinstall, in the context of the bhyve guest, what value should i use for ZFSBOOT_DISKS? 18:46:35 What the heck? Why can't find find pdf files? If I do an "ls | grep -i pdf" it shows pdf files in the directory I'm searching but "find /*path to same dir* -iname pdf" shows no results. 18:52:37 BillyJoeBob: that's not how find works 18:54:12 find /path/to/dir -iname \*.pdf 18:55:46 i have 2 options, ada0 bhyve sata disk, and vtbd0 virtio block device, for my guest vm zfs stripe disk to use 18:55:51 not sure which 18:59:55 polyex: for vm-bhyve? 19:00:17 no i'm doing it manually 19:00:43 my client is bugged brb 19:07:35 that freebsd-defaults dude certainly has some opinions 19:15:49 meena: thanks. I'm trying to force myself more into cli stuff so I'm new to that and I wasn't making much sense of the man page. 19:16:45 wtf is my isp doing 19:17:06 hi, testing unix socket file for existence, and getting false-positive 19:17:47 #!/bin/sh script, [ ! -f "/var/run/mysocket.socket" ] && echo "socket not exists" 19:18:24 outputs "socket not exists" while socket is there 19:18:43 sometimes this works as intended 19:18:59 BillyJoeBob, I share this reference with anyone who takes an interest in the find command and why it is unique. http://doc.cat-v.org/unix/find-history 19:19:00 Title: The History of the Design of Unix's Find Command 19:19:50 The find command is definitely worth learning to use and using it because it is a very powerful concept. 19:37:20 i shouldn't give any swap to a freebsd bhyve guest right? it's running zfs on both host and guest 21:25:45 maybe not freebsd-specific, but related to our shell it seems, anyone know why cron running `script.sh >> logfile 2>&1` kicks back with "Ambiguous output redirect." ? Maybe cron is running csh, so i need to do `script.sh >>& logfile`. Hm. Seems like that worked. Thanks for being a rubber duck. 21:37:36 wew my isp taking a dump today 22:11:15 why dont u use devuan that is at least without a jail :> 22:13:20 who 22:19:48 can someone review this? :)) https://wiki.freebsd.org/Scientific any feedback is appreciated. 22:19:49 Title: Scientific - FreeBSD Wiki 22:32:54 Can I get some handholding unfucking my iohyve / bhyve networking? I think the issue might stem from moving server / interface name changing. I tried redoing iohyve setup net= bit with new interface name but it's still not working 22:33:22 I can tcpdump tap0 and see traffic but it doesn't seem to make it to the lan. so something tapN <-> bridge0 is wrong perhaps? 22:36:35 or could also be the bridge is not connected to igb0 my new interface. old interface was ... em0 maybe? 22:40:27 well ifconfig the bridge and see what members it has 22:45:18 that was actually it ... saw addm blah blah then saw that in man page and added tap0 now things maybe work 22:45:29 but why was it not added automatically by iohyve magic or bhyve or whatever 22:48:31 anyway next issue: serial console is all fucked up when trying to boot linux guest? characters not echoed properly, can't use backspace, enter doesn't make a newline, console is also extremely slow 22:49:00 bsd guest seems fine ish? haven't messed with it much but I could boot and install freenas 22:51:46 and it used to work ok on the old system? 22:52:26 the networking bit yes. I'm just now trying to do linux guests though. I don't think that's related to the system change 22:53:12 just console enough to get ssh up and do that 22:53:27 rtprio cool username! 22:53:55 I need the console to install the goddamn thing 22:54:34 well, not entirely true, but i understand 22:54:41 alternatively , how can I get docker working on freebsd? oooorrrr is there a native nvr? trying to get frigate nvr or another nvr working that's the larger overall goal here 22:54:45 i don't know, i don't use iohyve 22:55:03 you can use podman on freebsd, but of course you can't use any of the images on dockerhub 22:55:10 so you'll need to build it all yourself 22:55:48 what about ... bastille? 22:56:05 i don't know anything about that 22:56:51 qmr do you want Docker, or Docker images 22:57:10 because Docker doesn't work and will never work. but you can get Docker images running as Linux Jails 22:58:21 hmm bastille seems to be jails only 22:58:41 qmr it can run Linux Jails, but it doesn't boot them 22:58:49 you can use it to run commands, not services 22:59:01 if you want, I have a blog post on how to boot a complete devuan system in a Jail 22:59:05 qmr: https://wiki.freebsd.org/Containers 22:59:06 Title: Containers - FreeBSD Wiki 22:59:38 antranigv: I need *something* to use as a security camera NVR, that's the bigger goal here. frigate looks cool, which lead me to trying to boot a coreos guest so I can run docker to run frigate 22:59:46 https://weblog.antranigv.am/posts/2023/08/freebsd-jail-devuan-linux-openrc/ 22:59:48 Title: FreeBSD Jail booting & running Devuan GNU+Linux with OpenRC | Freedom Be With All 23:00:04 what's frigate? any link to that please? 23:00:07 qmr: i run docker in debian in bhyve and it works... just about as well as you could expect 23:00:24 I do that too, but for me it works flawlessly 23:02:49 qmr as far as I can tell, frigate needs to run on bare-metal inside docker. which means that Linux Jails will not work for you. sorry 23:17:24 any suggestions for security camera software? I have found some cheap cameras I can de shittify with an alternative firmware but now I need to catch the video and have recordings and configurable alerts 23:19:07 hmm maybe I can run frigate on kodi machine 23:43:56 antranigv what's the difference between docker and docker images? 23:44:48 how much overhead is there, if any, running docker through a linux guest in bhyve? 23:45:12 polyex docker is a program/daemon that you run, to manage containers on Linux. Docker images are based on the OCI spec and can be run without docker, for example with Podman. and can also be converted to, say, tarball with rootfs, that you can technically run as a Linux Jail :) 23:45:38 oh wow so any docker image can be run on freebsd!!!!!!!!!!!!!!!!!!!!!!!!!!!! 23:45:54 "any" is a bit much. most, yes 23:46:18 but if it relies on very linux-y things, such as frigate relies on the GPU, then no 23:46:40 or, NextCloud's Docker image for example relies on the Docker actually being available, it's not just an image 23:47:04 (it hijack's the host's Docker to run other containers) 23:47:25 ah 23:47:31 and regarding your question of overhead. if you setup everything as a bridge (if_bridge + tap + veth on Docker) then it will all work fine. 23:47:41 altho Docker + NAT works fine too 23:49:35 how's the performance relative to running debian directly on hardware? 23:50:16 my machines are powerful, so personally I've never felt anything being slow, but I never run Linux on the host, so how would I know? :))) 23:50:30 ahh 23:50:40 can bhyve guest vms running zfs have zfs disk encryption enabled if it's a server that should be able to start up without entering a pw on boot? how would that even work? 23:51:11 it doesn't. you need to decrypt/load-key before booting the VM 23:51:46 ok so freever servers everywhere that are running zfs are NOT using disk encryption, right? 23:51:53 freebsd* 23:52:35 well. most computers are not using disk encryption :D 23:52:53 I guess macOS is the exception 23:53:09 however, it would not be hard to setup key loading during boot 23:53:17 say, from a USB key, or something similar 23:53:19 desktops can because you give pw and login beore using it 23:53:47 btw tyvm antranigv for working on freebsd in the scientific computing realm 23:54:04 out of the... I wanna say 500 servers that I manage... I say only 10 of them use some kind of disk encryption 23:54:53 store credit cards or why do those 10 use disk encryption? 23:55:21 PII -- personal identifiable information 23:55:29 even things like DNA 23:55:35 recently discovered a shitty "roommate" or their friend stole some parts from my desktop. the drive that had all my phones synced to it. so yea disk encryption is a good thing. 23:55:57 ya i run disk encryption on all desktops. not yet on any servers. need to figure that out 23:56:07 storing the key on a connected usb drive isn't a fix imo 23:56:14 someone hacks into server, they just read the key info off 23:56:27 someone steals server, steals usb key with it 23:56:33 well yea 23:57:19 i shouldn't give any swap to a freebsd bhyve guest right? it's running zfs on both host and guest 23:57:48 that's up to you. swap lets the machine make better use of real ram 23:58:46 but host machine runs SSDs and what i read was swap on a guest results in excessive host SSD/NVME i/o and burns them out 23:59:30 I hear driving your car puts miles on the engine transmission and tires too