anyone have docker fully running on freebsd?

rwp: whassup dude? Hey, that high CPU load was really related to memory consumption. More precisely with swap activation.

i wanna make a freebsd server that can run docker images so linux schmucks can run crap on my box

polyex: you mean natively ?

dunno, just any way where it's actually usable

semi competitive with a linux host

well, there's already the o'good jails. I'm not sure if docker will work on FreeBSD the same way as linux. I think there's components deeply tied with linux kernel features

so what it's just hit or miss lognull?

there should be a docker testsuite that can fully run on freebsd to show that it's working or not

ironwail quake compiles just fine for anyone that plays quake 1.

in docker?

polyex: have you tried those examples from wiki? ( wiki.freebsd.org/Docker )

Title: Docker - FreeBSD Wiki

While typing with you I made a quick read of this page and I must tell you that I was surprised with the reported progress... :o

it's very exciting

freebsd hosting docker is big

If I had the opportunity to chose, I would stick with jails ( with bastille ).

well ofc but sometimes someone wants to pay us to run a docker img whatever tf it is

s/chose/choose/

wanna do that with freebsd instead of lowering myself to linux

polyex: hey, you got a point there :o

so you use bastille btw? thoughts?

Not yet. But I can't wait to have a good hardware to test it.

seen runj?

I just have tried bhyve for small labs

another freebsd container thing

ya bhyve virt is tops

bhyve is one of my reasons to stick with FreeBSD. I mean OpenBSD and NetBSD hypervisors seems pretty good, but bhyve seems to have learned with big players. Not just the very basic virt like vbox.

and mixing it with ZFS is really a charm

ya and maybe it can take some inspiration from firecracker vm too

ya it is

and jails too

so you go for pf or ipfw?

I have no experience with ipfw TBH

so then pf

i use pf but ipfw is the native fw so thinking of changing to that

I moved from OpenBSD to FreeBSD because of the improvements of desktop stuff on FreeBSD

I was used to iptables stuff , but pf rules are way to simplier to read, you know ?

ya

if i use ipfw do i gotta use iptables too?

No , no. iptables is linux fw stuff. ( nowadays they have iptables and firewalld )

ah so you went from linux to bsd

lognull, Memory consumption? Swapping? How much RAM does your x220 have? Mine has 8GB. More is always better!

With 8GB one can surf but one must keep things contained. :-)

rwp: I have 16GB here.

rwp: I still not discovered yet the main reason, but my system was allocating all available memory even with a low value for ZFS ARC ( as mentioned before ).

I'm testing some adjustments in vm.kmem_size and vm.kmem_size_max. After set values to a limit lower than total RAM and higher than vfs.zfs.arc_max things seems under control now.

k maybe this doesn't make sense but every time i wanna add another bhyve vm slot i gotta add another tap# cloned_interfaces entry to rc.conf. any way a wildcard thing could eliminate that?

like cloned_interfaces="tap*"

kinda like how we can set skip tap in pf config and it applies to the whole tap group meaning any tap#

do you have to, though?

i note that tun/tap supports the magical devfs cloning so that just trying to access /dev/tapN will create tapN

oh so maybe i don't even need a cloned_interfaces entry for the tap stuff?

i suspect not

that would be amazing. gonna try it now

leave bridge0 in there tho right?

yes

oh, you probably add it to the bridge in rc, huh?

well i got autobridge interfaces having bridge0, and autobridge bridge0 has igb0 foo and tap*

but then i also got the cloned_interfaces with bridge0 then a ton of tap#

0+

lognull, I have no idea! Good hunting to find the root cause.

i can still get rid of the tap# entries from cloned interfaces prolly?

maybe

it depends on how exactly autobridge is implemented

i have a suspicion that it will need rc to create the tap to do the necessary autobridging... I don't really see any devd magic that could be handling interfaces spontaneously created

trying now

yeah, it's cheap enough to just try it and see, at least

just commented the line out in rc.conf and rebooting

gonna start vm up and see if networking is normal

ah damn. i see in ifconfig the tap interfaces created lazily on demand, but trying to rdp to vm fails

fails to connect

we were so close

yeah, so autobridge needs the rc framework to actually handle the tap

i wonder how ppl handle unbounded vms on a host and all their taps. they run ifconfig and see 30,000 entries?

rwp: thx! :D

you can ifconfig by class

?

only show physical interfaces or whatnot

would that help my situation?

it would hide the 29997 tun interfaces

or you could use a framework (like vm-bhyve) which cleans up and clones them for you

how do i do ifconfig by class then? to hide the endless tap# in cloned_interfaces in rc.conf?

ifconfig -a -G tap

can it be done permanently in rc.conf?

how often are you running ifconfig -a and visually acting on it?

and how many tap interfaces do you have?

tbh not often, and i'm up to a few dozen by now

huh, i have 11, and i guess i never considered it to be a problem

every vm i make i have to provision a tap# to assign to the vm

once it scrolls off the terminal, it's kinda all the same

so how can i fix it in rc.conf?

polyex: do you have a shell script that runs `bhyve -c -m 4G ....` dozens of times or what

i have 1 script per vm that starts it up

passes its assigned tap# etc

so a few dozen

i have 3 lines for bhyve in my rc.conf

what are they?

vm_enable="yes"

vm_dir="/vm"

vm_list="dorn sirtis steward wheaton burton"

what's vm_dir for?

i guess where you put all the bhyve vms?

they're all for sysutils/vm-bhyve

are your vms all auto started when the host boots?

there are other vms that do not autostart

interface is cloned and bridged automatically

ah nice. i do everything manually still

might change later not sure

i don't think i have the patience to do it your way

check it out; you could provision one vm on it and see how it feels alongside your scheme

tyvm

what do you think of runj?

i haven't used it

Hello

I'm having problems with portmaster using multiple cores when compiling ports

it's under utilizing my hardware and it takes forever to compile ports

I have MAKE_JOBS_NUMBER=13 in make.conf

I have 6 cores

when I do htop, it looks like I have 12 threads

Only one is being used when I compile ports

Grell: some ports specify MAKE_JOBS_UNSAFE.

I have MAKE_JOBS_UNSAFE=yes in my make.conf as well

hi there

how to test process with id: xxxx is alive or not

for scripting

the exit code "ps -p xxxx" doesn't depends on pid alive or not

never mind it does

what do i gotta put in a installerconfig file for a bhyve vm to set up zfs? the bhyve host runs zfs and makes a dataset or whatever for the bhyve vm to use

sorry i meant volume

so host runs zfs and it creates a zfs volume for a bhyve vm to use

zfs host gives a zfs volume to bhyve vm, bhyve boots bsdinstall, then at partitioning i select auto zfs. the zfs configuration options need to get translated into bsdinstall script at /etc/installerconfig

in the context of the bhyve guest, what value should i use for ZFSBOOT_DISKS?

I've a question. I'm trying to figure out why `jail_set` within a jail is failing with "Operation not permitted" (i.e.: nested jail); I was wondering if there was a way to get some more detail as to *why* it's not permitted?

I know I'm probably missing some setting in the outer jail, but.

FTR, this is `jail_set` being called by podman.

(and I'm on 13.2-REL)

I also have a 2nd question (VNET-related), but one problem at a time :D

Oh right, correction, it's `buildah run` calling the `jail_set`.

If it helps, here's the params to `jail_set` (extracted via gdb): codepad.org/ETciFkkq

Title: Plain Text code - 42 lines - codepad

Basically `$iov[$i]`, `(char*)$iov[$i]`, and `*(int*)$iov[$i]`, respectively. Hence 3 lines for each.

the parent jail was created with children.max greater than 0?

i just tried to setup ufs with 2M blocksize and it says the filesystem doesn't support that size?

well duh

which is bullshit because more than half the files on disk are <2M

so I've got double overhead on the disk space

huh?

the limits for ufs are that the fragment size must be a multiple of the disk sector, the block size must be 1,2,4 or 8 frags (should always be 8), the block size must be large enough for a superblock and no larger than MAXBSIZE (default 64k) which must be no larger than MAXPHYS

4k/32k is the current standard

RhodiumToad, yup, 24.

And I've tried setting securelevel to -1, enforce_statfs to 0, all that (out of, frankly, desperation)

what are those set to in the parent jail?

child jail's securelevel can't be lower than parent's, child's enforce_statfs can't be lower than parent's, etc.

child's devfs ruleset must be same as parent

(same ruleset number, that is)

RhodiumToad: those are numbers in the parent jail.

Hm, devfs ruleset in the child is different.

that's one of the code paths that returns EPERM

Unrelated question, as I just ran into this error. I upgrade from 12.4 to 13.2, and I'm having some weird trouble with ports being "semi-installed". E.g. `php82` is installed, but it won't show up in `pkg info`. But it *does* show up as a conflict on `pkg install php83`.

RhodiumToad: must be it, will try that.

that's probably a result of being present but with the wrong ABI.

Yeah. How do I list all such ports, though? (so that I can upgrade them "properly")

(also, my bad: devfs_ruleset *does* match up: it's `4` in both jail and subjail)

does it show up in pkg query -a '%n %q'

is the child jail's children.max <= the parent's?

It does show up in there, it seems.

As for the jail, correct: `1 <= 24`, respectively.

do any allow.* flags differ between parent and child jail?

try pkg query -e '%q ~ FreeBSD:12*' '%n %q' to see what needs updating

No, the child jail doesn't seem to have any `allow.*` flags. At least according to what I could extract via gdb.

The parent does have a bunch of `allow.*`, but I don't imagine that would be a problem, since it's more permissive than the child.

RhodiumToad: interesting, no `php82`.

Even though `php-fpm` is definitely missing. Or at least its rc script.

I only have `vim-console` in there.

(which is a simple enough fix)

what abi did php82 show up with in the pkg query -a output?

Oh. It's gone completely. My bad, I could've sworn I saw it earlier.

are you sure it's installed?

It definitely *was* before the 12->13 upgrade. But it seems like it got removed during it.

it could show up as a conflict on pkg install if some port is trying to install it as a dependency while some other port is trying to install a conflicting version

Who knows. Either way, that part's fixed.

is networking in a Debian jail supposed to work like a regular jail?

I have it configured with its own IP on the same subnet as the host's, sharing the bge0 interface, I can connect to it over ssh from host/local network

but I can't seem to be receiving data through the connexions I initiate

vnet or non-vnet jail?

non-vnet

I can ssh out & ssh in

but I'm trying to use rtorrent, it initiates outbound connexions but nothing is being downloaded

net.inet.ip.forwarding = 1 & security.jail.allow_raw_sockets = 1

sockstat -4 on host shows a bunch of connexions from jail:port but no data transfer happens after the handshake

Title: FreeBSD - a lesson in poor defaults

making a scripted bsdinstall, in the context of the bhyve guest, what value should i use for ZFSBOOT_DISKS?

What the heck? Why can't find find pdf files? If I do an "ls | grep -i pdf" it shows pdf files in the directory I'm searching but "find /*path to same dir* -iname pdf" shows no results.

BillyJoeBob: that's not how find works

find /path/to/dir -iname \*.pdf

i have 2 options, ada0 bhyve sata disk, and vtbd0 virtio block device, for my guest vm zfs stripe disk to use

not sure which

polyex: for vm-bhyve?

no i'm doing it manually

my client is bugged brb

that freebsd-defaults dude certainly has some opinions

meena: thanks. I'm trying to force myself more into cli stuff so I'm new to that and I wasn't making much sense of the man page.

wtf is my isp doing

hi, testing unix socket file for existence, and getting false-positive

#!/bin/sh script, [ ! -f "/var/run/mysocket.socket" ] && echo "socket not exists"

outputs "socket not exists" while socket is there

sometimes this works as intended

BillyJoeBob, I share this reference with anyone who takes an interest in the find command and why it is unique. doc.cat-v.org/unix/find-history

Title: The History of the Design of Unix's Find Command

The find command is definitely worth learning to use and using it because it is a very powerful concept.

i shouldn't give any swap to a freebsd bhyve guest right? it's running zfs on both host and guest

maybe not freebsd-specific, but related to our shell it seems, anyone know why cron running `script.sh >> logfile 2>&1` kicks back with "Ambiguous output redirect." ? Maybe cron is running csh, so i need to do `script.sh >>& logfile`. Hm. Seems like that worked. Thanks for being a rubber duck.

wew my isp taking a dump today

why dont u use devuan that is at least without a jail :>

who

can someone review this? :)) wiki.freebsd.org/Scientific any feedback is appreciated.

Title: Scientific - FreeBSD Wiki

Can I get some handholding unfucking my iohyve / bhyve networking? I think the issue might stem from moving server / interface name changing. I tried redoing iohyve setup net= bit with new interface name but it's still not working

I can tcpdump tap0 and see traffic but it doesn't seem to make it to the lan. so something tapN <-> bridge0 is wrong perhaps?

or could also be the bridge is not connected to igb0 my new interface. old interface was ... em0 maybe?

well ifconfig the bridge and see what members it has

that was actually it ... saw addm blah blah then saw that in man page and added tap0 now things maybe work

but why was it not added automatically by iohyve magic or bhyve or whatever

anyway next issue: serial console is all fucked up when trying to boot linux guest? characters not echoed properly, can't use backspace, enter doesn't make a newline, console is also extremely slow

bsd guest seems fine ish? haven't messed with it much but I could boot and install freenas

and it used to work ok on the old system?

the networking bit yes. I'm just now trying to do linux guests though. I don't think that's related to the system change

just console enough to get ssh up and do that

rtprio cool username!

I need the console to install the goddamn thing

well, not entirely true, but i understand

alternatively , how can I get docker working on freebsd? oooorrrr is there a native nvr? trying to get frigate nvr or another nvr working that's the larger overall goal here

i don't know, i don't use iohyve

you can use podman on freebsd, but of course you can't use any of the images on dockerhub

so you'll need to build it all yourself

what about ... bastille?

i don't know anything about that

qmr do you want Docker, or Docker images

because Docker doesn't work and will never work. but you can get Docker images running as Linux Jails

hmm bastille seems to be jails only

qmr it can run Linux Jails, but it doesn't boot them

you can use it to run commands, not services

if you want, I have a blog post on how to boot a complete devuan system in a Jail

qmr: wiki.freebsd.org/Containers

Title: Containers - FreeBSD Wiki

antranigv: I need *something* to use as a security camera NVR, that's the bigger goal here. frigate looks cool, which lead me to trying to boot a coreos guest so I can run docker to run frigate

Title: FreeBSD Jail booting & running Devuan GNU+Linux with OpenRC | Freedom Be With All

what's frigate? any link to that please?

qmr: i run docker in debian in bhyve and it works... just about as well as you could expect

I do that too, but for me it works flawlessly

qmr as far as I can tell, frigate needs to run on bare-metal inside docker. which means that Linux Jails will not work for you. sorry

any suggestions for security camera software? I have found some cheap cameras I can de shittify with an alternative firmware but now I need to catch the video and have recordings and configurable alerts

hmm maybe I can run frigate on kodi machine

antranigv what's the difference between docker and docker images?

how much overhead is there, if any, running docker through a linux guest in bhyve?

polyex docker is a program/daemon that you run, to manage containers on Linux. Docker images are based on the OCI spec and can be run without docker, for example with Podman. and can also be converted to, say, tarball with rootfs, that you can technically run as a Linux Jail :)

oh wow so any docker image can be run on freebsd!!!!!!!!!!!!!!!!!!!!!!!!!!!!

"any" is a bit much. most, yes

but if it relies on very linux-y things, such as frigate relies on the GPU, then no

or, NextCloud's Docker image for example relies on the Docker actually being available, it's not just an image

(it hijack's the host's Docker to run other containers)

ah

and regarding your question of overhead. if you setup everything as a bridge (if_bridge + tap + veth on Docker) then it will all work fine.

altho Docker + NAT works fine too

how's the performance relative to running debian directly on hardware?

my machines are powerful, so personally I've never felt anything being slow, but I never run Linux on the host, so how would I know? :)))

ahh

can bhyve guest vms running zfs have zfs disk encryption enabled if it's a server that should be able to start up without entering a pw on boot? how would that even work?

it doesn't. you need to decrypt/load-key before booting the VM

ok so freever servers everywhere that are running zfs are NOT using disk encryption, right?

freebsd*

well. most computers are not using disk encryption :D

I guess macOS is the exception

however, it would not be hard to setup key loading during boot

say, from a USB key, or something similar

desktops can because you give pw and login beore using it

btw tyvm antranigv for working on freebsd in the scientific computing realm

out of the... I wanna say 500 servers that I manage... I say only 10 of them use some kind of disk encryption

store credit cards or why do those 10 use disk encryption?

PII -- personal identifiable information

even things like DNA

recently discovered a shitty "roommate" or their friend stole some parts from my desktop. the drive that had all my phones synced to it. so yea disk encryption is a good thing.

ya i run disk encryption on all desktops. not yet on any servers. need to figure that out

storing the key on a connected usb drive isn't a fix imo

someone hacks into server, they just read the key info off

someone steals server, steals usb key with it

well yea

i shouldn't give any swap to a freebsd bhyve guest right? it's running zfs on both host and guest

that's up to you. swap lets the machine make better use of real ram

but host machine runs SSDs and what i read was swap on a guest results in excessive host SSD/NVME i/o and burns them out

I hear driving your car puts miles on the engine transmission and tires too