Hi I'm trying to use the new service jails but they don't seem to start at all

Is it only specific services that work with it?

Ohh I see, you have to actually enable the service as usual before too. I just assumed if you had <servicename>_svcj="YES" it would enable the service to run it in the jail

Anyone have experience lumping mixed sized disks together into a single large ZFS zpool (raidz2 of similar) ?

SponiX: Yeah. You just build the ZFS into partitions sized for the smallest disk.

It wastes a little space but it works well.

Is there a better way, like grouping a handful to a dozen of the same sized disks as one like vdev, and then lumping the vdevs together from there?

My smallest disks are 3TB, and my largest are like 16-20TB

you could probably do that with ccd(4), but you wouldn't be able to boot from it, and i dislike that sort of complexity in my storage, personally

Yeah, I have a couple of 2TB NVME drives to do the OS/boot itself. These array(s) would be specifically for data/Media only

ivy: I didn't know this ccd existed. It might be helpful

I would rather do it all within ZFS, but mixing it with ccd could actually give me better solutions

afaik, diffrent sizes for root vdevs don't matter, but you don't have redundancy between those root vdevs, only inside a vdev (and only if it isn't just a single disk)

SponiX: riffing off of what nimaje noted, you might string together things that are at least mirrors, and that might use more of your available space while still protecting your data.

heya. i can't for the life of me figure out pf. i'm trying to set up my rules so that i block all traffic except explicitly a few ports, and then i block all traffic for one user, and reroute its traffic to a wireguard vpn. however... the line `block user <user>` seems to screw up the rest of the config, stopping me from doing basic stuff like ping (permission denied), etc. what's up?

the pf.conf in question: paste.c-net.org/InducedPiercing

hmjsp: Hm, I didn't know you could tie rules to users. That's kind of interesting.

Neat.

it's not working tho ToT

hmjsp: Is it just that user who's impacted for the rest of the rules? Or everyone?

everyone is impacted for some reason, that's what's puzzling me

if i uncomment the last two lines, ping works. if i uncomment "block user transmission"... then even as root i can no longer ping

which doesn't make sense, as man page example is similar: block out proto { tcp, udp } all

pass out proto { tcp, udp } all user { < 1000, dhartmei }

yeah, just looking at that now

Is it your syntax? You don't match the man page there.

I'm just getting into pf, although it's a lot like the ipf I used to use.

im not sure, dont think so

block all user { transmission }

doesn't change anything

it's weird, i copied a simple ruleset (not this one) from an openbsd box and it did not work the same

not sure why

OpenBSD is a different, sometimes newer PF. But that syntax is still not quite the same. Try exactly what they show maybe. At a guess, block out proto { tcp, udp } all user { transmission }

Although maybe {} is just a list syntax and you don't need it for a single entry? Dunno.

that's what i thought too

that's how it usually is

grep -r user . shows nothing un /usr/share/examples/pf on 14, looking on 15

oh!

it works?

with proto...

Does it?

yeah... mysterious O_o

hold on lemme double check

Good idea.

But yeah, when in doubt, slavishly adhere to the examples and only deviate from there once you have something that works.

Also, I wonder if what you really want is a block if what you really want is to redirect/twist/whatever.

ok so transmission is blocked and my user can ping fine... now i need to check vpn config

i want to block all trafic that isnt redirected through the vpn interface

That was anglolinguicide, but I wonder if maybe what you want is a modification if you catch packets from the user, not a block and then later something else.

hmjsp: tcpdump might help here, seeing what packets make it where

oops my client pooped itself

(I forget if it's last-rule-wins and you have to specify "quick" of if that's some other syntax.)

it's last rule wins

so block by default

and then pass through vpn

quick can be used to override last rule wins

ah, so this *is* a system where you want "quick" if you need to apply immediately?

right

good

yeah

I'm playing around with some simple PF this weekend. I've been mucking about with ipfw but it doesn't really make me happy.

if the idea is to force all your torrent traffic through a vpn, you may want to consider running transmission in a different fib, rather than trying to do it with a firewall

ivy++ even better idea

a different what now

FWIW, I found this the other day and was blown away by now minimal it was: blog.frankleonhardt.com/2020/networking-freebsd-jails

im new to freebsd :')

hmjsp: routing table == fib

fib = routing table... let me find an example

man.freebsd.org/cgi/man.cgi?query=setfib is useful

i mean it's sunken cost atp i just want this to work lol

hmjsp: I think ivy is right - you want to impose a routing table for this.

probably the best approach but... i may as well learn why my current pf approach is not working while im at it

hmjsp: FWIW, at a fairly recent job, I was able to do something with setfib in seven lines that was way more complex with other options. I was compelled to do it on Ubuntu despite it being way more effort for no additional functionality, despite my showing a lovely demo using FreeBSD.

hmjsp: people.freebsd.org/~ivy/app-in-fib.txt

ivy: can i do <any process>_fib=<number>?

is that how this works?

it needs to be a service that exists in rc.d, so anything that you do foo_enable=YES, you can set foo_fib

waow, neat

so simple

see rc.conf(5), there's a bunch of service-specific options

suppose it's not a service tho (not my scenario atm, but im curious), what would i do?

setfib 1 <command>

waow

okay, thanks!

i'll try what you just showed me and see if it works

hmjsp: Here's some random thrashing I was doing during the process mentioned. Basically, I needed different default routes for different things listening on the same /24. Look how easy it was with FreeBSD: termbin.com/i94r

note, if you're using wg-quick, you'll need to do it a different way... probably using 'ifconfig <interface> fib 1' as a post-up command. i prefer not to use wg-quick as i find it simpler to configure that in rc.conf

mason: the sheer reduction xD

ivy: yeah i avoided wg-quick because it routes the whole network to the vpn

which is not what i want at all

hmjsp: I made it work in Ubuntu, but it wasn't the right call.

wait ivy im still confused. i got rerouting to wg0 correctly now with fib, but im a bit lost on where to put the peer's ip now...? since it's not in the wireguard config itself...

hmjsp: the wireguard config doesn't change, you're just putting the local ip address in rc.conf. (that's the address you'd put in Address= if you were using wg-quick)

in place of this? "ifconfig_wg_vpn="inet 1.2.3.4/32 fib 1""?

yes, replace 1.2.3.4/32 with the local ip address for inside the tunnel (not your public address)

ivy: still doesn't work. tcpdump on wg0 shows sending, but no receiving

is the tunnel actually working? 'wg show wg0' should show traffic in both directions

yeah it does. in the kbs, but it does

transfer: 2.61 KiB received, 110.67 KiB sent

if you run tcpdump on the internet interface, do you see traffic for the vpn endpoint in both directions or only outgoing?

when i run ping, only outputting. tho previously there was SOME input, presumably the server checking if im alive?

but with ping it's 100% packet loss

hmjsp: ICMP can be funny - also try netcat/telnet to various ports

in both directions

is it possible pf is blocking traffic? you may want to add a pass rule for the wg interface and remove any user-specific rules just for testing

perhaps

lemme disable pf and see

ok it's definitely pf xD

hmjsp: Did you leave in the user block rule?

nop

not sure what it is

pass on $vpn_if proto tcp from any to any port $tcp_services

i added this at the end, and it still doesn't work

im not sure what's blocking it in the first place

well, 'proto tcp' won't allow ping

hmjsp: Are you testing with more than just ping? Try telnet too, to various ports. From both directions.

pass on $vpn_if proto { tcp, udp } from any to any port $tcp_services doesn't work either

mason: im trying curl and ssh also

non vpn traffic works

hmjsp: Always simplify. Those aren't bad ideas, but plain telnet is simplest. It'll send a SYN wherever.

okok

just not too familiar

telnet foo 1234

or nc foo 1234 I guess

this probably isn't related to the current issue, but you can't really do port-specific outbound filtering for BT because peers can use any port they want... so that pass rule probably wants to be a pass in rule, with a blanket pass out

ah ok thank you

oh!

doing a blanket pass out made everything work

\o/

tcpdump got a... dump of inbound traffic

xD

okay this is good. thanks a lot yall :)

fib is neat

yes indeed it is

is it only freebsd with it?

or other bsds?

same on OpenBSD

Linux has a similar feature but it works in a strange, harder-to-use way for no apparent reason

mason: neato

Hm, trying to figure out how to specify different tables with that NetBSD thing.

linux is very silly

oh well im way overdue on sleeping

mason: i'm not sure routed(8) is the right manpage? that's the RIP daemon

hmjsp: You can do similar with Linux using namespaces.

but i can rest easy with this solved

gn :)

ivy: Yeah, I saw "tables" and it seemed like it might be the right thing, but I'm thinking it's not.

hmjsp: enjoy

o/

we have routed too, although if i have my way, it'll be gone before 16.0 :-)

I'm not immediately seeing how to go about it in NetBSd.

MultiProtocol Label Switching seems interesting, but isn't quite right.

mpls is for the other sort of vpn :-) (virtual wires)

Well. I have and like setfib, so I won't worry about it too much I guess. :)

incidentally, one of this year's GSoC projects is an MPLS implementation for freebsd

Oh, neat.

Port of NetBSD, or ground up?

i'm not sure, the project idea mentions the openbsd and netbsd implementations but doesn't directly suggest porting those. although it would seem silly not to take at least some code

mm