So mason, I'm happy to report back that my jails are working just fine with neither the prestop ifconfig -vnet hack nor having a sacrifical epair (although the second one might be down to just using a realtek card)

mason: In fact, my jail is even IPv6 only and it Just Worked (TM)

BaloneyGeek: nice nice - I haven't had a chance to amend the docs but that's good to hear. I'm hoping maybe tomorrow.

nice

So how do I send you my new config in a private-ish way?

mason: Also, (a) it's static IP, since my hoster doesn't do DHCP, and (b) I was wondering if I could get rid of the $ep var entirely and use jid, but jail complains ${jid} variable not found

BaloneyGeek: mason⊙bo if you want to email it

Yes, that would be the sanest option I think

BaloneyGeek: looking, half a sec

Hrm, yeah, I don't see the jail ID available right off the bat. Looking.

BaloneyGeek: How about using the name?

Unless you wanted to use the jail ID as part of the address.

BaloneyGeek: Something you can do (and I used to) is have the jails all use DHCP, also.

mason: Actually the name would be fine, but I'll email you my old (ep) based config as a base

Ideally I want to get the config to do its set-up using the in-jail system start-up scripting.

kk

I forget what wall I hit on the way there but I'll be tackling it again before long.

I'd ideally like to simply drop out the start and prestop sections here, aside from launching /etc/rc: bpa.st/CNGWM

Title: View paste CNGWM

Another thing that interests me is the notion of more modular config, and I believe someone's been working on /etc/jail.conf.d or similar.

mason: I sent you the email, but probably check your spam as well. I do my own email server but even though SPF, DKIM and DMARC are all correct email still ends up in spam from time to time

I bet it goes right through. I don't think it'll make my bayesian filter blink.

I mean I got rid of the net config in jail.conf and used rc.conf

The exec.{pre,post}stop just creates, renames and tears down the epairs

I even tested tearing down the jail with a netcat connection open, worked just fine

(the point of this server and jail is to host my very public mastodon instance so I want to do it properly)

Well - "properly" would involve using netgraph and not epairs but oh well

Ah, so, rc.conf used to be the way you'd do it, but that's deprecated now.

Oh, tell me more

I'm still struggling with a modular network configuration on freeebsd. Here I actually miss systemd-networkd somewhat

hehe

BaloneyGeek: I don't know if configuring the jails via rc.conf will ever actually go away, but the notion is that it's not the desired method for configuring jails now.

Oh so on a host I'd still configure the network using rc.conf this way?

(I spent the last several years on the team at Red Hat that fielded major problems customers had with systemd.)

mason: So how would I configure the network inside a jail?

BaloneyGeek: Ideally, you'd make the interface available, and then have a config in rc.conf inside the jail.

This should just work but my last run at it saw rc finish up with no network, and I didn't see anything sufficient logged to get a handle on why.

More effort debugging it will crack it, but I moved on. This week I'll poke at it agian.

again*

i dont really use jails much for a while..

but i was from the 'school' of those who manually started jails

since if it was broke, you dont want it doing the same thing after a reboot

but it's been a whle... like 8.x days maybe

gman999: This cropped up even with manual starts. I need to just get some debugging output saved to see what's happening.

set -x a few places, say

easier to trouble shoot when it's manual..

yes.. in shell

Yeah.

imagine it was remote and kept breaking on reboots without console?

Well. So, set -x and some sort of capture to a file. I'll figure out something reasonable.

ah, exec.consolelog

BaloneyGeek: Your config is nice and clean. Good stuff.

Thanks, I tend to err on the side of over organizing :-P

Alright, heading off for the night, but thanks for the inspiration. I'll get some time in massaging the config this week. o/

o/

mason: I ran into a lot of issues with jail networking due to issues with trying to use rc confs with NOJAILS options, like dhclient. Kept having to use the jail.conf to execute that stuff instead of getting it to just work via rc. Is that going to be changing in the future?

Ah nevermind, I misread the convo a bit. My bad.

help

exit

quit

lol

Setesh: I can use dhclient in jails. why can't you?

I've been searching for some unix/linux scripts for my dedicated server, i need get-psybnc get-eggdrop get-znc command which when the user types it into ssh it will prompt him to install the psybnc porcess on his shell account, same thing would need for eggdrop and bnc, also a command to start all bouncers would also be usefull.

what's the different between hooking a commercial console and hooking a plain monitor on a server?

mictty: what's a commercial console?

The ~/.shrc file coming with a FreeBSD installation has two lines in it that I cannot make sense of: paste.debian.net/plainh/2f8849b4

I mean, I know /home is a symlink to /usr/home.

But I can't make sense of the condition in that code.

What I find particularly confusing is `[ "$PWD" -ef "$HOME" ]`.

omg I finally got Waimea window manager working.

meena: IPMI/iLO, probably

or lantronix

meena: terminal device

Setesh: No worries. I'll document what I find, probably alongside the existing config. I want to include some innovations BaloneyGeek is using.

mason: oh? what innovations?

meena: The one that jumped out was setting a consistent name for the in-jail-side epair, so I can address it without having to do anything funny to find the interface name.

I had some ugly machinery to poke the right name into rc.conf, but this is cleaner/better.

ah, yeah. so how are epairs named then like the jail itself?

Well, just "net0" internally, for instance.

So the jail's internal rc.conf can just reference that, no hocus-pocus required.

ah, yeah

but outside?

Outside doesn't matter as much, because I can specify the right thing in jail.conf.

(i used to call them vnetX, but calling them all vnet0, unless you have more than one, makes vastly more sense)

cbsd does that, too, and on the host the epair has jail name in interface description

Yeah, that'd be fairly reasonable.

I've been encoding part of the network address locally, but if I can cleanly go back to DHCP, which I suspect I can in the next iteration, I'll use name-based ... names.

libioc did that too

mason: main reason i'd love to get away from DHCP is so i can have slimmer jails and hosts

Hm, do you trim out dhclient and/or other things?

I don't install it, unless it's needed, but i mean the CPU and memory footprint more so than storage

kk, fair enough

I run my stuff on tiny cloud instances, and I'm poor / broke

Eh, fewer cycles is a better carbon footprint, so that's reason all by itself.

yupp, that too

Even without DHCP I can have the network config entirely self-contained if I rename the inside NIC anyway.

Then as the jail moves between hosts, it doesn't need to accomodate anything different in terms of NIC naming.

another reason for SQLite, other than not wanting to become a DBA thedailywtf.com/articles/Behavioral-Deficiencies-

Title: Behavioral Deficiencies - The Daily WTF

Heh: their development strategy had gone from “cutting edge” to “barely holding an edge,”

meena: Thanks. We enjoyed that.

Is there a way to go to a port's directory and have all build and run dependencies installed via pkg? I mean without awk acrobatics?

yep

make install-missing-packages

What is it?

Ah.

man 7 ports

Thank you.

Building chromium now. With 4 Haswell cores :-/

see also: poudriere bulk -b

make install-missing-packages only fetched 2 packages and the build is now going into overdrive building dependencies from scratch. I'm still doing something wrong.

Actually not that bad, just a couple python things.

cracauer: do you have latest enabled? and why are you building chromium, if i may ask?

Yes, latest pkg. No big deal, it was a quick build of a couple Python ports.

I want/need pulseaudio in Chromium.

Actually I want jackd support, but PA on top of jack is the next best thing.

why isn't that on by default

Apparently libsnd support, which is default, clashes with pa and alsa support. At least that is what the www/chrromium/Makefile says.

cracauer: makes sense

Not sure. Myself I would be more admissive to other sound options. And I wouldn't know why a libsnd module would disable the ability to use the others.

Ah, it is because FreeBSD switches to the OpenBSD sound manager when SNDIO is being used. So it is not just another module, it is a different manager.

BSD desktop users are rare amongst rare

sound is the real factor, hard to notice as a non desktop user

FreeBSD users are spoiled by the in-kernel mixer for OSS audio. So the pressure to use a sound demon is just not nearly as intense as on Linux.

spoiled sounds good in this context

imagine iterating on seven subsystems for sound in twenty years, and this time, i hear, pipewire is going to be really good

it better be, and if it does, we should port it

oh, it already is

saas

Pipewire looks good.

Let's put it in the kernel!

cracauer: as module?

mason: I'm pinging you since you have some context on the jail networking I'm trying to do

I just realised my jail loses all IPv6 connectivity after about an hour of starting up

I did some tcpdump-ing, re0 and bridge0 on the host see ICMP packets addressed to the jail, but epair2a does not

BaloneyGeek: I don't know IPv6 very well, but any chance it's related to auto_linklocal?

mason: auto_linklocal should just create the fe80:: address and nothing else

The fe80 address is autocalculated from the interface's MAC address, and is mandatory

BaloneyGeek: Hrm, I'd recommend opening a bug for this. Anything in dmesg noting activity around the time you lose the addresses?

I didn't really check, let me try

BaloneyGeek: Randomly, this kind of thing comes up regularly and is why I'm constantly putting my FreeBSD projects on the back burner.

What's interesting is that if I ping an external host from the jail, I see stuff flowing from the epair to bridge0 to re0 and out

The reply gets stuck at bridge0 and never makes it to epair2a

after the address is "gone"?

Yes

Or ... it's connectivity, you're not losing config?

That'd be something different.

Well, gone in the sense of connectivity

Not config

The config is fine

I misunderstood.

The routing tables are fine

That's interesting. So, you see in pcaps where the reply comes in, and then it's just eaten?

Precisely

Well, tcpdump

Can you crank up logging in your firewalling?

sudo tcpdump -ni net0 'icmp6 and (ip6[40] == 128)'

BaloneyGeek: Are you just looking at tcpdump textual output?

I have no firewall

BaloneyGeek: If you write tcpdump output to a file you'll actually capture a ton more data, which you can then peruse with wireshark, tshark, whatever.

Unless there's something turned on by default

I've noted things in a full capture that weren't summarized in the textual output.

I'm way out of my depth here though, I have no idea what to look for and I have never used wireshark

Is there a simpler way, or do I need to go to school now? :-P

BaloneyGeek: Nah, do what you're doing, but write it out to a file.

-w somefile

Alright, let's see what comes up. I'll do this from the host and check re0, bridge0 and epair2a

Then if you feed that into wireshark, you'll immediately benefit.

BaloneyGeek: what's your config actually look like?

meena: It's a standard VNET jail, re0 is the physical, bridge0 is a bridge joining everything and epair2{a,b} is to connect the jail to the bridge. I can share detailed configs in a bit, but I'd have to remove real IP info

mason: maybe I should leave out the (ip6[40] == 128) and capture all ICMP packets

BaloneyGeek: Yeah, if you aren't pumping through so much traffic that it's immediately painful, I find that I shoot myself in the foot by imposing too many filters.

mason: Nope, it's just ICMP request/reply pairs in Wireshark

Well, re0 and bridge0 have pairs, epair2a just has the request

I'm thinking this is now actually an issue in FreeBSD itself. Why would IPv4 still work just fine but IPv6 not

Yeah, definitely open a bug for it.

I should migrate to IPv6 here but I've not had the free time to tackle it.

(said most of the world in unison)

mason: I think I found something

The echo requests have a source MAC of the epair interface

The echo replies have a destination MAC of the physical bridge0 interface

Now I'm confused who's the naughty boy

Hi, if I buy an old SuperMicro MicroBlade server, would it work with the latest FreeBSD if SuperMicro claims that it is compatible with FreeBSD 10.0?

In general which is the level of support of datacenter rack servers?

uskerine: I've run into bugs in SuperMicro firmware but it tends to be a safe bet.

I use old SM servers with no problems.

Just make sure they do 64 bits.

is it just my impression or SuperMicro refurbished servers tend to be cheaper than HP or others?

(that is a bit offtopic)

They're fairly cheap.

when I get the server with the caddies but not the HDs, can I use regular 3.5" HDD or do they have to be specific for servers? That is something I never fully understood in servers

server-grade disks usually have more cache than more common non-commercial offerings.

plus, the RPM of the commercial spinning disks is usually higher for commercial options

that being said, if you're using this for a lab or personal use, you can get away with cheaper disks; the RAID controller can work fine with those.

Yes I am looking to have this as my personal lab (professional but still intended to be used as a "workstation")

Any old 3.5" is fine, except SAS when you don't have a SAS controller and backplane.

uskerine: You can stick SATA in a SAS slot, but not vice versa.

good to know, thanks a lot

mason: I think it's definitely Hetzner's fault. I just rebooted the jail and tried tcpdump again, and now the ICMP replies have the correct destination MAC address and everything works

Thanks for pointing me to Wireshark :-)

Sure, enjoy. It's kind of fun.