stupid question, but I couldn't find an answer to it, does zfs checksum zvols?

scrubbing the disk would find corrupt data within zvols right?

polarian: yes

cool! Just had to make sure, otherwise my plans would have been ruined :)

but (obviously) it can't identify individual files inside a zvol so if it's corrupt, that's it

also on the offhand you're using zvols for bhyve, be aware flat files are faster

ivy: zfs can still recover if there is spare copies of the blocks, or I believe I can also zfs send zvols to another machine and pull it if there is ever a need to recover?

ivy: I know flat files are faster, but zvols are easier :)

I actually discussed it at EuroBSDCon

polarian: yes it can recovery if there are other copies of the blocks

recover

in order to separate file disks for each different vm, you would need a separate dataset for each one, which is just a headache... but hey ho maybe I will try the faster way at some point :P

pkg can not be installed: "A pre-built version of pkg could not be found for your system."

when using "df -h" to view disk information, i noticed the sum of used isn't equal to sum of each used item, where does the disappear disk space go ? paste.centos.org/view/c943170b

xxy you're on ZFS. OpenZFS have snapshots, compression, metadata...Normally the "sum" won't add up.

devnull: i didn't set any shapshot yet .

snapshot

xxy zfs list -t snapshot. But there is also compression and metadata. All of this can interfere. See: zfs get all zroot/ROOT/default

devnull: if freebsd as a desktop for daily use , then zfs isn't the best choise, is this right? paste.centos.org/view/51d426e5

xxy at least I saw 3 properties: compressratio, compression and checksum activated, and it can cause the difference. This explain logicalused property value differ from used value. Also, you can use zfs list -o space to see more detailed information

xxy, and no. I use freebsd as desktop for daily use, and I use OpenZFS

It depends your needs

If you are not familiar with OpenZFS and the features it offers, choose the UFS file system.

should there be a default route inside a jail?

mns vnet jail?

learn ZFS

it may take a while, but when it clicks, you will be happy you learned it

devnull: i love freebsd ,it's brand-new for me , i am contact with it only several days, and need times to get used to. paste.centos.org/view/7e93a87a

xxy: try bastille jail manager

(thats what i recommend)

vm-bhyve for virtual machines

xxy: and good luck, freebsd is great.. hope you find your way

devnull: I don't think its a vnet jail. I created it using bastille, but its just a regular thin jail.

mns your default router inside the jail needs to be the IP Address from your main network interface (em0/ena0), if your jail needs to communicate with internet. Otherwise, the jail will not reply to the main host packets sent.

btw i just learned the other day that nintendo switch is also freebsd

mns netstat -rn to see the routes.

xxy, nothing wrong with your Z file system.

devnull: netstat -rn does not give me a default route. the jail used to work under 13.4-RELEASE, but after upgrading to 14.1-RELEASE is no longer working.

mns, what specifically is not working?

devnull: can not reach the outside world from inside the jail, hence pkg does not work inside the jail

mns, unless you use ipv4/ipv6 = inherited, you need to set defaultrouter. Generally it is defined in rc.conf, but I don't use bastille, so I don't have certainly if bastille define default router there.

xxy, Even on Linux not zfs df and du don't add up: gnu.org/software/coreutils/faq/core…and-du-report-different-information

devnull: bastille does not define it there.

I think its time to bite the bullet, move to creating jails using jail.conf.d

rwp: i get it , thanks.

xxy, There is also a standard customized zfs list for disk space. zfs list -o space

mns well, try to define default route inside de jail. jexec yourjail, route delete default, route add default IP_FROM_YOUR_MAIN_em0_INTERFACE. I don't use bastille, the things can be different.

I prefer Vanilla jails.

devnull: I keep getting Operation not permitted messages when trying to delete or add the route.

mns, as root?

as root inside the jail

yeah I think I want to switch to vanilla jails, probably a good time to do so is now.

mns, can I see the flags for ls -lo /sbin/route ?

mns, Previously I read in the docs that bastille creates a VNET jail if you give it the -V option. Did you give it the -V option to create a VNET jail? You can only run route setup commands in VNET jails.

devnull: in the jail?

Otherwise the network stack is shared with the host and it is up to the host to set up networking.

yes

rwp: I don't believe I created the jail with -V. It was a year and half ago :-)

Getting errors like that in the jail indicates to me that it is not a vnet jail and in that case the networking is shared with the host and the route is set on the host.

devnull: -r-xr-xr-x 1 root wheel uarch 56072 May 31 09:05 /sbin/route

Yes rwp, it makes sense.

rwp: I'm pretty sure you're right, that its not a vnet jail. my bastille config isn't setup in a manner that would be needed for vnet

I am surprised that bastille didn't create the jail and also assign an IP address in a way that would Just Work.

I have fallen in love with vnet jails because it is one step further along in what appears as a virtual machine. And then there is no problem with port collisions.

But at the same time a vnet jail will pretty much require a full network administrator's knowledge to set it up properly.

all of this happened after upgrading to 14.1-RELEASE from 13.4-RELEASE, which I did last week. Otherwise it did Just Work.

Hmm... When I upgraded from 13 to 14 I didn't notice any change in behavior for any of the jails.

I upgraded host from 13.4 -> 14.1, then noticed errors for libssl from inside the jail, and started upgrading the jail from 13.2 to 14.1, and that is when I saw issue I'm having.

I have some jails with shared network with host (ipv4=inherited) and some vnet jails...In both cases I had no problems with the upgrade. I hope you did the zfs snapshot before upgrade? hehe

I just started up an older 12.3R non-vnet jail that hasn't been started since upgrade and it started and has no trouble with networking.

I must have done something incorrect in my upgrade process

So a non-vnet jail simply uses the hosts' networking directly. With an additional IP address "alias" attached to the network. The default route of the host is the same default route as it is global.

Example from my system with the jail running now: paste.debian.net/plain/1332317

That's the view from the host system. The main IP is 192.168.230.122 and after starting the jail it has an additional IP 192.168.230.48 alias associated. And then the route is really almost unchanged. It adds another host route entry for the 192.168.230.48 host-route.

This is what it looks like from inside the jail: paste.debian.net/plain/1332318

You can ignore the vm-public because that's for vnet jails running from vm-bhyve and I should have deleted it from the paste so as not to confuse.

You very well might have an upgrade problem with the jail though. And that might have snarled things up now. I don't know.

for me, bastille creates a bridge and that, along with pf rules, does the control of the traffic

I don't know anything about bastille so I don't know how it functions but what you say seems reasonable. My example I pasted is not using a bridge. But using a bridge seems reasonable too. We always use a bridge with vnet jails and with bhyve.

Yes, I don't know how bastille works. In vnet jails I create the bridge and epair interfaces.

mns , btw, how is your jail.conf entry?

I'm creating one :-) I didn't have anything before as bastille handled it all

bastille handles the creation of the bridge and epair interfaces

mns Does bastille take care of jail upgrade too?

devnull: its supposed to, this is the first time I've done a major OS upgrade for it. Otherwise its just a minor upgrade

Someone saw koobs here?

devnull: nickserv says seen 100 weeks ago

woof, hope he's alright

devnull, rwp: do you use thin jails or classic thick jails?

Classic thick jails. So much easier to manage.

You don't need thin jails until you know you need thin jails. Because you have a thousand of them running for example.

but I can also have thick and thin running side by side as well

Of course.

can nullfs be used with thick jails?

Sure. Why not?

mns it depends. Mostly thick jails. But for development tests (new ports), thin jails. So I can clone many of them to spare disk space.

mns yes, you can

I have things like certificates that I would want to use amongst the jails and some data directories that I would want to share between host and jails

nullfs would come in handy for that.

I have a vanilla 14.1-RELEASE jail up and running! yay!

\0/

but no access to the outside world. I followed the handbook to create this. hmm seems like the bridge0 didn't get an ip address.

mns: Sounds like you're using VNET? You'll likely need to set up some firewall rules to allow jails access out of the subnet from the bridge.

Something like: boucek.me/blog/freebsd-jails-with-vnet-and-nat maybe?

i downloaded Netbeans binary file under ordinary directory "~/Netbeans". when running java program, it can't invoke console and terninal is invalid, it might a privilege issue, but i don't know how to do in detail

mns, congrats :)

rwp I successfully booted the mfsbsd ISO in qemu and attached to the disk (gpart show /dev/ada0). I don't have network connectivity from the VM but I believe this is not necessary. zpool-import(8) reports: cannot import 'zroot': pool was previously in use from another system. [...] The pool can be imported, use 'zpool import -f' to import the pool.

-- not sure what happens now. -f would force the import, but how is that going to interfere with future mounting of the pool by the real™️ system once I fixed it all?

hey all, im creating some VIMAGE based jails because I want to use one to connect to an openvpn server. I want to this via the tun300 device (which is created via cloned_interfaces in the jails rc.conf) and in my devfs.rules I have: dpaste.org/w9bqj in my jail startup I am of course doing: devfs_ruleset=11;

however when the jail is started tun300 cannot be seen in /dev or via ifconfig, but it does exist, if I do a 'ifconfig tun300' it is shown. What have I got wrong here?

moviuro: if it's boot pool, it will be imported anyway by that real system

so no worries in forcing the import?...

moviuro: yep

ok, so the pool is mounted on /mnt , what now? The server refuses to boot, but zfs-scrub(8) and smartctl(8) don't show anything wrong

I have reinstalled the bootcode with gpart(8) . I'll try booting the system from qemu by passing the drive

boot seems to take a very long time querying CDROM (there's none)

but it does boot. Let's try rebooting to the drive for real™️

yay, it worked

sweet, my 4-port serial card just arrived and it seems to be working great

nice :) cu time

chatting from my text terminal atm

Does '0' value for 'vfs.zfs.arc.max' and 'vfs.zfs.arc.min' means unlimited amount of RAM to be used, for both?

i found this. Is it accurate?

what is `nameserver 8`. do you mean `nameserver 8.8.8.8`?

ouch sorry, i scrolled too much

mns yes, I did something similar. I use nullfs to share certificate files from my xmpps erver to other xmpp server admins. So they can only have access to xmpp ceftificates, not other certificates: So in mount.fstab: /usr/local/etc/letsencrypt/archive/myxmppdomain /usr/local/jails/jxmpp/nullfs_certs nullfs ro 0 0

Of course I could use rsync in cron task...but...well, I prefer this way.

Nice step-by-step ek! (freebsd-jails-with-vnet-and-nat)

hjf: is that a PCI(e) card? did you happen to test if FreeBSD can use it as system console?

devnull: is that a link to the step-by-step? didjn't come in as an URL

moviuro, Yay! Very good to hear you were able to recover it. Seems the root cause of the boot failure was needing to update the bootcode? That all makes sense then. And you recovered it all! Good! :-)

mns the step-by-step was sent by ek: boucek.me/blog/freebsd-jails-with-vnet-and-nat

devnull, did you do that?

devnull, be careful with vnet. Any time you stop a vnet jail on freebsd there's a chance the whole kernel will panic

?

it could be true, but 10+ years ago sfox

mzar, no it's been back for the last couple of releases and still affects 14.1-RELEASE

OK, but I am not affected

sfox: are you ?

yes

any idea why you aren't affected?

I don't know

somehow my few vnet jails gained immunity

TBH I like jails without vnet more

yeah but jails without vnet will leak onto your management vlan

and use the hypervisor's ip address

I am not much worried about it

you have no network isolation whatsoever can jails can use ip addresses not even bound to it, including IP addresses from other jails and the hypervisors inside of different vlans

huge security hazard

it's rahter not a bug, BTW, it's more like a flaw in design

it's a class of security vulnerability called vlan-hopping

heh..

what about jail without IP address

I haven't tested that but I think it could still work

there's no security bounds check to make sure if you request to use ip, that the ip actually belongs to you

it's still useful, but you need new loopback interface for such a jail

so I don't think it would make a difference

100% safe, can spoofing IP address on loopback be abusive ?

sfox: BTW 270492 seems to be more like feature request than real bug

what about spoofing IP address on another jail or hypervisor's loopback?

I can see how that could be abused

the scope is limited, it will not brak things sfox

i disagree

by 270492 I meant PR 270492 on Bugzilla

the agreement is not required

sfox I'm not going to stop doing something simply because there was a bug once. Bugs must be fixed.

I hope so. I'm not sure what's taking so long to fix this one or what the problem is. Unfortunately I'm not skilled enough to fix it myself.

I'm just warning you about the kpanics

maybe you know what's wrong and how to fix it?

If it's just a matter of gathering more information I have a spare system set aside for testing

sfox I mean, many people says "I was able to reproduce this issue", but mostly is other issue. Most people do not have the knowledge to investigate further and confirm that it is the same issue. They just get a kernel panic and they say "hey, I have the same issue", but probably is different issue. The causes for a kernel panic, a coredump or page fault to occur can be varied.

yes, vnet jails are usable nowadays

Do you know who maintains the code that is failing?

btw, someone linked this ticket as related bugs.freebsd.org/bugzilla/show_bug.cgi?id=279653

I think reviews.freebsd.org is down too. I get error 502

maybe hold off until that ticket gets resolved.

yup rwp. I would have had the certainty if I had just tried to boot the system from qemu first (I beelined to getting mfsbsd running instead)

moviuro, Ah! I hadn't thought of that direct qemu booting option. Well regardless you have gained a more general purpose skill that might help you in a different situation in the future.

And you saved your data and system. Even better! :-)

"That debian rescue system isn't that useless after all" "don't forget about bootcode" "all hail mfsbsd"

;)

ivy: yes PCIe card, no i haven't tried it as a system console. it appears as "cuau" (not cuaU)

uart3@pci0:3:0:1: class=0x070002 rev=0x00 hdr=0x00 vendor=0x125b device=0x9100 subvendor=0xa000 subdevice=0x1000, vendor = 'Asix Electronics Corporation', device = 'AX99100 PCIe to Multi I/O Controller', class = simple comms, subclass = UART

this is what it looks like

i like the octopus style cable

hey all has anyone got an example of nat firewall for ipfw or pf

that uses fibs

I have three external gateways em0/fib0 ng0/fib1 and ng1/fib2

all is fine for em0, but I cannot use ng0 or ng1 at all

I attempted to set it up just to use either with both firewalls and just cannot seem to get it to work

dpaste.org/vGEfp was the last attempt with ng0/fib 1

I can use setfib 0/1/2 ping 1.1.1.1 to send and receive via any of the interfaces

I also am not using setfib at all for natd, im not sure if I have to or not it seems to make no difference

hey guys

sorry disconnected while messing with the firewall, if anyone responded please prod me :)

I imagine I will be bouncing on and off all night, so if I appear dead (this is a irc bouncer) could you please respond on forums.freebsd.org/threads/pf-or-ip…-basic-nat-firewall-with-fibs.95355

time to try pf and rtables -_-

daemon: for ipfw, i don't think there should be much to change in your ruleset relative to a normal ipfw nat configuration. you will just need 3 ipfw nat config invocations, and your ipfw nat rules will need to be specific to interface

i might be misunderstanding your use of setfib though, since i haven't used it

got it working with pf :)

still no luck with ipfw and setfib / fib etc :(

OK

"setfib 1 ip from any to any out keep-state :default" looks suspicious, since if the packet is already out, i would think the routing decision has been made and the fib is no longer relevant

it seems inbound packets get the fib correctly but outbound do not even leave the firewall

0 matches in ipfw -a list

dpaste.org/dOdYx was what worked for pf

em0 for any jails or vms or the freebsd gateway its self

ng0 for the lan and ng1 for my system

hmm, i'll try this on ipfw some time and if i can get it to work, i'll contact you

cheers :) if possible please reply in the forum as there was quite a few posts with people looking for a solution involving fibs

im going to add my pf one to the thread

I'm having trouble figuring out why llvm fails to build for me. Does the error shown mean I need to update TLS certs or the build requires network access?

why is a build even bothering with TLS certs in the first place. Seems like a bug IIMHO

expired cert?

The jail I'm building in the 14.1-RELEASE-p5 and the ports tree is the latest master branch as of yesterday around this time

I don't know if the host system's pkg's and OS matter much?

but even that's no older than 3 -4 months

revoked then

But that's not a problem on my end though?

IDK - so the source tree of what you compile contain any certs?

s/so/do/g

not that I know of. It's freshports.org/devel/llvm15

there seems to be a branch cut for that every quarter - you could try to go back to an earlier one to see, if the error disappears

guess I'm gonna update to the latest 14-stable from the pkgbase repo to see if that makes any change regarding this issue with drm-61-kmod

though I'm almost wondering if there is some incompatibility between 14-stable and the binary of drm-61-kmod available from the repo, like if I should try compiling it from ports or something

VT: Driver priority 0 too low. Current 101 fbd0: not attached to vt(4) console; another device has precedence (err=17)

whoops, there should be a newline in there

but that's the error I'm getting when booting up, after which the screen just stays blank since the modesetting VT didn't initialize properly

I'm wondering how I can find out what this "other device" is that has precedence because there's only a single GPU in this laptop

Ah, I didn't notice this before, but the problem appears to be related with sphinx-doc: "The full traceback has been saved in /tmp/sphinx-err-d77h61lg.log, if you want to report the issue to the developers."

Still, the question is why would sphinx-doc be messing around with TLS certs during a build. hmmm

yep, every port that's using py-sphinx is showing the same problem

"Sphinx uses requests as a HTTP library internally. If tls_cacerts is not set, Sphinx falls back to requests’ default behaviour. See SSL Cert Verification for further details."

seems to be an odd way of doing things, but it definitely does some TLS gymnastics