00:09:14 stupid question, but I couldn't find an answer to it, does zfs checksum zvols? 00:09:29 scrubbing the disk would find corrupt data within zvols right? 00:10:34 polarian: yes 00:10:46 cool! Just had to make sure, otherwise my plans would have been ruined :) 00:10:52 but (obviously) it can't identify individual files inside a zvol so if it's corrupt, that's it 00:11:09 also on the offhand you're using zvols for bhyve, be aware flat files are faster 00:11:47 ivy: zfs can still recover if there is spare copies of the blocks, or I believe I can also zfs send zvols to another machine and pull it if there is ever a need to recover? 00:12:04 ivy: I know flat files are faster, but zvols are easier :) 00:12:24 I actually discussed it at EuroBSDCon 00:12:57 polarian: yes it can recovery if there are other copies of the blocks 00:13:01 recover 00:13:13 in order to separate file disks for each different vm, you would need a separate dataset for each one, which is just a headache... but hey ho maybe I will try the faster way at some point :P 00:45:22 pkg can not be installed: "A pre-built version of pkg could not be found for your system." 02:10:57 when using "df -h" to view disk information, i noticed the sum of used isn't equal to sum of each used item, where does the disappear disk space go ? https://paste.centos.org/view/c943170b 02:31:12 xxy you're on ZFS. OpenZFS have snapshots, compression, metadata...Normally the "sum" won't add up. 02:31:58 devnull: i didn't set any shapshot yet . 02:32:06 snapshot 02:34:28 xxy zfs list -t snapshot. But there is also compression and metadata. All of this can interfere. See: zfs get all zroot/ROOT/default 02:36:41 devnull: if freebsd as a desktop for daily use , then zfs isn't the best choise, is this right? https://paste.centos.org/view/51d426e5 02:41:54 xxy at least I saw 3 properties: compressratio, compression and checksum activated, and it can cause the difference. This explain logicalused property value differ from used value. Also, you can use zfs list -o space to see more detailed information 02:42:35 xxy, and no. I use freebsd as desktop for daily use, and I use OpenZFS 02:42:57 It depends your needs 02:44:06 If you are not familiar with OpenZFS and the features it offers, choose the UFS file system. 02:44:33 should there be a default route inside a jail? 02:44:56 mns vnet jail? 02:47:42 learn ZFS 02:48:10 it may take a while, but when it clicks, you will be happy you learned it 02:48:53 devnull: i love freebsd ,it's brand-new for me , i am contact with it only several days, and need times to get used to. https://paste.centos.org/view/7e93a87a 02:50:15 xxy: try bastille jail manager 02:51:45 (thats what i recommend) 02:51:58 vm-bhyve for virtual machines 02:52:27 xxy: and good luck, freebsd is great.. hope you find your way 02:52:39 devnull: I don't think its a vnet jail. I created it using bastille, but its just a regular thin jail. 02:53:06 mns your default router inside the jail needs to be the IP Address from your main network interface (em0/ena0), if your jail needs to communicate with internet. Otherwise, the jail will not reply to the main host packets sent. 02:53:18 btw i just learned the other day that nintendo switch is also freebsd 02:54:09 mns netstat -rn to see the routes. 02:54:45 xxy, nothing wrong with your Z file system. 02:54:58 devnull: netstat -rn does not give me a default route. the jail used to work under 13.4-RELEASE, but after upgrading to 14.1-RELEASE is no longer working. 02:55:29 mns, what specifically is not working? 02:56:17 devnull: can not reach the outside world from inside the jail, hence pkg does not work inside the jail 02:59:30 mns, unless you use ipv4/ipv6 = inherited, you need to set defaultrouter. Generally it is defined in rc.conf, but I don't use bastille, so I don't have certainly if bastille define default router there. 03:00:16 xxy, Even on Linux not zfs df and du don't add up: https://www.gnu.org/software/coreutils/faq/coreutils-faq.html#df-and-du-report-different-information 03:00:30 devnull: bastille does not define it there. 03:01:12 I think its time to bite the bullet, move to creating jails using jail.conf.d 03:02:26 rwp: i get it , thanks. 03:02:32 xxy, There is also a standard customized zfs list for disk space. zfs list -o space 03:05:25 mns well, try to define default route inside de jail. jexec yourjail, route delete default, route add default IP_FROM_YOUR_MAIN_em0_INTERFACE. I don't use bastille, the things can be different. 03:07:35 I prefer Vanilla jails. 03:08:44 devnull: I keep getting Operation not permitted messages when trying to delete or add the route. 03:09:23 mns, as root? 03:09:30 as root inside the jail 03:09:59 yeah I think I want to switch to vanilla jails, probably a good time to do so is now. 03:17:05 mns, can I see the flags for ls -lo /sbin/route ? 03:18:10 mns, Previously I read in the docs that bastille creates a VNET jail if you give it the -V option. Did you give it the -V option to create a VNET jail? You can only run route setup commands in VNET jails. 03:18:18 devnull: in the jail? 03:18:24 Otherwise the network stack is shared with the host and it is up to the host to set up networking. 03:18:25 yes 03:18:57 rwp: I don't believe I created the jail with -V. It was a year and half ago :-) 03:19:29 Getting errors like that in the jail indicates to me that it is not a vnet jail and in that case the networking is shared with the host and the route is set on the host. 03:20:16 devnull: -r-xr-xr-x 1 root wheel uarch 56072 May 31 09:05 /sbin/route 03:20:25 Yes rwp, it makes sense. 03:21:04 rwp: I'm pretty sure you're right, that its not a vnet jail. my bastille config isn't setup in a manner that would be needed for vnet 03:21:29 I am surprised that bastille didn't create the jail and also assign an IP address in a way that would Just Work. 03:21:45 I have fallen in love with vnet jails because it is one step further along in what appears as a virtual machine. And then there is no problem with port collisions. 03:22:12 But at the same time a vnet jail will pretty much require a full network administrator's knowledge to set it up properly. 03:22:24 all of this happened after upgrading to 14.1-RELEASE from 13.4-RELEASE, which I did last week. Otherwise it did Just Work. 03:22:59 Hmm... When I upgraded from 13 to 14 I didn't notice any change in behavior for any of the jails. 03:23:57 I upgraded host from 13.4 -> 14.1, then noticed errors for libssl from inside the jail, and started upgrading the jail from 13.2 to 14.1, and that is when I saw issue I'm having. 03:25:30 I have some jails with shared network with host (ipv4=inherited) and some vnet jails...In both cases I had no problems with the upgrade. I hope you did the zfs snapshot before upgrade? hehe 03:27:41 I just started up an older 12.3R non-vnet jail that hasn't been started since upgrade and it started and has no trouble with networking. 03:28:36 I must have done something incorrect in my upgrade process 03:29:18 So a non-vnet jail simply uses the hosts' networking directly. With an additional IP address "alias" attached to the network. The default route of the host is the same default route as it is global. 03:30:19 Example from my system with the jail running now: https://paste.debian.net/plain/1332317 03:31:22 That's the view from the host system. The main IP is 192.168.230.122 and after starting the jail it has an additional IP 192.168.230.48 alias associated. And then the route is really almost unchanged. It adds another host route entry for the 192.168.230.48 host-route. 03:32:17 This is what it looks like from inside the jail: https://paste.debian.net/plain/1332318 03:32:54 You can ignore the vm-public because that's for vnet jails running from vm-bhyve and I should have deleted it from the paste so as not to confuse. 03:33:52 You very well might have an upgrade problem with the jail though. And that might have snarled things up now. I don't know. 03:35:28 for me, bastille creates a bridge and that, along with pf rules, does the control of the traffic 03:36:46 I don't know anything about bastille so I don't know how it functions but what you say seems reasonable. My example I pasted is not using a bridge. But using a bridge seems reasonable too. We always use a bridge with vnet jails and with bhyve. 03:38:16 Yes, I don't know how bastille works. In vnet jails I create the bridge and epair interfaces. 03:38:53 mns , btw, how is your jail.conf entry? 03:39:33 I'm creating one :-) I didn't have anything before as bastille handled it all 03:39:57 bastille handles the creation of the bridge and epair interfaces 03:42:53 mns Does bastille take care of jail upgrade too? 03:49:03 devnull: its supposed to, this is the first time I've done a major OS upgrade for it. Otherwise its just a minor upgrade 03:54:07 Someone saw koobs here? 03:56:59 devnull: nickserv says seen 100 weeks ago 04:03:33 woof, hope he's alright 04:08:33 devnull, rwp: do you use thin jails or classic thick jails? 04:09:03 Classic thick jails. So much easier to manage. 04:09:23 You don't need thin jails until you know you need thin jails. Because you have a thousand of them running for example. 04:09:56 but I can also have thick and thin running side by side as well 04:37:53 Of course. 05:00:46 can nullfs be used with thick jails? 05:08:53 Sure. Why not? 05:12:31 mns it depends. Mostly thick jails. But for development tests (new ports), thin jails. So I can clone many of them to spare disk space. 05:14:22 mns yes, you can 05:19:49 I have things like certificates that I would want to use amongst the jails and some data directories that I would want to share between host and jails 05:19:56 nullfs would come in handy for that. 05:20:24 I have a vanilla 14.1-RELEASE jail up and running! yay! 05:24:29 \0/ 05:27:08 but no access to the outside world. I followed the handbook to create this. hmm seems like the bridge0 didn't get an ip address. 05:40:28 mns: Sounds like you're using VNET? You'll likely need to set up some firewall rules to allow jails access out of the subnet from the bridge. 05:41:06 Something like: https://www.boucek.me/blog/freebsd-jails-with-vnet-and-nat/ maybe? 07:56:48 i downloaded Netbeans binary file under ordinary directory "~/Netbeans". when running java program, it can't invoke console and terninal is invalid, it might a privilege issue, but i don't know how to do in detail 08:15:36 mns, congrats :) 10:16:01 rwp I successfully booted the mfsbsd ISO in qemu and attached to the disk (gpart show /dev/ada0). I don't have network connectivity from the VM but I believe this is not necessary. zpool-import(8) reports: cannot import 'zroot': pool was previously in use from another system. [...] The pool can be imported, use 'zpool import -f' to import the pool. 10:16:02 -- not sure what happens now. -f would force the import, but how is that going to interfere with future mounting of the pool by the real™️ system once I fixed it all? 11:52:29 hey all, im creating some VIMAGE based jails because I want to use one to connect to an openvpn server. I want to this via the tun300 device (which is created via cloned_interfaces in the jails rc.conf) and in my devfs.rules I have: https://dpaste.org/w9bqj in my jail startup I am of course doing: devfs_ruleset=11; 11:52:29 however when the jail is started tun300 cannot be seen in /dev or via ifconfig, but it does exist, if I do a 'ifconfig tun300' it is shown. What have I got wrong here? 12:06:34 moviuro: if it's boot pool, it will be imported anyway by that real system 12:07:36 so no worries in forcing the import?... 12:10:18 moviuro: yep 12:19:57 ok, so the pool is mounted on /mnt , what now? The server refuses to boot, but zfs-scrub(8) and smartctl(8) don't show anything wrong 12:34:05 I have reinstalled the bootcode with gpart(8) . I'll try booting the system from qemu by passing the drive 12:41:11 boot seems to take a very long time querying CDROM (there's none) 12:44:59 but it does boot. Let's try rebooting to the drive for real™️ 12:48:01 yay, it worked 13:03:32 sweet, my 4-port serial card just arrived and it seems to be working great 13:03:52 nice :) cu time 13:04:59 chatting from my text terminal atm 14:26:01 Does '0' value for 'vfs.zfs.arc.max' and 'vfs.zfs.arc.min' means unlimited amount of RAM to be used, for both? 15:10:57 https://cdn2.nuegia.net/d4a184b3-90e0-4339-bff7-cd1640d59cfb/apple_donating_24_dollars_to_freebsd.png 15:11:02 i found this. Is it accurate? 15:16:05 what is `nameserver 8`. do you mean `nameserver 8.8.8.8`? 15:16:29 ouch sorry, i scrolled too much 15:21:36 mns yes, I did something similar. I use nullfs to share certificate files from my xmpps erver to other xmpp server admins. So they can only have access to xmpp ceftificates, not other certificates: So in mount.fstab: /usr/local/etc/letsencrypt/archive/myxmppdomain /usr/local/jails/jxmpp/nullfs_certs nullfs ro 0 0 15:23:23 Of course I could use rsync in cron task...but...well, I prefer this way. 15:24:49 Nice step-by-step ek! (freebsd-jails-with-vnet-and-nat) 15:51:59 hjf: is that a PCI(e) card? did you happen to test if FreeBSD can use it as system console? 16:13:56 devnull: is that a link to the step-by-step? didjn't come in as an URL 16:29:58 moviuro, Yay! Very good to hear you were able to recover it. Seems the root cause of the boot failure was needing to update the bootcode? That all makes sense then. And you recovered it all! Good! :-) 17:07:09 mns the step-by-step was sent by ek: https://www.boucek.me/blog/freebsd-jails-with-vnet-and-nat/ 17:44:20 devnull, did you do that? 17:45:31 devnull, be careful with vnet. Any time you stop a vnet jail on freebsd there's a chance the whole kernel will panic 17:45:51 ? 17:46:20 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=238326 17:46:27 it could be true, but 10+ years ago sfox 17:46:56 mzar, no it's been back for the last couple of releases and still affects 14.1-RELEASE 17:47:41 OK, but I am not affected 17:47:53 sfox: are you ? 17:47:58 yes 17:48:10 any idea why you aren't affected? 17:48:28 I don't know 17:48:50 somehow my few vnet jails gained immunity 17:49:43 TBH I like jails without vnet more 17:50:27 yeah but jails without vnet will leak onto your management vlan 17:50:39 and use the hypervisor's ip address 17:50:57 https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=270492 17:51:18 I am not much worried about it 17:52:14 you have no network isolation whatsoever can jails can use ip addresses not even bound to it, including IP addresses from other jails and the hypervisors inside of different vlans 17:52:26 huge security hazard 17:52:45 it's rahter not a bug, BTW, it's more like a flaw in design 17:53:35 it's a class of security vulnerability called vlan-hopping 17:54:19 heh.. 17:54:29 what about jail without IP address 17:54:49 I haven't tested that but I think it could still work 17:55:19 there's no security bounds check to make sure if you request to use ip, that the ip actually belongs to you 17:55:21 it's still useful, but you need new loopback interface for such a jail 17:55:28 so I don't think it would make a difference 17:56:50 100% safe, can spoofing IP address on loopback be abusive ? 17:57:43 sfox: BTW 270492 seems to be more like feature request than real bug 17:57:59 what about spoofing IP address on another jail or hypervisor's loopback? 17:58:06 I can see how that could be abused 17:58:54 the scope is limited, it will not brak things sfox 17:59:17 i disagree 17:59:20 by 270492 I meant PR 270492 on Bugzilla 17:59:48 the agreement is not required 18:13:17 sfox I'm not going to stop doing something simply because there was a bug once. Bugs must be fixed. 18:14:05 I hope so. I'm not sure what's taking so long to fix this one or what the problem is. Unfortunately I'm not skilled enough to fix it myself. 18:14:20 I'm just warning you about the kpanics 18:14:30 maybe you know what's wrong and how to fix it? 18:15:24 If it's just a matter of gathering more information I have a spare system set aside for testing 18:20:00 sfox I mean, many people says "I was able to reproduce this issue", but mostly is other issue. Most people do not have the knowledge to investigate further and confirm that it is the same issue. They just get a kernel panic and they say "hey, I have the same issue", but probably is different issue. The causes for a kernel panic, a coredump or page fault to occur can be varied. 18:21:29 yes, vnet jails are usable nowadays 18:21:37 Do you know who maintains the code that is failing? 18:21:49 btw, someone linked this ticket as related https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279653 18:22:40 I think reviews.freebsd.org is down too. I get error 502 18:24:01 maybe hold off until that ticket gets resolved. 20:05:33 yup rwp. I would have had the certainty if I had just tried to boot the system from qemu first (I beelined to getting mfsbsd running instead) 20:07:35 moviuro, Ah! I hadn't thought of that direct qemu booting option. Well regardless you have gained a more general purpose skill that might help you in a different situation in the future. 20:07:52 And you saved your data and system. Even better! :-) 20:08:16 "That debian rescue system isn't that useless after all" "don't forget about bootcode" "all hail mfsbsd" 20:08:23 ;) 20:15:05 ivy: yes PCIe card, no i haven't tried it as a system console. it appears as "cuau" (not cuaU) 20:17:12 uart3@pci0:3:0:1: class=0x070002 rev=0x00 hdr=0x00 vendor=0x125b device=0x9100 subvendor=0xa000 subdevice=0x1000, vendor = 'Asix Electronics Corporation', device = 'AX99100 PCIe to Multi I/O Controller', class = simple comms, subclass = UART 20:18:11 https://http2.mlstatic.com/D_NQ_NP_2X_901786-MLU73628992175_122023-F.webp 20:18:16 this is what it looks like 20:18:37 i like the octopus style cable 20:46:22 hey all has anyone got an example of nat firewall for ipfw or pf 20:46:26 that uses fibs 20:46:45 I have three external gateways em0/fib0 ng0/fib1 and ng1/fib2 20:46:52 all is fine for em0, but I cannot use ng0 or ng1 at all 20:47:09 I attempted to set it up just to use either with both firewalls and just cannot seem to get it to work 20:47:31 https://dpaste.org/vGEfp was the last attempt with ng0/fib 1 20:47:54 I can use setfib 0/1/2 ping 1.1.1.1 to send and receive via any of the interfaces 20:48:25 I also am not using setfib at all for natd, im not sure if I have to or not it seems to make no difference 21:05:54 hey guys 21:19:42 sorry disconnected while messing with the firewall, if anyone responded please prod me :) 21:21:57 I imagine I will be bouncing on and off all night, so if I appear dead (this is a irc bouncer) could you please respond on https://forums.freebsd.org/threads/pf-or-ipfw-basic-nat-firewall-with-fibs.95355/ 21:22:02 time to try pf and rtables -_- 21:47:04 daemon: for ipfw, i don't think there should be much to change in your ruleset relative to a normal ipfw nat configuration. you will just need 3 ipfw nat config invocations, and your ipfw nat rules will need to be specific to interface 21:47:38 i might be misunderstanding your use of setfib though, since i haven't used it 21:49:59 got it working with pf :) 21:50:07 still no luck with ipfw and setfib / fib etc :( 21:50:16 OK 21:51:01 "setfib 1 ip from any to any out keep-state :default" looks suspicious, since if the packet is already out, i would think the routing decision has been made and the fib is no longer relevant 21:52:25 it seems inbound packets get the fib correctly but outbound do not even leave the firewall 21:52:28 0 matches in ipfw -a list 21:53:03 https://dpaste.org/dOdYx was what worked for pf 21:54:58 em0 for any jails or vms or the freebsd gateway its self 21:55:06 ng0 for the lan and ng1 for my system 21:59:07 hmm, i'll try this on ipfw some time and if i can get it to work, i'll contact you 22:06:07 cheers :) if possible please reply in the forum as there was quite a few posts with people looking for a solution involving fibs 22:06:14 im going to add my pf one to the thread 22:44:03 I'm having trouble figuring out why llvm fails to build for me. Does the error shown mean I need to update TLS certs or the build requires network access? 22:44:04 http://pkg.ent.morante.net/poudriere/data/14amd64-monthly-desktop/2024-10-14_20h55m50s/logs/errors/llvm15-15.0.7_10.log 22:46:25 why is a build even bothering with TLS certs in the first place. Seems like a bug IIMHO 22:48:59 expired cert? 22:53:17 The jail I'm building in the 14.1-RELEASE-p5 and the ports tree is the latest master branch as of yesterday around this time 22:53:55 I don't know if the host system's pkg's and OS matter much? 22:54:12 but even that's no older than 3 -4 months 22:56:05 revoked then 22:57:22 But that's not a problem on my end though? 22:58:05 IDK - so the source tree of what you compile contain any certs? 22:58:16 s/so/do/g 22:58:34 not that I know of. It's https://www.freshports.org/devel/llvm15/ 23:03:29 there seems to be a branch cut for that every quarter - you could try to go back to an earlier one to see, if the error disappears 23:09:29 https://github.com/freebsd/freebsd-ports/tree/main/devel/llvm15 23:16:40 guess I'm gonna update to the latest 14-stable from the pkgbase repo to see if that makes any change regarding this issue with drm-61-kmod 23:17:24 though I'm almost wondering if there is some incompatibility between 14-stable and the binary of drm-61-kmod available from the repo, like if I should try compiling it from ports or something 23:20:51 VT: Driver priority 0 too low. Current 101 fbd0: not attached to vt(4) console; another device has precedence (err=17) 23:21:17 whoops, there should be a newline in there 23:22:00 but that's the error I'm getting when booting up, after which the screen just stays blank since the modesetting VT didn't initialize properly 23:23:29 * ober finds no way to mount an sdcard reader showing up as da0 instead of mmcs 23:24:28 I'm wondering how I can find out what this "other device" is that has precedence because there's only a single GPU in this laptop 23:41:41 Ah, I didn't notice this before, but the problem appears to be related with sphinx-doc: "The full traceback has been saved in /tmp/sphinx-err-d77h61lg.log, if you want to report the issue to the developers." 23:43:22 Still, the question is why would sphinx-doc be messing around with TLS certs during a build. hmmm 23:47:18 yep, every port that's using py-sphinx is showing the same problem 23:52:24 https://www.sphinx-doc.org/en/master/usage/configuration.html 23:52:37 "Sphinx uses requests as a HTTP library internally. If tls_cacerts is not set, Sphinx falls back to requests’ default behaviour. See SSL Cert Verification for further details." 23:53:35 seems to be an odd way of doing things, but it definitely does some TLS gymnastics