00:08:27 By the time of late 3.x and 4.x I had a parallel port Zip drive and floppies with bad blocks, save the boot disk were history. 00:10:42 4ish and later I had a CD drive with its own ISA card interface. 00:13:24 And FreeBSD Mall was a think along with Walnut Creek's store front that did more than FreeBSD CD sets iirc. 07:31:52 was free built off the tailwinds of BSD or did it carve its own path? 14:32:05 https://docs.freebsd.org/en/books/handbook/introduction/#intro-history 17:37:04 What's policy of FreeBSD on non-free software? 17:39:19 what do you mean policy? 17:39:27 run non-free software if you want to run it 17:47:10 I would like to avoid non-free software 17:47:40 then do so 17:47:46 ok, then don't install it? what is the question 17:50:40 Usually in Linux distros package managers you can disable/enable non-free repositories to avoid accidental injection of non-free dependencies. And I wonder how is it in FreeBSD. 17:51:04 there are some switches in the ports tree to refuse to build ports of a particular license 17:51:18 you can look into that, if that's a path that would bring you joy 17:52:38 mlxdy2, or you could check whether that software is free or not before installing? 17:53:44 there's no functionalty on the pkg repo for a license excluder 17:53:55 rtprio Oh, okay 17:54:14 free vs not-free is really a social issue. you can't fix social issues with technology 17:55:06 who said anything about fixing social issues. not installing software just doesn't install software. don't read too much into it 18:08:23 DoofusCanadensis, license preference is also a social issue. 18:10:33 maybe thats the problem 18:11:16 it should be a software technical question 18:11:48 Where humans are involved, morality is involved. Software licneses have a moral aspect to them. 18:11:56 what are you doing, what is it, who can change it, and who can sell it 18:13:18 now you just need a a measure for "moralaity" 18:14:29 but generally I agree 18:15:36 The lack of an absolute measure of it makes us humans :-) 21:15:52 Hello I have a FreeBSD 15.1 server w/ 192GB RAM running bhyve and sylve GUI for 20 VMs. Every morning at 3am the server crashes. It seems that by default the daily cronjob for each VM and the server runs at the same time, overconsuming all of the RAM without releasing it, leaving only ~4gb RAM causing it to crash. I have another FreeBSD server (15.0 p9) with similar configurations but only 5 VMs and have not experienced any errors. The malfunctioning 21:15:52 server states processes are killed due to a failure to reclaim memory. is anyone aware of any possible bugs or any insight I would appreciate, thanks! 21:18:11 Are you using zfs? 21:18:26 ZFS' arc cache can use up a lot of memory 21:18:41 192G ram, must have cost a fortune 21:18:57 I have this setting in my /etc/sysctl.conf on my 128GB server: vfs.zfs.arc.max=34359738368 21:19:37 And a slightly larger setting on the 512GB server. Both had problems of oom killer killing bhyve processes randomly before tuning this setting 21:20:12 If your cronjob traverses the filesystem, that could contribute to ZFS needing more and more memory 21:20:17 do you need periodic to run in those VMs 21:20:26 Or on the host 21:20:37 yes i use zfs, for which i have a 4tb nvme and a 4tb spinning drive. but all the machines are sitting on nvme. normally all the machines don't use >16gb RAM, so the remaining should have been plenty to handle the periodic daily cronjobs 21:20:40 mary751: you can also tune cronjobs to have jitter, look into "-j" and "-J" (man 8 cron) 21:21:26 to be honest i think perioidc is a double-edged sword. it can really swamp a machine. esp if run on a host with many jails. 21:21:57 bsdrobert: I have to learn the benefit of that periodic run in VMs but again, my other server with less VM's also configured identically does not run into these errors 21:21:59 bsdrobert: been there, done that, now I use 30sec jitter for my cronjobs everywhere 21:24:00 karolyi:thank you I have not heard about jitter I will have to look into it. I manually staggered the daily cronjobs in three different batches at different time intervals but it does not seem the memory gets cleaned after periodic daily runs 21:26:11 tuning the ARC by hc's suggestion might also be a good idea 21:26:54 hc: that is very helpful, thank you, I am using zfs and have not limited the arc size. I need to learn more about how cronjob traverses filesystem (by the way, my host has zfs and the VMs are zfs). anything else I could look into? 21:29:59 mary751: Not really; the one I mentioned is the one setting that I needed to adjust on all my FreeBSD bhyve hosts. Since FreeBSD switched its zfs implementation to the zfsonlinux one, the kernel sometimes cannot communicate quickly enough to the zfs subsystem that it needs more memory 21:30:35 With that setting you're limiting it proactively 21:30:50 the periodic does a lot of stuff, if you don't need security reports within your VMs, turn them off. there is a security report and a daily report that will traverse all your files, checking for possible security issues, reports that normally only few people read 21:32:32 karolyi: alright I need to get educated on what periodic security reports looks like, I don't even know where they are located. Can netdata view them, or do i need to view them manually, or web interface, or just text? 21:32:54 mails to root by default iirc. 21:32:54 I can imagine how those simply overload ZFS if you have a lot of them running at the same time 21:33:04 ( With "not really" I meant I can't think of anything else, not that there isn't anything else ) 21:33:25 mary751: cat /etc/crontab, ls /etc/periodic/ 21:33:35 thanks so much guys I'll be back in an hour or so 21:35:28 my jail template basically comments out everything in the crontab so the jails don't hammer my server, and then I enable things on a custom basis. some jails don't even have cron running 21:35:43 just some ideas to ponder on 21:35:49 The zfs cache is very memory hungry, it basically uses all of the available free memory. Kind of like the idle process uses up all idle cpu time. If the zfs cache doesn't release the memory quickly enough, well... 21:35:54 But I've never heard of zfs getting 'overloaded' 21:36:11 i lean towards just turning periodic off. it doesn't have defaults that scale. 21:36:38 hc: imagine 20 VMs doing it independently of each other at the same exact time 21:36:52 it can easily become an OOM in my opinion 21:37:08 karolyi: Why? Are the VMs overprovisioned? If not, should be fine 21:37:29 hc: that's not a question I should answer. :) 21:37:43 It wasn't mentioned in the initial question so I take it that is not the case 21:37:55 then again, overprovisioning is relative to what else you have running at the same time 21:38:53 for example, you might have provisioned the VM's memory allocations well but the periodic run on the host machine can spike ARC usage and then all of a sudden, OOM 21:39:21 who knows to be honest, it's not like we see a log of it after the crash 21:39:45 But the arc cache should never cause an oom situation. It didn't use to when the old zfs implementation was in use 21:41:18 to be honest if it's proven not overprovisioning, I'd start suspecting the memory, and running memtest. I've seen weird crashes because of faulty memory, and it only surfaced when I started loading up on it 21:41:32 (of course, because it actually started storing data that got corrupteD) 21:41:54 Or you just try limiting the arc cache first ;-) 21:42:17 ya, as a first measure. and also adding cron jitter 21:42:30 or turning unnecessary shit completely off 21:42:35 I've never used cron jitter. Then again, my guests all use xfs or ext4 21:42:48 Turning unnecessary things off is good, but finding the root cause is even better :-) 21:43:37 the problem isn't zfs, but hammering the CPU at the same exact moment for no necessary reason. those jobs usually can run within a 30 second window 21:43:45 that's my case 21:43:59 You mean rowhammering the CPU? 21:44:33 nah, just overloading stuff unnecessarily. row hammering might also happen but I don't think because of cron :) 22:31:12 oh what, zuck is here. 22:33:06 was 23:00:10 karolyi: i was wondering if it could be a faulty memory. perhaps I should reboot the server and run memtest in bios 23:03:53 mary751: or a memtest usb stick/iso 23:16:00 mary751, . 23:26:52 also, ventoy for booting it 23:59:13 Ventoy, or the brick of a CD-RW drive and a blank :-?