00:01:04 megaTherion: you said mmap isn't my issue but others looked info subject much more deeply that i ever can and found that mmapping a file from zfs could lead to wasted memory 00:01:17 which would be "fun" 00:02:03 that would manifest itself much more quickly if you don't have much to waste 00:50:28 Ay 00:56:33 ketas: mapping a file from vfs is actually allowed and a feature 00:56:58 and that you let your system starve for memory is your decision, not those of others 01:04:32 there the question wasn't what's available but more like is it reasonably used... as far as i know mmap function allows one to use file as it were memory, it doesn't mean stick it all into ram immediately for fast access and kill sshd off 01:04:36 no? 01:05:02 *lol* 01:05:09 unsure what zfs does in order to do the same 01:05:09 dude please get that freakin more 4GB RAM and be done with it :) 01:05:20 you are wasting days on a problem which isnt a problem but a totally expected situation 01:06:10 maybe it could be fixed without harm? 01:06:33 Cloud VPS, 8GB RAM 4,99 EUR XD 01:06:49 this has affected machines with more than 4g ram 01:06:56 no for software you need RAM, for even more complex software you need even more RAM 01:08:02 previously it was, if you want it faster, add more ram? 01:08:45 if your system is swapping and OOM then it's not about speed but about working correctly 01:09:33 but what if it's not swapping 01:09:47 and still going oom 01:09:50 then its not 01:09:51 what then 01:11:34 so why not get more RAM? I really dont understand the issue 01:11:43 whats keeping you from adding more than 2GB RAM? 01:13:16 might as well change entire machine then, really 01:13:21 thats fine too 01:13:24 or change the FS, also possible :) 01:13:27 UFS is still a thing 01:13:31 but that's not real answer 01:13:38 you didnt ask a real question 01:13:42 yeah people tried ufs, worked 01:14:22 OS' behave differently if they are OOM, its a design question but I bet almost all have an OOM killer which then goes for hunt 01:14:28 if you want to know the details read the sourcecode 01:14:45 it's not because of mmap, its an side effect - the OS has no other choice 01:16:08 it's fine to use entire free ram for caching purposes, it makes everything blazing fast 01:16:24 well it has choice to not do it? 01:16:40 that doesn't affect anyone with 3t ram 01:17:05 you dont need 3TB RAM, 8GB is totally fine ... how much you actually need depends on the zpool layout and if its about zraid etc. 01:17:33 it cannot not do it, it's the job of the OS to hand out memory if applications demand it 01:17:50 well it can say no' 01:17:51 ? 01:17:54 no it cant 01:18:02 read about operating system design and those principals 01:19:03 the actual situation depends how it arises, but if its inside the kernel then it must kill other software in userland 01:19:22 if you try to alloc some huge amount of RAM in a single program then of course that program may be killed or denied the request 01:19:38 but then in the end... what good is an OS which has no memory at hand 01:19:53 if its about user restriction then yes you can restrict those limits in FreeBSD 01:19:55 it can decide that now zfs works slower because there's no memory left 01:20:10 well it's not a program 01:20:24 it would swap then 01:21:11 hmm 01:21:11 well then maybe investigate how Solaris or OpenSolaris descendents behave, thats where ZFS is coming from in the first place 01:21:36 how they do it? 01:21:42 it's too complex too 01:21:44 I dont know, I never read the source code 01:22:09 maybe Solaris can handle an less RAM situation better - who knows 01:25:25 would be fun if this is valid bug when using mmap and with lot of small files and with low ram and with zfs and with slow io 01:25:32 like perfecf storm or so 01:25:48 its no valid bug 01:26:00 the person XYZ coding C and using mmap has zero control about the file system which is used 01:26:03 there is no relation 01:26:17 it's a trigger? 01:26:22 well who knows 01:27:27 its a totally normal function, almost every, if not every OS has it. Not sure how it's called on Windows but on Mac OS you also have an mmap 01:27:31 its just totally nrmal 01:27:32 +o 01:29:35 is there something in dmesg if the problem appears? 01:29:43 s/if/when 01:30:12 i don't think there was 01:30:15 as in...? 01:30:22 io error? :) 01:30:35 no about oom 01:31:04 only the fact that things were killed now 01:32:10 i wonder if tuning oom features would change anything 01:32:16 or not 01:32:35 because what needs memory is kernel 01:34:20 it was worse before actually i recall 01:34:38 maybe vm.overcommit changes something for you, dont know 01:34:46 unsure if it made zfs worse now 01:34:53 i hope not 01:35:26 there are also other ways kernel can take memory 01:35:46 should it always say yes? 01:36:07 maybe you want to read tuning(7) 01:39:25 maybe thay manpage needs augmented :p 01:39:52 to be 01:42:15 really never had a lot of issues, my NAS has 16GB with a zraid1... what always helps is limiting the ARC to a sane level 01:42:27 but I have other machines though non of them has less than 8GB with ZFS 01:43:31 nowadays it's doing it for you i read 01:43:39 the sane arc 01:43:49 can be... yeah 01:43:54 0.6*ram iirc 01:53:13 zfs actually works in 512m too :p 01:54:07 it was even unexpectedly fast 01:54:26 as long as there is no load.. sure 01:55:57 it started falling over due hw faults 01:56:17 dead now 01:56:55 lost half the disk too due i was lazy and didn't add 2nd disk nor did regular backups 01:57:06 which is another problem 01:57:31 well I usually always have zraid's except for some single VPS instances where I use zrepl to replicate my stuff over wireguard 01:58:21 yeah now it's fixed 01:58:53 damn git is too good stress test tool eh 01:58:55 :p 01:59:56 well it does read and write million small files and likes them cached as well for speed? 02:00:04 never ran into the issue, I guess you can also use rsync as replacement? 02:00:17 I'd expect that those programs have kind of similar behaviour 02:00:25 they do not 02:00:38 that's the surprise 02:01:02 using rsync to sync over terabytes will not do it 02:01:11 well you could get a systrace from it, then you can compare 02:01:29 far past what recommend ram can do... yet no failures 02:01:50 oh that seems like meh hard 02:01:58 what to even look for? 02:02:00 FreeBSD has truss and friends 02:02:10 well you said mmap was evil, so maybe rsync doesn't do mmap? 02:02:26 at least you'd see how it looks when both applications "copy" files 02:02:27 it probably has no reason to do so? 02:02:32 but i don't know 02:02:50 or you can read the soure code of course 02:02:54 or it has or it does 02:02:55 +c 02:04:02 right now git has 02:04:05 [core] packedGitWindowSize = 8m packedGitLimit = 32m preloadIndex = false [diff] ren 02:04:08 ameLimit = 100 02:04:31 meh forgot the split 02:04:33 but anyway 02:04:43 this "fixes" it 02:05:09 it didn't go deeead slow either 02:07:17 rsync doesn't mmap 02:07:36 btw whole issue started when there was huge change at once happening in ports tree, which is massive repo for what i usually use 02:07:53 openrsync does and we've had some problems with that, but we've mostly managed to 'fix' it 02:09:01 what is the 'fix' for this? 02:10:07 dovecot nearly does it as well... that thing also has legit reason for this 02:10:10 hmm 02:10:22 using dovecot too 02:11:37 * kevans reads scrollback a bit more 02:11:40 also, abnormally large for machine maybe... tho dovecot did it whenmi tried to tune it upwards 02:15:22 I guess from the context it sounds like mmap is being accused of eating a bunch of memory, but that's not really how that works 02:15:26 index related values there were 128m...256m which i probably wrongfully assumed to not do anything bad because it's small on disk or in ram 02:15:52 mmap helps to eat it up? 02:16:04 not any more than usual, really 02:16:38 you can mmap a file but you still have to fault in pages/blocks/etc, it's not really common to prefault everything in 02:17:16 I mean, yeah, there's some trade-off there, but the odds of it being whatever anomaly your observing aren't really likely 02:17:17 what's the actual issue here i wonder 02:17:47 you're 02:17:53 :p 02:18:25 anomaly-ketas 02:18:43 it's only 21:18, I shouldn't really be oops'ing yet 02:27:34 run git without those limits, wait until git actually starts doing that particular unknown activity, hopefully you had top open somewhere before, observe wired memory shooting quickly in few seconds to 94% of ram, probably went past it... end result is no userland, kernel works just fine after it, you just only have limited ways to interact with it, like ping or keyboard 02:27:48 how do i know what happened 02:27:52 :/ 02:31:32 what release/branch is this? 02:32:02 zfs default arc max is something like all but 1GB of RAM, but it should be responsive to pressure to relieve some of that if the system needs it 02:32:17 how much ram are we talking total? 02:32:35 limiting ARC is always a good idea 02:34:35 13, it started 13.2 or so, now 13.4 still seems to have other similar issues... maybe it's fixed in 14 or current, do you know? 02:34:38 and ram is 4g 02:35:46 while i have tried to use zfs with 512m, wasn't kernel panic festival at all 02:36:45 i even tried to look any offending openzfs bugs but that thing is huuuge and i don't even know what to look 02:36:51 not zfs issue at all? 02:37:11 having not enough ram is no bug ;) 02:38:58 well it rather sounds like perfect storm than tiny ram 02:39:22 and was observed elsewhere 02:48:42 funnily i tried to allocate ton of ram, tried some file tests, including mmap, increasing limits slowly so i could still observe it, i couldn't find anything, even arc and whatever else was in wired was scaled down and only test that ended up in actual failure was when just trying to consume ram, and what was killed was just tester itself 02:49:01 can't figure out smarter tests 02:53:04 if it wouldn't hide it, i could run some tests in small vm or so, easier and safer 03:35:23 Hello folks, im having a demential issue maybe you can help me to get out of it :) 03:36:05 after upgrading from 14.1 to 14.2-p3 my FreeBSD desktop (zfs) is going total unusable after 1 day of uptime or so 03:37:18 this system is always on, i had 122 days uptime before upgrade to 14.2, never had a problem like this on FreeBSD desktops so i suspect its a ZFS kernel module problem someway 03:38:17 in brief: after +/- 20hs of using the system the desktop starts freezing every time you do something 03:39:39 first thing i checked up was my NVMe, used nvme tools from base userland, S.M.A.R.T, etc the disk is in perfect conditions also ran the NVMe RW testing commands, seems neat. 03:40:44 also, when the system starts to get UNUSABLE i just reboot it and everything goes just fine, so I/O problem discarded 03:41:16 can anybody guide me to debug the HANGS/FREEZING on the kernel/openzfs.ko? thanks in advance :D 03:43:02 i think its ZFS related because, my ARC is the same as always (1G) i dont even need to do any heavy R/W to reach unusable state, just happens after 1 day of uptime more or less 03:43:07 how did you diagnose the issue to zfs problem? 03:44:08 rtprio: i need help to discard ZFS, but i think it is because the only thing that would change along time is the ZFS ARC CACHE, thats my guess 03:44:54 badkat: well, use bectl to boot back to 14.1 and confirm that it stays stable or still freezes 03:45:08 kernel memory is not exhausted neither, i mean there is 20G free memmory why would be hang like that 03:45:38 rtprio: i did that, the same day that i concluded it was not the NVMe 03:45:46 14.1 goes ok 03:46:12 i would love to try to report the bug, maybe contribute something to fix it :D 03:47:06 fun part is that when freezing starts, audio keeps going like nothing 03:56:15 so kernel works well still 03:56:45 how does it freeze, again? 03:59:15 also where the audio comes from 04:00:09 ketas: for example, you are playing music in background while coding in your IDE, reading articles on firefox or just operating some server over SSH. I call it "freeze" when suddenly i just can keep hearing the audio from the browser or VLC player but cant do nothing more. 04:00:42 not even change to other TTY 04:01:09 just xorg thing or more? 04:02:27 ketas: thats the tricky part for me, not sure what originates this.. 04:02:52 can't ssh into that from elsewhere? 04:03:08 ketas: i restarted xorg some day to see if it was related to some QT library/AMD gpu driver and it was doing the same 04:03:33 and? 04:03:40 was there a change 04:03:48 no 04:04:05 or fresh xorg still choke up from time to time 04:04:06 thats why i think on ZFS ARC is corrupted or something i dont know 04:04:08 hmm 04:04:25 what about no x 04:04:40 you were able to kill it 04:04:54 same issues in console? 04:05:38 ketas: yes, freeze lasts less but still hangs weirdly the commands 04:05:51 hmm 04:06:03 if i navigate folders, or open files while thats happening it doesnt show any I/O problem 04:06:47 open top -qaHSPs1 and systat -v 1 and so there if not tried them 04:06:51 hmm 04:07:03 the freeze effect is random but not constant, it just evolves to UNUSABLE, day 3 of uptime is totally messed up 04:07:14 ok 04:07:27 would be fun to watch it evolve 04:07:28 i tried iostat 04:07:48 maybe over ssh 04:07:49 and when freeze starts the iostat doesnt even log that moment 04:08:03 if it finally locks up 04:08:26 maybe it's not io 04:09:07 nice idea with ssh, i could try that this weekend, i work in that setup so i restarted each morning XD 04:09:33 so finally... what happens? 04:09:36 freezes? 04:10:30 just becomes unusable, but keeps responding after the hangs from 10sec to 2minutes 04:10:38 hmm 04:10:40 crazy 04:11:07 wanna see that top during this 04:12:14 ok i will try it, thanks for the suggestion 04:12:47 and what happens after a pause 04:12:54 or before 04:13:14 maybe tail -F some log too 04:14:15 i did, nothing in /var/log 04:17:13 well did it freeze yet? 04:17:32 if it doesn't show up in any of those 3 places 04:17:34 it's weird 04:18:14 maybe i dont have ZFS debug messages enabled in the right way 04:18:23 all need separate window too since frozen system won't be able to do tmux window change 04:19:10 its working fine now, i just restarted like 1 hour before i did came here 04:19:19 i have time till monday to debug it 04:19:56 zpool iostat -v 1 & zpool status -svv for zfs 04:20:19 i will try to make some big R/W to see if i can make it freeze again 04:20:27 ok i will add those too 04:20:29 for interesting things to look of 04:20:38 rw does it? 04:21:22 and this was some flash? 04:21:42 ketas: not sure what does it 04:21:52 single nvme one? 04:21:58 yes, Samsung EVO NVMe 04:22:10 yeah kind of hard 04:22:16 to find cause 04:22:22 i have no mortality signs on the drive 04:24:34 ketas: thanks for `zpool iostat` thats way better than plain iostat 04:25:06 also, with -svv switch i see this: "Some supported and requested features are not enabled on the pool." 04:25:40 and recommends to run `zpool upgrade`, i have this zfs pool from 2023 IIRC 04:26:03 that upgrade could be risky? 04:26:20 maybe pool is from pre v5000 + flags time 04:26:29 or 04:26:31 hmm 04:26:45 upgrade wasn't needed anymore iirc? 04:29:08 or it was different upgrade 04:29:55 i checked the zfs logs 04:30:08 its from 13.1 04:31:51 v5000 is much older 04:32:15 but i'm not sure, it still might use manual upgrades 04:32:19 betas! 04:32:22 er ketas 04:32:28 just i haven't see it yet 04:32:48 upgrade would be safe if you don't go back 04:32:57 unsure if you could 04:33:28 also, freezes came after 14.1->14.2?! 04:33:44 you can leave the zpool as old now i guess 04:33:50 cpet is back 04:34:14 ketas: exactly, i had 120+ days uptime, then i said "Oh new release, lets upgrade" :P 04:34:27 uptimes are useless now 04:34:37 any service that has an uptime of more than 60 days you question it 04:34:56 perfect reason to test upgrades before using 04:35:03 just like I question my firewall with an uptime of 48 days 04:35:05 cpet: lmao i dont turn my computers off xD 04:35:14 it could be specific stupid bug 04:35:14 I don't either 04:35:21 whatever reasoned 04:35:27 but I like to be updated 04:35:33 and if that requires n reboot 04:35:35 then so be it 04:35:38 betas can wait 04:35:40 er ketas 04:36:00 could it be the applications.. 04:36:20 ? 04:36:26 betas can wait! 04:36:36 well there is an IDE, firefox, vlc player 04:36:37 betas the ketas 04:36:51 which ide, also? 04:36:56 no software would require a reboot 04:36:59 and firefox is one stupid hog 04:36:59 cpet: well i always try to update asap, thats why i did 14.1 -> 14.2 not 11.0 :P 04:37:01 only a kernel update 04:37:11 anyway wait until 04:37:17 freezes 04:37:19 then test 04:37:20 ketas: we need firefox 04:37:25 i would wait to see if it happens again.. and just monitor the apps 04:37:30 cause that other browser hasn't been updated in years 04:37:37 what was it called netsurf ? 04:37:53 maybe leave sshs open with all above commands 04:38:06 zpool status via topless or so 04:38:07 have it mail you if the app dies 04:38:21 "yo foo your app crashes here is the core dump" 04:38:31 well firefox is only way yes 04:38:34 pretty much 04:38:39 I use safari 04:38:47 my android phones do ff too 04:38:49 the only reason why I don't update is Firefox is just as bad :P 04:39:43 the issue with updating zfs is the possibility of the system not booting after update due to forgetting to reinstall the boot stuffs 04:39:48 so I have never bothered to upgrade 04:40:14 pool you meant? 04:40:29 yes i guess it's fine 04:40:37 either of you still need to reinstall the boot code 04:40:55 and that has a possibility of having a 50/50 you can reboot 04:40:59 where does the bootcode go? 04:41:05 heaven 04:41:10 or you mean updated one? 04:41:24 I have always updated the boot code after any updates with zfs 04:41:32 now is that good practice who knows but that's what I do 04:41:43 voy4g3r2: could be, i wonder if something that goes through the ARC cache corrupts it,dont know if that could be even feasible, just thinking in the air.. 04:42:19 unsure if it can corrupt 04:42:39 machine did pass at least one full memtest86+ run, right? 04:42:41 anything can corrupt something 04:42:45 and buildworld 04:42:50 maybe 04:42:54 as a good measure 04:42:56 yeah.. well the whole corrupt thing.. is drawing at straws. 04:43:04 honestly if it can do a make build world without a core dump its good enough 04:43:06 does the freebsd logs say anything? 04:43:08 ketas: i did memtest yes, stick is fine 04:43:22 freebsd-version fetch install ones that is 04:43:23 meatiest is not a really good way to test 04:43:29 er memtest 04:43:34 meatiest! 04:43:50 betas a beer a day keeps the docker away 04:43:57 hah 04:44:01 for now on betas is betas 04:44:06 wtf 04:44:15 now that made me giggle 04:44:26 voy4g3r2: no, im sure i can improve the logs tho, preparing now for the shitty moment 04:44:28 you didn't mean docker? 04:44:43 docker is evil 04:44:57 any mention of it you'll become one of the ketas 04:45:27 voy4g3r2: what exactly is the issue 04:45:39 if you want to retell the story while I drink Guinness 04:45:42 would be appreciated 04:45:49 cpet: no issue.. helping badkat 04:46:01 beer world records 04:46:06 badkat: what is the issues ? 04:46:09 i am over here.. just doing a poudriere run, updating my mac with brew upgrade and doing some python code.. all good 04:46:27 I can't sleep 04:46:38 cpet: he has machine running xorg freezing up for 10s...2m 04:46:42 after a day or so 04:46:48 which one 04:46:50 exactly, thanks ketas 04:46:53 after 14.1->14.2 04:46:55 voy4g3r2: or badkat 04:46:58 has zfs on nvme 04:47:09 that's all 04:47:29 which video card brand ? 04:47:32 it's stupid bug 04:47:42 cpet: AMD GPU (Renoir) 04:47:51 model? 04:48:03 so, it did continue to happen after xorg was killed off? 04:48:26 cpet: Ryzen 7 PRO 4750U 04:48:27 badkat: how long have you used a BSD ? 04:48:38 cpet: 7 years 04:48:51 started in 10.2 or so with my servers 04:48:51 ok so you know you'll always have issues with the latest swizzle right ? 04:48:57 moved to desktop 4-5 years ago 04:49:04 23y here 04:49:12 never had this kind of problems 04:49:15 I have been using *BSD for 30 yrs 04:49:18 i miss solaris zfs 04:49:27 I miss opensolaris 04:49:37 you can hug illumos lmao 04:49:48 hug openzfs 04:49:48 that's old 04:50:11 ketas: im trying xD 04:50:29 have you recompiled xorg+ with debug 04:50:40 and use all the debugging system to help you debug why it crashes ? 04:50:42 don't try to hold openzfs down with hug 04:50:43 :p 04:51:19 cpet: i have this weekend to try that kind of troubleshooting 04:51:26 is this the onboard GPU r ext ? 04:51:26 2025-04-26 04:04:54 < ketas> same issues in console? 04:51:27 2025-04-26 04:05:38 < badkat> ketas: yes, freeze lasts less but still hangs weirdly the commands 04:51:33 im building the kernel now and checking for debug options 04:51:37 doesn't this absolve X of guilt? 04:52:07 well you can wait for fail before 04:52:10 or does it start in X and then continue 04:52:10 have you tried current 04:52:29 my desktop that runs current is 2020 04:52:30 cpet: onboard GPU 04:52:36 so in computer terms old 04:52:46 i have one armv7 current machine here 04:52:52 could do more 04:53:08 I was unhappy when rasp pi 5 has such shitty support all over 04:53:12 i have similarly stupid bug here 04:53:13 so I sold my rasp pi 5 04:53:26 how much did you get 04:53:27 rpi are not good SBC's for freebsd IMHO 04:53:34 badkat: how come you haven't invested in a AMD vid card ? 04:53:45 rasp pi 3 worked like a champ 04:53:48 er 4 04:53:56