02:00:08 it still is 02:02:43 maybe i misread the thread about pppoe 05:51:03 debdrup, I did gently relocate that user to a more appropriate channel. 08:59:21 Anyone here got infiniband hardware on FreeBSD? — I'd love to see an ifconfig -a over that stuff 09:01:53 hrmm… there's a bei low frequency https://lists.freebsd.org/archives/freebsd-infiniband/ 09:01:54 Title: freebsd-infiniband⊙Fo 09:03:25 by the end of this project, I'm going to be subscribed to every single FreeBSD mailing list 11:03:59 https://lists.freebsd.org/archives/freebsd-infiniband/2023-February/000005.html 11:04:00 Title: Soliciting infiniband ifconfig output 11:05:30 /win/win 3 11:07:59 hrm, i'm speccing out a dell t330 to run freebsd 13. was the zfs mirrored boot disks still problematic? 11:09:04 Demosthenex: what are you on about? 11:42:11 Demosthenex: I never ran into problems, and my installation is based on what I believe was 12.something-R : wdym? 11:58:25 i recall reading about a bug in the 13 installer where it didn't setup booting on the second drive 12:12:03 Probably best to check the bug report, then. 13:39:51 afaik the installer keeps logs on when and how it runs gpart 13:41:23 yes it's all logged 13:42:16 my old wokrstation BIOS is too old for my 4G disks so I need to boot from the 1st 1G or so 13:43:21 i'm looking in my history to find it, it was unique to 13. 13:43:50 my leeenurx server just died, ssd with OS was a SPOF. looking forward to having redundant os disks with freebsd 13:43:57 I wanted a mirrored zroot so I adapted the log to make a mirror of the size I needed 13:57:11 You can have mirror zfs disks in FreeBSD and on Ubuntu also if you do root on ZFS install, but FreeBSD is of course better in many ways. 13:57:23 Can you recover anything from the Disk with dd etc? 13:58:09 sorry I mixes these two sentences as if they were from the same user :) 14:15:55 I just had to do this, and figured I should mention it: If you're on the console and you can't work because of all the messages flowing past: Print Screen will get you to the next virtual terminal, which should be message free. 14:24:20 alt-f# ?? 14:26:10 gogofc_: mirroring on linux isn't officially supported by any major distro, and is often prone to breakage during package updates. yes, you can tinkertoy cobble it together, but it's unreliable. 14:27:16 on my AIX systems having a mirrored OS disk pair is the norm, and it's NEVER an outage to replace/move/repair your OS mirror. but linux has never had a good answer. 14:27:37 i think from my reading freebsd can do it with a zfs mirror and then one command per disk to update the boot block after an update 14:27:44 exactly the same as aix ;] 14:45:08 linux can do it with mdm configured mirror iirc 14:45:34 mdadm to --fail and --remove the disk, swap the drive and mdadm --add 14:45:51 but its been a few years since i've done it 14:50:07 dkeav: and it doesn't maintain the boot sectors. and /boot on raid isn't supported across package upgrade. again, you can do it. but it's fragile and unsupported 14:50:30 hell, redhat officially says they don't even support moving the OS drive across media. they say do a full reinstall. 14:55:56 yea but that's redhat 14:55:58 :D 14:56:10 didn't they switch to btrfs 15:02:35 that's a RHEL-specific trait imho 15:03:00 ..former job just used Debian (which we could support inhouse) 15:19:30 idwer: yeah i'm picky about "supported" features because in an enterprise environment with uptime prioritized, there's no time for me to experiment and i'd be held liable for any problem. so you only use supported featuers 15:49:09 no one ever got fired for paying for support 15:58:01 lies :P 15:58:02 a budget has to go _somewhere_ 16:01:50 oh yea, you totally get fired for blowing your budget 16:01:56 just not for what you blew your budget on 16:02:47 unless it was that weekend in vegas expenses, that will get ya for sures 16:02:49 >.> 18:03:34 dvx: huh, I din't realize Print Screen cycled TTYs. 19:06:14 and with ctrl it's supposed to call the debugger 19:10:53 I don't even know what package this bug is from... tmux? entr? 19:13:24 basically when using entr, lsof reports cwd as the correct path, but when you tmux bind-key c neww -c "#{pane_current_path}" it... uses HOME instead? 20:01:57 ltns :) 20:03:04 quick question, but are you able to take out a dedup that is applied to a speciifc dataset only but send/recv'ing that dataset, or does it require a full pool wipe? 20:04:25 trying to isolate a new env that has gitea in it that is abysmal compared to linux (10x). Eliminated a bunch of variables, but the bsd just falls short and the only other thing I could think of was a dedup test 20:04:32 open to suggestions :) 20:22:10 troubled: what do you mean with it's abysmal? what are the symptoms? 20:23:28 meena: 100 requests per second from a linux gitea, 10 rps from bsd. yet zpool iostat shows basically zero reads happening, and disk io seems just find otherwise 20:24:20 figuring it's some go related stuff happening that is choking on api calls or doing some blocked lookups etc. Tried pull it out of jail and put on the host for some tests, but no difference. 20:25:06 not so much that I am leaning towards dedup, but wouldn't mind removing it as a variable just because 20:31:31 I don't think dedup will have a huge benefit for a git server 20:32:07 no doubt. found that the gitea should be using hard link'd clones anyway 20:32:33 was more a test originally, to avoid potentially 10 million cloned repo'd from spam type issues 20:33:38 that's a serious application level problem, that you need to tackle at the root, not at the tail 20:34:29 you close / secure sign up, and still listen out for people complaining about spam 20:36:00 no doubt 20:37:06 was also a test to see what kind of deupd stats we get when we reset and import the repos again. was only 1.5x so far, so barely worth anything, and the git gc's and packing will further reduce benefits 20:37:23 thus why I am asking about the removal choices for dedup :) 20:39:28 troubled: there's good chance that go isn't performing great, so let's remove dedup as a possible cause, and then look at how gitea is doing (I predict, roughly as bad as so far) 20:40:01 and then look at *what* it's doing, with detrace-toolkit 20:40:53 I was gonna run truss and blast through the wall of calls to look for something specific, but thats' rather crude. I don't know dtrace unfortunately 20:45:33 troubled: dtrace-toolkit is a package with lots of useful Dtrace scripts. such as dtruss, which is an excellent start 20:48:25 troubled: what about the database? are the Linux version and the FreeBSD version of gitea using the same database instance? type? etc 20:56:53 meena: it's pgsql in a jail on same host, all running on a cloned lo1 with an rfc1918 20:57:15 pf was in effect, but with it disabled, and stuff on host, there was little change 20:57:27 I understood some of those words 20:57:48 we removed the nginx proxy from the equation and went direct to gitea, no change 20:58:09 cloned lo1 == no vnet? 20:58:35 nah, didn't want the jails to really have their own, but we mulled it over. can always switch to it later 21:02:40 meena: so you have any luck with that dtrace toolkit? 21:03:53 I'd love to have gotta into it, but it's honestly quicker for us to just spin the setup up in linux and go on with things than it is to learn dtrace and spend weeks guessing why gitea is slow 21:04:14 it's a vm so I can keep it around and point the lb at it for tests etc, so is low priority. more a post-mortem at this point 21:05:36 troubled: pkg install dtrace-toolkit 21:07:19 run: dtruss -c -f -p gitea-pid 21:07:34 that will give you an idea what's happening 21:07:52 timing might be more important than counts, tho 21:09:57 I honestly think that's pretty much how every debugging session starts 21:10:18 I have needed to write my own dtrace stuff maybe once in ten years 21:10:43 because usually, that's also where the debugging often ends 21:11:30 cool, gimmie a sec 21:12:14 "oh look at that syscall being issued seven thousand times per second, I wonder whats that about" or: "oh, look at that call taking five seconds, I wonder what's that about" 21:12:39 bugs, unfortunately, are hardly ever nor sophisticated than this 21:14:13 for sure. They mention a debug/trace built into the app, but just been swamped with a bazillion other tasks atm. gets hard to justify more time on some stuff when a workable solution that only takes a few hours is in sight 21:15:20 aye. 21:16:00 this could be a five minute job, where you run dtruss, and then open a bug 21:16:25 okay, getting a pid on gitea itself could be tricky since its spawned via daemon tools like thing 21:16:27 troubled: are there /dev devices you need to expose in the jail for gitea to work well? 21:16:50 pertho: nice one! 21:17:08 possible, tbh im not sure what it might be trying. I figured something jail related, but we tried it all on the host too, no diff 21:18:38 dtruss should give us a clue 21:18:40 meena: will the ppid of the spawning ddaemon work? 21:18:55 also, ps should give you what you're looking for 21:19:13 i tried truss before but it would kill the gitea when I would ctrl-c off the process, and was too disruptive 21:20:16 a quick run :) clock_gettime 17366 21:21:03 https://pastebin.com/nyzdD6K6 21:21:05 Title: dtrace1 - Pastebin.com 21:21:18 let me get a better one, sec 21:24:27 truss and dtruss aren't even almost on the same level 21:24:37 https://pastebin.com/eyh6mnF5 there is a run of it doing some of the stuff that times out the gateway (PR from a 2016 branch that is massive) 21:24:38 Title: dtrace-gitea - Pastebin.com 21:25:38 Not really sure if that caught the children processes though 21:26:13 it does invoke a lot of cli git, which I assume is all the dup2? 21:26:37 or execve I guess eh 21:26:53 looks like some library dependency somewhere forgot to consider the fact that we have a poll API on FreeBSD, and is defaulting to nanosleep 21:27:13 * troubled blows the dust off of his limited api calls knowledge 21:27:31 so nanosleep is not a normal thing this? 21:29:34 do you really think it's normal to call clock_gettime in however many seconds you ran dtruss for? 21:30:15 it did stand out, but I also have zero bearing on what "normal" is for a cpu that is capable of literally billions of operations a second :) 21:31:07 I also don't know if that is catching the child processes that it forks where we get a gateway timeout waiting on a git diff between current and 2016 branch (we use as a stress test) 21:31:45 the linux system takes 17 seconds to diff, the bsd one tests 50+ seconds and some times hits the 1 minute timeout we currently have set 21:32:24 this is very typically pathological behaviour of an application (or usually one of its dependencies) that doesn't know about the poll API of a specific platform 21:35:52 meena: as in epoll? 21:36:25 troubled: I think there's a ebnf toolkit for Linux, and if you run that against the Linux gitea, you won't see this many clock_gettime calls. probably one of two orders of magnitude less 21:38:10 eBPF 21:40:38 might be new, and not even a regression https://github.com/go-gitea/gitea/issues?q=is%3Aissue+is%3Aopen+clock_gettime ? 21:40:39 Title: Issues · go-gitea/gitea · GitHub 21:40:49 well I believe the guy we have building the gitea app has a custom branch for it all, so he might have some insight into it. I will relay this info 21:41:13 or he fucked it up :P 21:41:29 well I don't know about that. he is working with the devs on this 21:41:46 I was tackling the environment, but he's in a sleeping timezone right now :) 21:42:06 so far, you can feel the mess: https://developer.blender.org/ 21:42:07 Title: Home 21:42:22 oops wrong url: https://projects.blender.org/ 21:42:23 Title: Blender Projects 21:42:42 why are ye building out yourself, instead of just using a package? 21:42:48 phab is going to gitea, so thats been backed and put on archive.blender.org 21:43:04 there's a bunch of complaints and requests from the devs I guess is the gist 21:43:18 * meena wonders which way FreeBSD's phab will go 21:43:25 so they are doing a bunch of features and customizations to keep the workflow and stuff like they want it 21:44:14 https://code.blender.org/2023/02/gitea-test-drive/ has some of the locust tests we ran though. linux absolutely destroyed it 21:44:15 Title: Gitea Test Drive — Developer Blog 21:45:21 I tried add a second vm hdd as a zfs mirror to help maybe speed up reads, tried iothrad scsi single devices, tried increase the ram, the cpu, enabled numa, even enabled the memached, but nothing helped us get past maybe 12 rps 21:46:42 atm its going through pfsense/HAProxy/snort with carp HA to a proxmox backed ceph ssd pool with 3/2 to the vm 21:47:00 I just can't squeeze any more out of it though :/ 21:47:25 I'm telling you, it's broken 21:47:32 aye, about what we figure 21:47:39 but i'd rather freebsd than linux 21:48:03 the ceph isn't going to win any races, but man 21:48:11 yeah, then trace the error to the broken library or build option , and get it fixed 21:48:29 is on a pair of 10g nexus in vpc, so nothing fancy. but she's sturdy 21:48:47 oh I passed along the info, trust me :) 21:49:04 hey, hey, hey, HEY! 21:49:09 the amound of time we sunk into the bsd setup and the salt states etc... 21:49:20 https://lists.freebsd.org/archives/freebsd-infiniband/2023-February/000005.html 21:49:22 Title: Soliciting infiniband ifconfig output 21:49:40 in case you got any of that lying around 21:50:12 I don't have anything infiniband handy sadly 21:50:40 we have the xeon phi sitting idle with 100g infiniband, but it's no running. i decommissioned it ages ago 21:51:07 aye 21:51:31 my remote counterpart found it though, so he might have it plugged in still. it's a dell 6300p or something iirc, so its limited use. Knights landing 21:51:47 like raspberry pi on steroids :D 21:52:26 * meena doesn't know anything about hardware 21:53:32 I didnt know they made inifiniband up to that speed, makes sense 21:53:41 i have a talent for exposing bugs. in software, that's not too bad, but hardware can catch fire or explode, so I'm not allowed to touch that 21:53:42 (i'm very familiar with it at 56gbps) 21:54:32 nacelle: I saw 100g in the mellanox docs 21:54:48 ifconfig -m dev0 21:55:47 all i want is one damn ifconfig -a tho 21:56:01 meena: I see it now, I just wasnt aware (I have little use for ib, I have ib/ethernet nics and use them in ethernet mode) 21:56:40 well if get no response and we have some spare time, perhaps I could pursuade my coworked to fire up bsd on one for you, but no promises of their time 21:59:39 nacelle: ib has very very very low latency. can you still show me an ifconfig -a? 22:01:04 meena: that is your email btw? 22:03:37 meena: check your inbox ;) 22:09:52 well it's been a long day for me, time to enter lurk mode. Thanks as usual! o/ 22:33:41 https://cgit.freebsd.org/src/commit/?id=69d94f4c7608 neat! 22:33:42 Title: src - FreeBSD source tree 22:38:55 I hope you don't like random I/O 22:40:40 tape archives, a format known for its sequential nature, probably won't have much random I/O 22:40:57 endrift: what has random i/o ever done for us? 22:41:18 meena: both PauAmma and you keep forgetting the first part 22:41:44 debdrup: it remains to be seen how many people try to use this and end up confused about why their random I/O is so slow though 22:41:49 "Who needs X? What has it X ever done for us anyway?" 22:42:03 I've seen people do really "why are you even doing this, that's not what it's meant for" things in the past 22:42:29 endrift: it's read-only. 22:42:38 I would hope so 22:42:43 but yeah I suppose that helps 22:42:44 Did you read the commit? 22:42:54 I skimmed part of it 22:43:02 I'm not saying it's a bad thing, I just don't trust users 22:43:15 The very first hunk of the diff contains the manual page, which mentions it being read-only. 22:43:51 debdrup: ill try to remember that for next time 22:46:56 meena: you better! (but if you don't, i almost certainly will have forgotten too, so who cares) 22:47:28 Hello! I've been reading the freebsd HandBook for a few days now, so far I like everything, although I don't understand much. Today I tried to boot into LiveCD, I managed to check the sound and my video card was detected in the list of devices. Please tell me, when using LiveCD, the files I created are saved somewhere or is all information stored in RAM and lost when the computer is turned off? 22:49:08 Kit_Leopold: the liveCD is read-only by default and if you're using a CD/DVD you can't write to the disk as the image is "fully authored" (closed, cannot be written to anymore unless it's a rewritable that's has its table of contents overwritten). 22:50:23 Even if you use a writable medium like a USB flash device or something similar, it's still read-only by default - and changes made won't affect the installation media unless you go out of your way to attempt that (which you really shouldn't) 22:51:51 the changes you made are probably in a memory-backed tmpfs 22:51:58 Unless you really, really have to. (Yes, it happens sometimes. Speaking from personal experience.) 22:52:19 I...can't say I've ever done that 22:52:50 I've done it too, but I broke things the first time I tried. :P 22:54:12 debdrup: Thank you. Sorry for my stupidity. Does this mean that I can use the pkg package manager on the LiveCD to explore the operating system more deeply and any changes I make will not be saved after the computer is turned off? 22:55:15 Kit_Leopold: not without making it read-write, and even then I'm not sure it'd work out very well. 22:55:54 ..although I say that, but it's been so long since I've used the LiveCD environment, that I think it dates back way before bsdinstall (the current installer). 22:56:40 So it's entirely possible I'm talking out of my derriere. 22:57:00 Thank you for your answers and help. 22:58:22 I think helloSystem has a semi writable LiveCD thing 23:00:14 NomadBSD is geared toward running from a USB stick, with persistent data stored there. 23:29:28 debdrup, Note that helloSystem is based on FreeBSD not Linux. I didn't realize that at first either. 23:30:16 rwp: I'm aware. 23:30:34 First I have heard of it.