it still is

maybe i misread the thread about pppoe

debdrup, I did gently relocate that user to a more appropriate channel.

Anyone here got infiniband hardware on FreeBSD? — I'd love to see an ifconfig -a over that stuff

hrmm… there's a bei low frequency lists.freebsd.org/archives/freebsd-infiniband

Title: freebsd-infiniband⊙Fo

by the end of this project, I'm going to be subscribed to every single FreeBSD mailing list

Title: Soliciting infiniband ifconfig output

/win/win 3

hrm, i'm speccing out a dell t330 to run freebsd 13. was the zfs mirrored boot disks still problematic?

Demosthenex: what are you on about?

Demosthenex: I never ran into problems, and my installation is based on what I believe was 12.something-R : wdym?

i recall reading about a bug in the 13 installer where it didn't setup booting on the second drive

Probably best to check the bug report, then.

afaik the installer keeps logs on when and how it runs gpart

yes it's all logged

my old wokrstation BIOS is too old for my 4G disks so I need to boot from the 1st 1G or so

i'm looking in my history to find it, it was unique to 13.

my leeenurx server just died, ssd with OS was a SPOF. looking forward to having redundant os disks with freebsd

I wanted a mirrored zroot so I adapted the log to make a mirror of the size I needed

You can have mirror zfs disks in FreeBSD and on Ubuntu also if you do root on ZFS install, but FreeBSD is of course better in many ways.

Can you recover anything from the Disk with dd etc?

sorry I mixes these two sentences as if they were from the same user :)

I just had to do this, and figured I should mention it: If you're on the console and you can't work because of all the messages flowing past: Print Screen will get you to the next virtual terminal, which should be message free.

alt-f# ??

gogofc_: mirroring on linux isn't officially supported by any major distro, and is often prone to breakage during package updates. yes, you can tinkertoy cobble it together, but it's unreliable.

on my AIX systems having a mirrored OS disk pair is the norm, and it's NEVER an outage to replace/move/repair your OS mirror. but linux has never had a good answer.

i think from my reading freebsd can do it with a zfs mirror and then one command per disk to update the boot block after an update

exactly the same as aix ;]

linux can do it with mdm configured mirror iirc

mdadm to --fail and --remove the disk, swap the drive and mdadm --add

but its been a few years since i've done it

dkeav: and it doesn't maintain the boot sectors. and /boot on raid isn't supported across package upgrade. again, you can do it. but it's fragile and unsupported

hell, redhat officially says they don't even support moving the OS drive across media. they say do a full reinstall.

yea but that's redhat

:D

didn't they switch to btrfs

that's a RHEL-specific trait imho

..former job just used Debian (which we could support inhouse)

idwer: yeah i'm picky about "supported" features because in an enterprise environment with uptime prioritized, there's no time for me to experiment and i'd be held liable for any problem. so you only use supported featuers

no one ever got fired for paying for support

lies :P

a budget has to go _somewhere_

oh yea, you totally get fired for blowing your budget

just not for what you blew your budget on

unless it was that weekend in vegas expenses, that will get ya for sures

>.>

dvx: huh, I din't realize Print Screen cycled TTYs.

and with ctrl it's supposed to call the debugger

I don't even know what package this bug is from... tmux? entr?

basically when using entr, lsof reports cwd as the correct path, but when you tmux bind-key c neww -c "#{pane_current_path}" it... uses HOME instead?

ltns :)

quick question, but are you able to take out a dedup that is applied to a speciifc dataset only but send/recv'ing that dataset, or does it require a full pool wipe?

trying to isolate a new env that has gitea in it that is abysmal compared to linux (10x). Eliminated a bunch of variables, but the bsd just falls short and the only other thing I could think of was a dedup test

open to suggestions :)

troubled: what do you mean with it's abysmal? what are the symptoms?

meena: 100 requests per second from a linux gitea, 10 rps from bsd. yet zpool iostat shows basically zero reads happening, and disk io seems just find otherwise

figuring it's some go related stuff happening that is choking on api calls or doing some blocked lookups etc. Tried pull it out of jail and put on the host for some tests, but no difference.

not so much that I am leaning towards dedup, but wouldn't mind removing it as a variable just because

I don't think dedup will have a huge benefit for a git server

no doubt. found that the gitea should be using hard link'd clones anyway

was more a test originally, to avoid potentially 10 million cloned repo'd from spam type issues

that's a serious application level problem, that you need to tackle at the root, not at the tail

you close / secure sign up, and still listen out for people complaining about spam

no doubt

was also a test to see what kind of deupd stats we get when we reset and import the repos again. was only 1.5x so far, so barely worth anything, and the git gc's and packing will further reduce benefits

thus why I am asking about the removal choices for dedup :)

troubled: there's good chance that go isn't performing great, so let's remove dedup as a possible cause, and then look at how gitea is doing (I predict, roughly as bad as so far)

and then look at *what* it's doing, with detrace-toolkit

I was gonna run truss and blast through the wall of calls to look for something specific, but thats' rather crude. I don't know dtrace unfortunately

troubled: dtrace-toolkit is a package with lots of useful Dtrace scripts. such as dtruss, which is an excellent start

troubled: what about the database? are the Linux version and the FreeBSD version of gitea using the same database instance? type? etc

meena: it's pgsql in a jail on same host, all running on a cloned lo1 with an rfc1918

pf was in effect, but with it disabled, and stuff on host, there was little change

I understood some of those words

we removed the nginx proxy from the equation and went direct to gitea, no change

cloned lo1 == no vnet?

nah, didn't want the jails to really have their own, but we mulled it over. can always switch to it later

meena: so you have any luck with that dtrace toolkit?

I'd love to have gotta into it, but it's honestly quicker for us to just spin the setup up in linux and go on with things than it is to learn dtrace and spend weeks guessing why gitea is slow

it's a vm so I can keep it around and point the lb at it for tests etc, so is low priority. more a post-mortem at this point

troubled: pkg install dtrace-toolkit

run: dtruss -c -f -p gitea-pid

that will give you an idea what's happening

timing might be more important than counts, tho

I honestly think that's pretty much how every debugging session starts

I have needed to write my own dtrace stuff maybe once in ten years

because usually, that's also where the debugging often ends

cool, gimmie a sec

"oh look at that syscall being issued seven thousand times per second, I wonder whats that about" or: "oh, look at that call taking five seconds, I wonder what's that about"

bugs, unfortunately, are hardly ever nor sophisticated than this

for sure. They mention a debug/trace built into the app, but just been swamped with a bazillion other tasks atm. gets hard to justify more time on some stuff when a workable solution that only takes a few hours is in sight

aye.

this could be a five minute job, where you run dtruss, and then open a bug

okay, getting a pid on gitea itself could be tricky since its spawned via daemon tools like thing

troubled: are there /dev devices you need to expose in the jail for gitea to work well?

pertho: nice one!

possible, tbh im not sure what it might be trying. I figured something jail related, but we tried it all on the host too, no diff

dtruss should give us a clue

meena: will the ppid of the spawning ddaemon work?

also, ps should give you what you're looking for

i tried truss before but it would kill the gitea when I would ctrl-c off the process, and was too disruptive

a quick run :) clock_gettime 17366

Title: dtrace1 - Pastebin.com

let me get a better one, sec

truss and dtruss aren't even almost on the same level

pastebin.com/eyh6mnF5 there is a run of it doing some of the stuff that times out the gateway (PR from a 2016 branch that is massive)

Title: dtrace-gitea - Pastebin.com

Not really sure if that caught the children processes though

it does invoke a lot of cli git, which I assume is all the dup2?

or execve I guess eh

looks like some library dependency somewhere forgot to consider the fact that we have a poll API on FreeBSD, and is defaulting to nanosleep

so nanosleep is not a normal thing this?

do you really think it's normal to call clock_gettime in however many seconds you ran dtruss for?

it did stand out, but I also have zero bearing on what "normal" is for a cpu that is capable of literally billions of operations a second :)

I also don't know if that is catching the child processes that it forks where we get a gateway timeout waiting on a git diff between current and 2016 branch (we use as a stress test)

the linux system takes 17 seconds to diff, the bsd one tests 50+ seconds and some times hits the 1 minute timeout we currently have set

this is very typically pathological behaviour of an application (or usually one of its dependencies) that doesn't know about the poll API of a specific platform

meena: as in epoll?

troubled: I think there's a ebnf toolkit for Linux, and if you run that against the Linux gitea, you won't see this many clock_gettime calls. probably one of two orders of magnitude less

eBPF

might be new, and not even a regression github.com/go-gitea/gitea/issues?q=is%3Aissue+is%3Aopen+clock_gettime ?

Title: Issues · go-gitea/gitea · GitHub

well I believe the guy we have building the gitea app has a custom branch for it all, so he might have some insight into it. I will relay this info

or he fucked it up :P

well I don't know about that. he is working with the devs on this

I was tackling the environment, but he's in a sleeping timezone right now :)

so far, you can feel the mess: developer.blender.org

Title: Home

oops wrong url: projects.blender.org

Title: Blender Projects

why are ye building out yourself, instead of just using a package?

phab is going to gitea, so thats been backed and put on archive.blender.org

there's a bunch of complaints and requests from the devs I guess is the gist

so they are doing a bunch of features and customizations to keep the workflow and stuff like they want it

code.blender.org/2023/02/gitea-test-drive has some of the locust tests we ran though. linux absolutely destroyed it

Title: Gitea Test Drive — Developer Blog

I tried add a second vm hdd as a zfs mirror to help maybe speed up reads, tried iothrad scsi single devices, tried increase the ram, the cpu, enabled numa, even enabled the memached, but nothing helped us get past maybe 12 rps

atm its going through pfsense/HAProxy/snort with carp HA to a proxmox backed ceph ssd pool with 3/2 to the vm

I just can't squeeze any more out of it though :/

I'm telling you, it's broken

aye, about what we figure

but i'd rather freebsd than linux

the ceph isn't going to win any races, but man

yeah, then trace the error to the broken library or build option , and get it fixed

is on a pair of 10g nexus in vpc, so nothing fancy. but she's sturdy

oh I passed along the info, trust me :)

hey, hey, hey, HEY!

the amound of time we sunk into the bsd setup and the salt states etc...

Title: Soliciting infiniband ifconfig output

in case you got any of that lying around

I don't have anything infiniband handy sadly

we have the xeon phi sitting idle with 100g infiniband, but it's no running. i decommissioned it ages ago

aye

my remote counterpart found it though, so he might have it plugged in still. it's a dell 6300p or something iirc, so its limited use. Knights landing

like raspberry pi on steroids :D

I didnt know they made inifiniband up to that speed, makes sense

i have a talent for exposing bugs. in software, that's not too bad, but hardware can catch fire or explode, so I'm not allowed to touch that

(i'm very familiar with it at 56gbps)

nacelle: I saw 100g in the mellanox docs

ifconfig -m dev0

all i want is one damn ifconfig -a tho

meena: I see it now, I just wasnt aware (I have little use for ib, I have ib/ethernet nics and use them in ethernet mode)

well if get no response and we have some spare time, perhaps I could pursuade my coworked to fire up bsd on one for you, but no promises of their time

nacelle: ib has very very very low latency. can you still show me an ifconfig -a?

meena: that is your email btw?

meena: check your inbox ;)

well it's been a long day for me, time to enter lurk mode. Thanks as usual! o/

cgit.freebsd.org/src/commit/?id=69d94f4c7608 neat!

Title: src - FreeBSD source tree

I hope you don't like random I/O

tape archives, a format known for its sequential nature, probably won't have much random I/O

endrift: what has random i/o ever done for us?

meena: both PauAmma and you keep forgetting the first part

debdrup: it remains to be seen how many people try to use this and end up confused about why their random I/O is so slow though

"Who needs X? What has it X ever done for us anyway?"

I've seen people do really "why are you even doing this, that's not what it's meant for" things in the past

endrift: it's read-only.

I would hope so

but yeah I suppose that helps

Did you read the commit?

I skimmed part of it

I'm not saying it's a bad thing, I just don't trust users

The very first hunk of the diff contains the manual page, which mentions it being read-only.

debdrup: ill try to remember that for next time

meena: you better! (but if you don't, i almost certainly will have forgotten too, so who cares)

Hello! I've been reading the freebsd HandBook for a few days now, so far I like everything, although I don't understand much. Today I tried to boot into LiveCD, I managed to check the sound and my video card was detected in the list of devices. Please tell me, when using LiveCD, the files I created are saved somewhere or is all information stored in RAM and lost when the computer is turned off?

Kit_Leopold: the liveCD is read-only by default and if you're using a CD/DVD you can't write to the disk as the image is "fully authored" (closed, cannot be written to anymore unless it's a rewritable that's has its table of contents overwritten).

Even if you use a writable medium like a USB flash device or something similar, it's still read-only by default - and changes made won't affect the installation media unless you go out of your way to attempt that (which you really shouldn't)

the changes you made are probably in a memory-backed tmpfs

Unless you really, really have to. (Yes, it happens sometimes. Speaking from personal experience.)

I...can't say I've ever done that

I've done it too, but I broke things the first time I tried. :P

debdrup: Thank you. Sorry for my stupidity. Does this mean that I can use the pkg package manager on the LiveCD to explore the operating system more deeply and any changes I make will not be saved after the computer is turned off?

Kit_Leopold: not without making it read-write, and even then I'm not sure it'd work out very well.

..although I say that, but it's been so long since I've used the LiveCD environment, that I think it dates back way before bsdinstall (the current installer).

So it's entirely possible I'm talking out of my derriere.

Thank you for your answers and help.

I think helloSystem has a semi writable LiveCD thing

NomadBSD is geared toward running from a USB stick, with persistent data stored there.

debdrup, Note that helloSystem is based on FreeBSD not Linux. I didn't realize that at first either.

rwp: I'm aware.

First I have heard of it.