f451: patience young skywalker

heh

well semi seriously it's a lot of juice building rust over and over again

its very not-green

it takes 45min-1hr30 to build, 450W @240V

that's offset by how efficient its software is

total fake news

just for that one thing, and it's over and over and over again

other OSes don't do it, so why us?

when pkg distributes binaries and they're rust and super efficient, it's 1000x more green than the same node.js or python interpreted junk being run over and over and over x millions of users

how efficient it is is a strawman. it has to be built in the first place, even for one software that needs it

its not so much it needs to be built, it has to be built everytime anything else is built. unlike python or llvm

its the number of times it needs to be built

multiplied by the energy required to build it

what are you saying has to be rebuilt?

im talking about rust

ya and what about it?

it takes a lot of energy to build, and for some reason the implementation on freebsd requires it to be rebuilt very frequently

i dont see either of those. it's a compiled lang, it has build time sure like any other. what does it mean rebuilt very frequently?

i have some rust that i build when i set up a new major version like 14.x then never recompile

well, if you build ports (not build -a, build -f portslist) every few days, because you need to keep on top of, say, updates, because you're running internet exposed services, you'll find that in almost every build, even if the rust version doesn't change, rust is rebuilt

observe for yourself

have a look at some posts about it in @hackers

k

doesn't mean anything to me though other than fbsd build infra needs more donations. whatever we can do to get more compiled rust pkgs into ppl's hardware, the bigger the power savings

the majority of power is in millions of users running software, not dozens of machines building it

the majority of power is in millions of users running software, not dozens of machines building it

you''re forgetting those of us who need to build locally because the default options are not what is required, You're missing my point, also - that is requires itself to be rebuilt while it has not in itself changed version

or has its options changed

nothing else, llvm, gcc - requires this

problem in search of solution, easy as

i build locally too fwiw

is it not using makefile?

externalized header guards?

pragma once equivalent, whatever?

is the problem that rust doesn't have a stable abi yet?

rust? no idea. in a month, with poudriere builds running every 3 dats, i can almost guarantee it'll need to be rebuilt every time, llvm maybe twice

if rebuilding the same thing results in changes to anything, you're doing it wrong

stable or not

so my question is really- why no stable abi under freebsd - yet other OSes don;t seem to need this?

remove the me from it - others have reported similar. we're all wrong?

is that not a poudriere thing? where it is rather conservative with guessing at means to save rebuilds

traditiionall ccache is used to workaround that wrt poudriere

i dunno if it's a poudriere thing or not. my poudriere uses ccache and sccache and sccache-overlay

all i know is rust rebuilds all the time with the same options and the same -f portslist

if it changed version number, something like that, i'd understand

im sure there's a solution. ill try to learn more about it

there was a thread in hackers@ about it a couple of months ago

hey, the zfs recordsize property on a dataset just limits at which point larger blobs are split across multiple records, but it doesn't result in files smaller than this limit having to fully allocate a single record of that size, right?

foxxx0: i thought it did, might be wrong though

f451: I mean, what branch of FreeBSD are you building for?

all

e.g. even with recordsize=128k a single 4k file won't allocate a full 128k record? (just 4k+metadata?)

if we're thinking of the same thread on -hackers@, that was a different issue

I'm pondering what recordsize to choose for my datasets. I've done a quick look to estimate distribution of filesizes and what not ...

"stable ABI" has nothing to do with how often we have to rebuild it

we have to rebuild it because either it was updated, or perhaps one of its dependencies, or maybe __FreeBSD_version bumped, etc

unfortunately all of my recordsize fio benchmarking this evening was a complete waste, since my cpu/geli is one heck of a bottleneck :D

kevans well pkgs aren't always fast to update. like grafana pkg is v10 but v11 been out for months. so just pin rust to a max frequency of updates like no more often than once a week?

i used to build compilers in freebsd. it was pretty fun.

l00py: right now I look at lang/rust and it hasn't seen an update since late August, so it's not exactly updating itself very often

kevans: i'd like to map that cos ive seen too many times where the version is the same the os hasnt changed

probably one or more of its dependencies

i guess curl is a dependency of rust

yeah, curl's probably a big one

lemme check the last few builds

curl gets bumped at least once a month, but there's also cmake, ninja, pkgconf, and python

you're probably just getting really unlucky with a set like this

origin: lang/rust reason: textproc/bat

f451: it appears that statement indeed holds true. "If you want to store a 500KiB file on a dataset with recordsize=1M, it goes in a single 512KiB block. If you then store a 3KiB file in the same dataset, it gets stored in a single 4KiB block (assuming ashift<=12)."

yes

maybe this will motivate you to develop a feature such as suppression periods for ports and their dependencies such that they won't rebuild until a threshold d/t period is passed

so it matters more for regular changes to existing files, as each modification *then* needs to rewrite a *whole* record

postgresql is known for wanting 16k recordsize

can't rebuild what you didn't rebuild in the first place

foxxx0: yeah

since probably ~80% of my data are mostly "write once, read a couple of times later", with very little modifications, I should be fine just picking something betweek 128k and 1m recordsize

between*

the_oz: ok if a port xyz needs gcc you're saying that gcc needs to be *rebuilt* in order to facilitate making xyz?

if you already built gcc previously...

remember that rust is already known, and in poudriere

I do not know the particulars of the problem, it isn't mine

which is my point - why did rust need to be rebuilt to facilitate updating bat?

foxxx0: if you're using postgres you'll need recordsize 16k

i dont know about other db

mysql/innodb wants 16k and postgres 8k from what I could find

something about write amplification etc, others know a lot more

I'm more curious now why nvme-cli doesn't find my drives

is nvme-cli not supported on freebsd?

what is it ;)

at the end of unattended bsdinstall, the debug log pops up and it says at the bottom "cannot unmount /mnt/tmp: pool or dataset is busy". so i added 'zfs unmount -f zroot/tmp' to my installerconfig, and now the debug log says "rm: /mnt/tmp/installscript: no such file or dir" so what's the right solution pls?

generic utility for interacting with nvmes, get+set controller/namespace properties, update firmware, etc.

i dunno, ill look in ports

it was available in package-ng but "nvme list" doesn't return anything :D

/usr/ports/sysutils/nvme-cli

ah well, the "proper" freebsd utility would likely be "nvmecontrol"

since that nvme-cli appears to be ported from linux

ive never used it, on my builder have ssds on pvi

pci

2x cards, zfs stripe

I've just put 4x 7.68tb micron 7300 pro into my new NAS

generic kernel has nvme stuff in

nice

but I'm struggling with geli+raidz1 performance, I'm *severely* cpu-constrained

geli will do that

I was fully aware of that, but I'd still though I would be able to push more than ~1GiB/s through to a raidz1

particularly geli swap

unfortunately not, it seems geli maxes out around 4x ~600MiB/s streams for me

have crypto extensions enabled in hw?

and loaded in fs

geli reports accelerated-software correctly, dmesg states AESNI available

so I'm guessing yes

f451: sounds more like you're looking at the reason it's needing to build rust in the first place, not the reason that it needs to rebuild rust for this build to continue

I have aesni_load="YES" and geom_eli-load="YES" early in my /boot/loader.conf, it appears to be working as intended, it's just not ... performing all that well

on my storage cluster i get 120-200MB/s with zfs but thats also with a hp-something card in a mirror config

well *each* of these 4 NVMEs is capable of ~3.4 GiB/s read and ~2.5 GiB/s writes.

kevans: rust is already there - by 'there' i mean in its pkg collection. someth=imes poudriere checks shlibs and doesn't have to build it - but thats a recent development and it's afaict only with latest poudriere-devel

foxxx0, sounds like either aesni isn't being used properly, or something else in the setup is chewing up the cpu. I've got 2 arrays of 4x drives raid-z2, and I can get 500-600MB/s on each one, on a 10 year old opteron CPU at 1.4GHz

f451: yes, and it will get removed if any of its dependencies, or their dependencies, so on and so forth, get a version bump

with maybe 30-40% cpu usage?

well I can get 500-600 MiB/s per drive, which, when combined into a 4-drive raidz1, results in ~1GiB/s usable write performance

f451: your real complaint here is that poudriere is overly aggressive on this, but there is a very good reason for that

yeah?

reads on that array go up to ~2.4 GiB/s, that is okay-ish

the very good reason is that we suck at handling PORTREVISION bumps of reverse deps when we need to :-)

just the writes appear notably slower

kevans: LOL

it could be less aggressive, but then we'd run into problems with shit going weird because people suck

some of it could be fixed by pkg handling shlib dependencies better, I think, but that's only part of the story

just trying to stop rust agressively chewing my leccy bill

there are non-shlib ways to wreck your reverse deps' day

foxxx0, out of curiosity, are you using GELI above or below zfs?

blockdev -> geli -> raidz1

so zfs *on top* of geli

ok cool

i have one geli provider per blockdev

that seems fine then

which appears to be running .... 6 threads each?

(judging by ps -auxH | grep g_eli)

I haven't used nvme on freebsd yet, so I'm wondering if something is up with the kernel support of those?

though if you've found it is the geli threads using up cpu cycles, maybe now

*not

what CPU + memory config is this using?

3cores/6threads of a ryzen 5600 with 24G memory

casual spreadsheet drop: docs.google.com/spreadsheets/d/1pdu…Udt6fkCjpnXxAEFlyA/edit?usp=sharing

it is a VM though. I've tried my best to isolate the memory and cpu cores for this VM, but there might still be some interference

yeah I wonder if theres some weird CPU pinning issue going on?

(V)CPUs are pinned

and I'm able to see the cores are only utilized by this VM, not even interrupts arrive on them

maybe try allocating physical cores to the VM, even if you drop it to 3

mh, excluding the hyperthreading siblings you mean?

otherwise the scheduler in the VM might be mixing up the SMT with the host

that's an easy test, let me check

now I only have 3 threads per geli provider, which matches the new cpu-corecount, so it does automatically scale that to all available cores on SMT systems

nope, didn't help

at the end of unattended bsdinstall, the debug log pops up and it says at the bottom "cannot unmount /mnt/tmp: pool or dataset is busy". so i added 'zfs unmount -f zroot/tmp' to my installerconfig, and now the debug log says "rm: /mnt/tmp/installscript: no such file or dir" so what's the right solution pls?

hrmpf, I've recreated the zpool without any geli involvement, still only getting ~1GiB/s throughput on writes

the bottleneck is actually not geli.

ah well, the NVMEs were stuck in the lowest power state m(

5.2 GiB/s writes, that's more like it. let's see what happens when I re-add geli inbetween

how'd you get them in a higher power state?

nvmecontrol power -p 0 nvme0 ; repeat for nvme1,2,3

unfortunately it doesn't really indicate what state they are in

how did you benchmark their 5.2g/s write speed? i wanna test mine now

my work-in-progress thingy: paste.foxxx0.de/vrY

take what you need

i'm still editing a bunch of stuff ondemand, currently log output to file is commented out as you might notice

foxxx0: 5.2GiB/s ? are those on raid ?

HER: yup, raidz1 with 4 drives

just slightly overkill for a personal NAS :D

I'm gonna re-run my benchmarks without and with geli now

foxxx0: how fast can you write to a single one ?

without raid

5.2/4 ?

but I'm probably just going to put the NVMEs back into the lowest powerstate afterwards, I only have 10G network anyways, so 1 GiB/s writes and ~3 GiB/s reads are already more than enough

single drive is ~3.4 GiB/s read, ~2.5 GiB/s write

foxxx0: ok

I'm pretty sure I'm cpu-limited now

foxxx0: strange, i cant seem to pass +- 400mb writes in a single one

HER: what make/model/size of drive?

foxxx0: its a corsair supposed to be 3000MB/s but it never passes 400MB/s =p

it all depends on the parameters, blocksize, queue-depth, sequential/random, etc.

ahh

foxxx0: what do you use ? i used 4k

the numbers above just now were on a dataset with 128k recordsize and 1M blocksize sequential for fio

nice, thanks for the info foxxx0

foxxx0, so it looks like the cpu "usage" of geli was actually in a state of iowait then

edenist: yep, I was also a bit irritated by the swings in throughput, occasionally looked like it would hiccup and almost stall but then crawl right back with a huge spike in throughput

something definitely looked off, which is precisely why I'm doing all these benchmarks to figure out what's happening

once I've collected all plaintext benchmarks with the highest powerstate, I'll probably repeat those for 2 or 3 additional (lower) power states. and then re-run everything with geli enabled

at the end of unattended bsdinstall, the debug log pops up and it says at the bottom "cannot unmount /mnt/tmp: pool or dataset is busy". so i added 'zfs unmount -f zroot/tmp' to my installerconfig, and now the debug log says "rm: /mnt/tmp/installscript: no such file or dir" so what's the right solution pls?

why your installscript is in /mnt/tmp and not in /tmp ?

I mean, if your install is created in /mnt, you do not want to run anything from /mnt

i didn't put it there so if there's a bug it's in bsdinstall

btw i tried at end of installerconfig ls -la /tmp then ls -la /mnt/tmp. /tmp has installscript and a few other things in it but /mnt/tmp says no such file or dir??

so if there is no /mnt/tmp to ls -la, why do i get the error "cannot unmount /mnt/tmp: pool or dataset is busy?

fs is busy if it has open files from it -- either it is current working directory for some program (so the dir itself is open), or there are currently open files from it, the files can be already unlinked - in that case ls will return empty, but the files are still there and removed when they are closed.

ok so what's the solution for me?

back, so what's the solution?

it can't be normal to just have bsdinstalls fail at the end with unable to unmount errors

no it is not

so what can i do?

make sure you file the bug, and rest depends on how much you can investigate the problem yourself.

how can i check if /mnt/tmp has been is unmounted or not? i want to try running zfs unmount zroot/tmp and sleep # combos

from /etc/mnttab (or just df command;)

Question: while I know that FreeBSD supports Intel integrated graphics really well, how good is AMD integrated graphics? Like with Ryzen xxxxG chips?

Hi; does wg interface compeltely baypass the pf firewall?

if you want it too, if you configure pf to manage traffic on it I would assume not

I only have skip on lo0 but any traffic comming from wg0 is accepted if the server is listening on e.g., *:22

And I don't even see the connection on the PF states list

imm_: do you have "block in" ?

I have a block drop at the beginning and pass out at the end

But even if I have PF misonfigured and it passes the connection, I would've seen it in the states list in pfctl -s, right?

fastest thing might be putting your pf.conf on dpaste.org so we can see :)

imm_: the traffic is somehow bypassed, not tight rules

daemon: dpaste.org/dbCuT

And I can do this: nc -l 3333 and get a connection on wg

what is "block drop"

are you blocking host named "drop" ?

Isn't that the block drop all rule?

drop being don't send ICMP reply

OK, it could work, but "drop in" is common solution

drop in works the same

I can access nc -l 3333 from wg remote

My assumption was that it should be blocked by PF or at least shown in states

nice, your rules are now more clear

if it's ignored then it won't be shown

Question: while I know that FreeBSD supports Intel integrated graphics really well, how good is AMD integrated graphics? Like with Ryzen xxxxG chips?

imm_: can you see and log this traffic with tcpdump ?

mzar: Yes, on wg, 14:31:08.707151 IP 10.13.0.33.22249 > 10.13.0.3.3333: Flags [P.], seq 1:5, ack 1, win 1035, options [nop,nop,TS val 3896215339 ecr 228481065], length 4

10.13* are wg IPs

OK, but I can't reproduce it

mzar: Maybe 13.3 is the reason?

mzar: The rules are now very simple, block in, pass ssh, pass wg, pass out; I see 3333 in sockstat but not in PF

imm_: I have no access to FreeBSD 13 to test it

mzar: I must be doing something wrong, I have no 'skip on lo0' but I can connect to any service on localhost

Is maybe skip on lo0 persistent even if I do pfctl -f?

imm_: put your rules in /etc/pf.conf and begin with "service pf onestart" then proceede with mastering and learnign pfctl(8) syntax

mzar: My bad, 'skip on *' is persistant even if you remove it from /etc/pf.conf and reload with pfctl -f

I put skip on wg0 when I had problems establishing the wg connection

So that's why it skipped the wg completely

I had to pfctl -F all, then it reloads properly

'

imm_: you should read manual and progress step by step

mzar: Thanks for the help

Question: while I know that FreeBSD supports Intel integrated graphics really well, how good is AMD integrated graphics? Like with Ryzen xxxxG chips?

i got a 2080 working fine

l00py90: Just to make sure, we're talking about integrated graphics on Ryzen chips, not discrete graphics on Radeon chips, right?

Because 2080 doesn't sound like a Ryzen CPU.

I'm trying to use egrep to match upper & lowercase letters, numbers, the dash and the period and running into syntax issues:

$ jq -M . ucrm.clean.json | egrep -o [a-zA-Z0-9\-\.]*@ | sort -u

egrep: invalid character range

CrtxReavr: Is the shell trying to expand the * as a glob?

vkarlsen, don't think so - if I drop the ``\.`` portion, it works (minus matching the .)

probably the shell is escaping \- -> - before it gets to egrep

CrtxReavr: I get no matches found unless I put the whole regex inside ' '. I don't have your json data though

quoting aside, you probably want [a-zA-Z0-9.-] which is how you normally include - in a char range

[-a-zA-Z0-9.]* seems to work.

Also not supposed to be escaped inside of []s apparently.

I didn't know order was important

I was just told in #linux that to match - is must be first or last.

order's not important, it's just ambiguous

Since - is also a range charcter (a-z)

Ah, makes sense

Keep getting ignored...

i would argue that we probably just don't have as many folks around with that experience using amd igpu on freebsd

hmm, managed to make p9fs return 'invalid argument' for a particular directory for no apparent reason

ivy: congrats

Reading back and I cringe at the unquoted regular expression that included shell meta-characters. Then cringed again at the attempt to backslash quote characters inside of a bracketed character class as character class already quote characters and therefore those backslashes become part of the character set in them.

mzar: probably not a very impressive achieve considering this code was merged in somewhat unfinished state :-) i will file a bug though

remiliascarlet: in principle it should be the same as it uses the same architecture and driver

What's the best way to debug this freebsd-update error: "Fetching 2 metadata files... failed."?

turbo23: Sharing the whole command line might help.

tuaris: Sharing the whole command line might help. (Tabfailed that first one.)

Just `freebsd-update fetch`

truss, shows these last system calls :fstatat(AT_FDCWD,"bc21c7281db44e7afade49a7aae9050f4603a2ed71629fbb3d452b9decbaf54d.gz",0x22ebb1ee8f8,0x0) ERR#2 'No such file or directory'

o

err

ld: error: undefined symbol: testing::internal::MakeAndRegisterTestInfo(char const*, char const*, char const*, char const*, testing::internal::CodeLocation, void const*, void (*)(), void (*)(), testing::internal::TestFactoryBase*)

>>> referenced by zfsd_unittest.cc:236 (/data/build/src/freebsd/lf/main/cddl/usr.sbin/zfsd/tests/zfsd_unittest.cc:236)

i just saw someone else reporting this earlier and now i can't remember where

ah, lists.freebsd.org/archives/freebsd-current/2024-October/006591.html

at the end of unattended bsdinstall, the debug log pops up and it says at the bottom "cannot unmount /mnt/tmp: pool or dataset is busy". so i added 'zfs unmount -f zroot/tmp' to my installerconfig, and now the debug log says "rm: /mnt/tmp/installscript: no such file or dir" so what's the right solution pls? at end of installerconfig i tried ls -la

/tmp then ls -la /mnt/tmp. /tmp has installscript and a few other things in it but /mnt/tmp says no such file or dir. any ideas?

the unmount problem doesn't happen when i don't do some config that temporarily puts files in tmp. so maybe i need to rm that dir first

a hacky, ugly thing that sometimes works is just putting a sleep 3 into the script before the command that fails, as though there's some contention that has a moment to clear up

i have an old x201, it boots fine under nomadbsd 14.1... but something triggers terrible coil whine or something? when i boot linux it's more silent. any ideas?

I have an x201. Last booted FreeBSD 13. Over on the shelf and not booted since 14. I don't recall any whine from it.

Other than the fan I can't imagine what would be making a whining noise. Is the fan running full speed on yours?

it's not the fan spinning

it's a very electrical chirping

One problem I never worked out on my x201 in 13 was the suspend-resume regarding graphics. It resumed but upon resume graphics was a black screen. I could log into it again with ssh and it was alive but I never figured out how to wake up the graphics.

Do you have a spinning hard drive or an SSD in it?

ah, didnt try that yet

only usb currently

There is no internal drive installed in it?

only arrives tomorrow

A spinning hard drive might be head parking repeatedly and producing what might be a clicking or chirping noise.

it could be a low fi noise on the speakers but it seems to come from a different area

no hdd for sure ^^

I pulled my x201 off the shelf and booted it up and unfortunately found that I have since then (Sep 2023) installed a different OS on it.

i just put latest nomadbsd on a stick, if you're willing to do that ^^

It's good to boot things up every year or so however.

when i boot with acpi on graphics is garbled hrm

Sure. I can try nomadbsd on it. But you said it is okay with nomadbsd though, right?

no, linux is okay

so it's not the hardware per se

With nomadbsd you get a whining noise from it. Correct?

yes

as soon as it uses the stick?

but linux runs from a same model of stick

it sounds like an old line printer :D

really weird

Literally I have about 10 minutes as at the top of the hour I have to AFK for something. But I am always in this channel and you can reach me here later. Later might have to be tomorrow though.

i hope to get a ssd by tomorrow and will try a regular install

I don't recall it making noise with FreeBSD 13 kernel. Also other people reported having the suspend-resume work with the graphics okay. So I know that works. I just didn't have it working for me.

I am decompressing the current nomad image now.

coil whine is IO bound

I get it on SSDs when writing a lot to disc, at least on those laptops.

currently just a usb stick tho...

Also there is no x201 page but here is the x201i page: wiki.freebsd.org/Laptops/Thinkpad_X201i

That de-compress step took quite a few minutes. Writing it to a USB SD card now. Will try a quick boot of it.