want me to try anything with bsdinstall_chroot or go right to the ls -l kevans?

just go straight there

k, building

destructing

ok for ls and cat got: /tmp/bsdinstall-install*: no such file or dir

which branch are you installing, to be precise?

oh right, in the chroot it doesn't have that prefiz

i'm installing from 13.3 repo pull

releng/13.3

just checked and up to date with that branch

hmm

everything should be mounted at that point... the lack if any bsdinstall-install prefixed files would seem to ndicate that we are infact chrooted

what else can i do to help you debug? i'm at the extent of my knowledge

maybe inspect the fs state a bit in the scripr

ls -l /mnt /usr/local

is that /mnt/usr/local or 2 separate paths like you typed?

two separate

k building

well /mnt has 0 and /usr/local has bin etc share www

is there some kinda wait command that'llmake the setup script pause for 1 second? it's kinda hard to catch the data sometimes

or sleep?

ok, so you are chrooted

but then why is the thing you're expecting not there

oh this doesn't have that. this is way stripped down to just what i pasted

and the extras distro is empty just to simplify testing

know how i can make the setup part sleep? i can add the other stuff back in and test

right, but /mnt is empty

just sleep, yeah

oh lol

building

kevans: termbin.com/lemi

putting jail dir at / root for less typing

seems when i create a zfs mountpoint, it doesn't use the chroot because of the altroot property?

it is in the chroot, though

ls -la /mnt

# --> empty

so the altroot property is affecting how it works IN the chroot

this feels like a bug so far

is it INTENDED for zfs altroot to be set to /mnt during the setup phase of a scripted bsdinstall?

almost certainly, yes

so what do you think the bug is?

oh, I see what happened now

ls -la /

# --> 22 items (no "/jail")

er

the next one, /mnt/jail exists

ya, the new /jail mountpoint is being created in the /mnt dir, not at /

because it looked up altroot but didn't consider that it's in a chroot

because of altroot

yep

what do?

are you sure it's not working as intended? if the altroot is /mnt, shouldn't it be creating the mountpoint in /mnt/jail instead of /jail? it's still within the chroot

it is, in a way

why is altroot even set? i remember asking about it in #zfs last week because i expected the mountpoint to be off of /, and they said it was in an altroot because during install it's on a tmproot or smth. but if it's in chroot, why does it need altroot at all?

because you can't let it mount at /

you can't chroot until it's mounted, and you can't let it mount at /

therefore you need an altroot to safely move it out of the way

why can't let it mount at /?

because you don't want it covering the install media while you're trying to build it out

feels like some part of the design here is painting us into a corner

if not then tell me how i can start and jexec against a jail in the installerconfig's setup phase?

try symlinking /mnt/jails into the root

you might need to cleanup the symlink on success, not sure yet

whoa

lemme try

alternative `zfs create -u` and just mount it manually

with symlink method i gotta clean it up before rebooting, with zfs create -u it's a temporary change that i don't have to clean up. i think the latter sounds cleaner

ill try symlink method first just to see if it works

you can also copy ZFSBOOT_DATASETS out of the bsdinstall scripts and set that in your script

adding in your jails dataset or any other you need

how would that fix it?

because ZFSBOOT_DATASETS get created by the installer before your script runs

thus avoiding the chroot issue entirely

so it wouldn't have the /mnt issue?

it would just be right off /

btw man page for bsdinstall ZFSBOOT_DATASETS seems to have an error. it says "about who to write" should be "about how to write" ya?

it's since been changed to "how to populate"

nice!

i gotta sleep sadly but the symlink at least shows up in ls -la like /jail -> /mnt/jail. i'll test if that lets the jail start and work normally tomorrow

tyvm for helping me

I have question related to jails exec.clean command parameter, specifying this parameter in jail.conf means any command parameters that runs in system environment should be run in clean environment? except USER, HOME, SHELL and TERM?

padukajorat, Commands that run through the jail commands, yes. Obviously commands that run normally inside the jail have their normal environment. But clean is like "env -i" with those named variables passed through.

This is documented in the jail(8) man page.

yeah, i went through the jail(8) man page, kind of confused i am, let us assume i am using exec.prepare="" so this exec.prepare will run in clean environment? if i have exec.clean set in jail.conf?

hm one thing what i have not played around is.. how i can control device led lights on freebsd?

does not seem to have much information on net

ivaat: there's led(4), but i don't know what drivers support it (em(4) does at least)

apparently they should show up in /dev/led if available

ls: /dev/led: No such file or directory

ok its duck tape then

the mini pc has so bright blue led :)

it might be one of these things where you have to load a random kmod to get the driver, a lot of system/i2c/whatever stuff is like that

okey

i will dig around

i want to build a dns/knot package but i can't until the build i started last night finished... currently doing electron28 and qt6-webengine. bleh. i hate this

can anyone help us compile freebsd?

Soni: have you read docs.freebsd.org/en/books/handbook/cutting-edge/#makeworld ?

Title: Chapter 26. Updating and Upgrading FreeBSD | FreeBSD Documentation Portal

lw: yeah it runs out of ram

(with -j1 ofc)

well, you may need to add more memory, i'm not sure there's any other workaround for that

how much?

i don't know what the minimum is, although kevans said 2GB when you asked yesterday i think? which sounds reasonable to me... the host i normally build on has 6GB and i've not had a problem there

oh only 2GB? okay

here we go building llvm again... this takes a while

well you can add swap too but then it will not be fast :)

just mentioning if someone is testing-doing on vs for example

vps

we are using a VM

what kind of sensitive information might one find in dmesg that should be redacted prior to publicly sharing it?

jbo: hopefully, nothing

maybe MAC addresses if you are super paranoid

lw, I thought so but was wondering since: bugs.freebsd.org/bugzilla/show_bug.cgi?id=278311#c9

Title: 278311 – amdtemp: Does not recognize AMD Threadripper 7960X

curious, i am not sure what that could be referring to

i know some people consider their IP addresses and stuff to be "sensitive information" though

(hackers can find your IP address!!!)

well I mean...

whatever?

jbo: delphij seems quite friendly IME, maybe just ask :-)

true

lw, hw.model: AMD Ryzen Threadripper 7960X 24-Cores :x

jbo: go away

( :-) )

yeah, typically nothing but it's nice to remind people

It's also very surprising to me what other people get upset about and then want to have it removed. After they already emailed it to 2000 people on a mailing list.

And of course sometimes I am on the admin side doing the redaction. And sometimes I am on the gawking side where I run over there quickly to see what it is that they want to hide from other people before it gets redacted!

right, in my experience it's better to act casual after you've accidentally leaked something really important :-)

lw, ok :(

jbo: you are gonna be so jealous when my Chinese router arrives from AliExpress, i bet it has more 10G Ethernet ports than you!!

lw, much jealous

lw, does your chinese router also have this?

kern.smp.cpus: 48

:D

My IP address is 127.0.0.1. Everyone here is allowed to use it to DoS me.

lw, I did decide not go get a new GPU yet tho. so running a 8 years old GPU

my workstations are ancient

whenever I get round to replacing the oldest I'm planning on just using amdgpu

no aircon and it gets hot in the office in summer

yep

the +350W TPD of the new CPU was a major "problem"

jbo: i have an NVidia GT710 if you want to borrow it?

lw, nah I just migrated the GPU I have/had in my previous desktop (Nvidia Quadro P5000)

picture: bin.jvnv.net/file/mG6y0

your computer looks like a borg drone

thank you

what's the lower PCIe card? an SSD?

also did you go with a different motherboard, i thought you were looking at one with like 10 PCIe slots

top to bottom: GPU, 4x PCIe m.2 Gen5 adapter card, 4x USB dedicated controller card (for bhyve pass-thru)

lw, that is when I was still considering to go Intel Xeon w5-3400 or w7-3400

ah right, you did AMD

but eventually I moved to threadripper

yeah, this is my first ever AMD machine

and yes, it only boots FreeBSD.

i heard on questions@ about some issues with TR recently, i hope you don't run into that

lw, there are 4x 4TB SATA drives in the very bottom (not visible)

that was about it not booting at all though so i guess if it boots it's fine :-)

lw, what kind of issue(s)?

ah

single boot would be a bit of a luxury

- L- 853/3265: Chris Torek 14-stable on AMD7950X: Good and bad news

thread on -stable (not -questions)

ah, that was specifically with amdgpu not working

that is a ryzen, not a threadripper

this one has no GPU

wait what

7950X is a Ryzen, 7960X is a TR?

yeah 7960X is threadripper

wtf is your product naming doing, AMD

yep... don't ask. I was also like " wut"

I mean, they do seem to call it "AMD Ryzen Threadripper 7960X"

so, kinda also a ryzen?

ok 7950X has 16 cores, i guess if 7960X has 24 cores that sort of makes sense

i thought in the past TR was a completely different CPU though (other than the Zen core itselF)

anyway, I hope to get some time over the weekend to setup the machine. too busy right now.

like, different socket and stuff

it is a very different socket.

this CPU is HUGE

jbo: do you have ECC memory? (and do AMD actually certify ECC on TR?)

lw, it's ECC memory, yes (real ECC, not the regular DDR5 "on-die ECC". I yet need to figure out whether it actually performs as ECC memory.

i've been wondering that too but i'm not sure how to test it other than hitting one of the chips with a hammer

btw the CPU ships with a torque wrench in the box

that's funny

i assume that's for attaching the heatsink, not for bolting it into the socket

the latter :D

...

it's not a ZIF socket?

the three screws

it's LGA

ah

that seems weirdly janky for a modern CPU, are Xeons like that too?

not the xeons I have worked with. but I haven't had anything new since 2017

I'm completely fine with it tho.

I like the torque wrench approach.

but I'm an electrical engineer so...

i'm okay with a torque wrench for a bolt but with a screw head like that it seems like an invitation to strip the head if you do it wrong

once I had some time to setup the system and ensure that everything works well I'll try to do some overclocking. That's why I went for 2x 360mm radiators

it's torx. hard to screw that one up.

jbo: so can i have a shell account to build ports?

it's the least you can do when i helped you commit all those ports

lw, no :>

lw, but I should be faster at building ports now.

it did a buildworld in 404 seconds

with clang? i don't believe that actually

it takes me about an hour to build clang and i don't believe your CPU is that much faster :-)

although i do have HT disabled

well, it's what I observed. but I had an extremely busy week and only played around with it for a little while yesterday. will do "proper testing" over the weekend

the first run was 418 seconds

I had a hard time believing it too

it felt like a reverse scam

rm -rf your obj directory and try again and make sure it builds toolchain, not just src... if it can really do that in 400 seconds i will reprioritise things to buy a new build system

lw: we've got a box that builds freebsd in ~5 minutes

CPU: AMD Ryzen 7 5800X3D 8-Core Processor (3400.15-MHz K8-class CPU)

lw, it definitely built clang too.

this is not a TR but it's also not super slow

lw, this is 24 cores, 48 threads at sustaining 4.2 GHz

and PCIe gen5 SSDs in mirror (if that matters (probably not))

jbo: yeah but proportionally, if it takes me an hour with 8 cores, shouldn't it take longer for you on 24 cores

or am i bad at maths

hmm it is true the SSD in this system is hilariously slow, that probably doesn't help... i've been meaning to replace that

the base system is on two Crucial T700 disks

i think freebsd build caps at -j64 or so, no big gains past that

i don't recall the exact number, but 64 rings a bell

I still 'like' the feel of -j80

kevans: -j256 is also satisfying, but not very useful :D

lw, just did a make clean, rm -rf /usr/obj, make -j48 buildworld, ttyl.

jbo: hopefully not too much later, lmk

i waste so much time waiting for kernel builds i really do need something faster (although i knew that already...)

lw, kernel builds in what feels like < 60 seconds

GENERIC, that is.

lw: do you use META_MODE?

dstolfa: yes, but building all of a kernel still takes a while

let me do a quick test to compare...

my incremental builds take around 2 seconds for kernel

fresh takes around... i wanna say ~2 minutes?

and this is on a 16c/32t xeon

it does have nvme storage and plenty of RAM, though

i ran into this earlier when i merged main into my local branch to test a one-line change and it recompiled the entire thing for whatever reason, even with meta mode

make -j12 KERNCONF=GENERIC buildkernel 756.05s user 135.85s system 738% cpu 2:00.72 total

so 2 minutes, not really that bad, it just feels slow

the youngsters these days are so impatient...

hey, i built electron28 earlier, that took like 10 hours

lw: yes, but do you have to do it every time you make a single line change? :P

dstolfa: it basically feels like it yes... probably because it depends on so many things

>>> World build completed on Fri Apr 12 19:56:38 UTC 2024

>>> World built in 412 seconds, ncpu: 48, make -j48

that is with non-OC and stock RAM timings

ok, i jsut deleted my /src/obj so let me try a buildworld to compare

(no EXPO)

i'm currently dealing with building v8 on morello, having to change a header to test something and it basically kicks off a fresh build because that change is inlined everywhere

pain

fun build error of the day, i guess this is caused by the llvm 18 update bugs.freebsd.org/bugzilla/show_bug.cgi?id=278336

Title: 278336 – multimedia/ringrtc: build failure

lw, are we still waiting for your build to finish?!

jbo: PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU COMMAND

6425 lexi 1 102 0 344M 293M CPU1 1 0:04 32.18% clang

24394 lexi 1 101 0 291M 258M RUN 7 0:04 30.27% clang

33559 lexi 1 98 0 291M 236M CPU5 5 0:03 22.27% clang

52510 lexi 1 98 0 259M 220M RUN 1 0:03 22.17% clang

58132 lexi 1 96 0 259M 212M CPU4 4 0:02 19.29% clang

66594 lexi 1 95 0 231M 181M RUN 2 0:02 15.58% clang

60076 lexi 1 94 0 231M 178M CPU3 3 0:02 15.09% clang

84965 lexi 1 92 0 170M 130M CPU2 2 0:01 10.16% clang

54243 lexi 21 20 0 1115G 211M uwait 7 17:47 0.88% QtWebEngineProcess

73098 lexi 1 68 0 36M 16M wait 1 0:00 0.88% ccache

49693 lexi 1 68 0 33M 14M wait 4 0:00 0.68% ccache

holy mackarel

ok, phew

... i did not mean to paste that

jbo always makes me paste stuff wrongly

so you still didn't learn how to paste? :D

ahahahaha

let's try this again

jbo: le-fay.org/tmp/30d/HMW95f.txt

i really need to script my IRC client to intercept multi-line paste and not send them

although on the subject of top output, look at this

54243 lexi 21 20 0 1115G 211M uwait 7 17:47 0.88% QtWebEngineProcess

that's 1.1TB of virtual memory

how is that even possible?

like, what is it even doing to map that much memory

hmpf, I don't seem to be able to log into phabricator anymore

was there something happening? I was mostly "AFK" the past few weeks

my account seems ok, or at least i can load the front page

never mind... it's not using kerberos

PEBKAC

jbo: 24 CPUs and you can't even log into a website? smh my head

lw, :D

lw, still on my old system. only SSH'ing into the new one to buildworld for you

meanwhile, buildworld still on clang

so glad this builds the PowerPC and LoongArch targets for all those PPC and MIPS systems i don't have

we could play factorio in the meantime

i would but i need to build dns/knot

have literally been waiting all day for electron to finish building so i can do that

lw, still building world?

oh no it finished

>>> World built in 2919 seconds, ncpu: 8, make -j12

kevans: ok back at it i'll tell you if the symlink method let jails start

lw, still faster than my current shit system

jbo: remind me what your score was on the TR?

or did you not try buildworld yet

lw, wtf? the 404 seconds was TR

ah right

or 412 in the run I did just for you earlier

2929/404 = 7.25

24/8 = 3

so your base CPU performance is 7.25/3 = 2.417x faster than mine?

alepzi: thanks; if not, `zfs create -u` to prevent it from mounting and just mount it yourself after it's created (or the ZFSBOOT_DATASETS we talked about)

lw, keep in mind that mine is powered entirely by hydroelectric power

yours are like 5GHz turbo right, i guess that's about right

nod

5800X3D has a really low turbo clock even for Ryzen

kevans, how's the ecc stuff going?

jbo: just have to do the pkg(7) side now

kevans, nice :)

I looked at the libstdc++ bits and man, this is a royal pain in the ass

:((((((

would you happen to enjoy having royal pain in the ass?

figuring out with gcc's build system how to build just libstdc++, I'm not yet convinced that it's possible

kevans, wasn't there an alternative approach we discussed?

what are you two working on?

i hear libstdc++, i am interested

I don't typically enjoy it, no :-)

bare metal libstdc++ built against newlib

we have arm-none-eabi-newlib, but arm-none-eabi-libstdc++ would unlock some stuff jbo is trying to do

lw, I was complaining to kevans@ that arm-none-eabi-gcc is "useless" in it's current form :p

ah

s/gcc/newlib

I think I had it building, but I think it wasn't actually using newlib

so because kevans is a lazy slacker I had to get a $6k PC upgrade to spawn VMs just to compile against Cortex-M targets :p

:/

$6k i guess you're an ecc enjoyer?

tbf the projects I was derailing on were significantly more fun

kevans,

alepzi, yes, ECC RAM. I'll never get non-ECC ever again. but I went for cheapo RAM because money is a problem at the moment.

ya same

"i bought a new $6k PC" "money is a problem at the moment"

$6k turkish lira

well technically it's the company's PC, not mine

and my current "workstation" is 7 years old

hmpf... I don't think that I can explain this in a way that would make it sound better

let's just change topics

pwned

[#freebsd] <jbo> lw, keep in mind that mine is powered entirely by hydroelectric power

wait you're Canadian?

no he's from dubai where they burn oil to pump water up a hill then let it run down a hydroelectric dam for Clean Energy

lw, as you know. yes.

(I'm swiss)

the canadians of europe

here in the alps where I'm located, we get 96% hydroelectric and 4% solar power

nice

not if you have to pay for it

you'd think they could combine those two things are boil the water with solar power to generate electricity

also, did you just assume my gender? >:D

jbo: so do you get 25Gbps Internet at home? i heard swiss people have this

lw, yes. although I only have 10Gbps at home.

I don't have the necessary hardware

are you looking for a housemate? i have switches

i'll come

lw, I actually am, yes.

jbo bsd flophouse

jbo: ok, can you move into our house and bring your internet with you?

I am currently in a 4.5 bed room 150m2 penthouse apartment I'm moving out of because it's too huge ass for living there alone (and again, money is a problem)

lw, not sure whether your bf would like that but sure

anyone who seriously wants to move in here with me has exactly 6 days to decide because my current plan is to move to a tiny one bedroom apartment soon

using virtio-9p with bhyve. works great, but i noticed on reboot (of the FreeBSD host) the guest VM sometimes has failed to mount the passed through filesystems. i can connect to the guest "mount -a" and all is well. i'm guessing it's a timing issue on the FreeBSD host at boot. If I just reboot the VM, everything also works fine, i.e., it's only on an actual host reboot I have the problem.

markmcb: what provides the filesystem on the host?

(UFS, ZFS, NFS, ...?)

zfs

what are you using to start bhyve?

vm: vm_enable="YES"

hmm, that's more or less the same config i use (zfs, vm-bhyve) and i haven't noticed this issue

markmcb: out of interest, if you edit /usr/local/etc/rc.d/vm to add 'LOGIN' to 'REQUIRE', does that fix the problem?

(this is my go-to method to fix rc ordering issues since LOGIN runs after everything has started... although tbh i can't imagine that's the issue here)

the other complexity is this guest is Ubuntu 24.04 running Incus (formerly LXD) ... another guest without Incus and also using 9p never has this issue

lw: I'll try the LOGIN tweak

i notice that rc script has a strange 'BEFORE' - 'dnsmasq ipfw pf'?

i'm not sure why you'd want VMs to start before those services

yeah, i was wondering about that too

this may be nothing to do with the issue but i would try removing that as well

although i think zfs filesystems should be mounted well before any of those

unrelated, has anyone noticed this on 15-CURRENT?

Apr 12 21:42:35 amaranth node_exporter[4926]: ts=2024-04-12T21:42:35.787Z caller=collector.go:169 level=error msg="collector failed" name=zfs duration_seconds=8.2479e-05 err="couldn't get sysctl: no such file or directory"

all my node_exporters are logging this, even on systems that are definitely using zfs

node_exporter_args="--no-collector.zfs" fixes it but i don't understand why it doesn't just work

lw: that seems to have done the trick, thanks for the work around!

markmcb: that's odd, i suggest filing a bug against sysutils/vm-bhyve because this shouldn't be required

will do

kevans: CONFIRMING that ln -s /mnt/jail /jail was enough to be able to start a jail in the setup phase of a scripted bsdinstall and run jexec shit on it

now i'll rip that out and switch to the zfs create -u method