I wonder whether I should try putting my own mutex around that line.

there's the problem, I bet

if you trace back through call sites, you land at elf_startup_loaded_object_callback

that itself is the result of startup_register_loaded_objects() calling dl_iterate_phdr

so you end up recursing on the lock, which we don't allow (but Linux does)

Ah.

There are 3 call sites in that file alone. I didn't notice that before.

Thank you, I know what to try.

yup, good luck

This exposes that I don't understand what the nested walking is supposed to accomplish...

the last time I had to deal with recursing in dl_iterate_phdr(), it's because they wanted the outer layer to ensure that nothing else was simultaneously loading objects

but that made more sense because it was a sanitizer runtime that knows nothing about what it's being used in

morning

a fantastic way to start my weekend -- broke my freebsd server

ran 'zpool upgrade' but forgot to upgrade the efi bootcode

system didn't boot, saying something about unsupported zfs capabilities

vext01: you need to boot from external media (e.g., installer), mount the EFI msdos partition and copy the new /boot/loader.efi into it

so booted a live usb stick and ran the 'gpart bootcode ...' command from the manpage

no don't do that!

am i hosed?

well, no, but gpart bootcode won't fix your problem, that's for old BIOS/CSM boot method

for UEFI the firmware loads the loader from the msdos partition

2!hemlock ~# ls -l /boot/loader.efi /boot/efi0/efi/boot/bootx64.efi

-rwxr-xr-x 1 root wheel 663040 Feb 17 11:00 /boot/efi0/efi/boot/bootx64.efi*

-r-xr-xr-x 2 root wheel 663040 Feb 28 17:08 /boot/loader.efi*

ok, before we continue, should i be worried that 'zpool status' dosn't show anything?

you need to copy the second file to the first file, basically

vext01: that's normal, you probably need to run 'zpool import'

oh phew

(maybe need zpool import -f)

i see my pools

right so i need to locate the msdos partition, is that with gpart?

yeah, so import your root pool, you need that to get the new loader.efi, but import it with an altroot: zpool import -f -R /mnt zroot

and yes, use gpart to locate the efi partition, it will look like this:

4!hemlock ~# gpart show ada0

=> 40 1953525088 ada0 GPT (932G)

40 2008 - free - (1.0M)

2048 532480 1 efi (260M)

if your zroot is on ada0 (or whatever) then the EFI partition is probably the first partition on the same disk

then mount it: mkdir /efi; mount -t msdos /dev/ada0p1 /efi # replace ada0p1 with the actual device

read only filesystem apparently

hm, try mount -u -orw /

ah, ill mount on /mnt

you can mount it on /mnt but then don't mount the zpool on /mnt as well :-)

if you want to do that, import the zpool first, then mount the msdos filesystem on /zroot/ROOT/default/boot/efi or something

(you may need to manually 'zfs mount zroot/ROOT/default' first)

i won't mount the zpool at all for now

hey, svnweb.freebsd.org/ports/head/sysutils cuts off after a certain number of ports. cgit.freebsd.org/ports/tree/sysutils works fine though

ivy: so the efi partition is index 1 of ada0, i ran 'mount -t msdos /dev/ada0p1 /mnt' and got 'mount_msdosfs: /dev/ada0p1 :invalid argument'

(having to type all this my hand :P )

could be because the fs is ro still?

it shouldn't, if /mnt exists you can mount on it

hrm

maybe my 'gpart bootcode' thing broke the partition?

vext01: maybe you need to 'kldload msdosfs'?

although that should be in GENERIC

says its already loaded

what gpart bootcode command did you run exactly?

the one from the manpage, let me type it out

also check dmesg, there may be a more informative error there

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

well, since your system is broken anyway and the EFI partition only contains a single file, you can just create a new filesystem there: newfs_msdos /dev/ada0p1 # make sure this is really your EFI partition before you do this, and not (say) your zroot

right

1 efi (260M)

^ sounds like the one

yeah, that seems right

ivy: by the way, i really appreciate this help

you are rescuing my weekend as we speak

wouldn't it be a better idea to figure out what's wrong with the current partition first

ok, new fs is mounted on /mnt

ring0_starr: i assume it was broken by installing BIOS bootcode, you're meant to install that on the freebsd-boot partition, not a FAT filesystem

makes sense

vext01: ok, mkdir /mnt/efi/boot; cp /boot/loader.efi /mnt/efi/boot/

my big fear about switching to freebsd for my server is that if i break thing, i'll have no idea how to fix them

assuming you're using installer media that's new enough

ive been using Macs for 35 years but freebsd, all i can really do is follow instructions

ivy: file copied

unmount and reboot?

huh

vext01: no, i gave you the wrong filename, sorry

vext01: it should be called /mnt/efi/boot/bootx64.efi

bootcode goes in the first sector of a FAT filesystem

(some firmwares might be able to load it from loader.efi but bootx64.efi is the standard name)

the first three bytes are defined by the FAT standard to be an x86 jmp instruction

lol

ok, renamed

vext01: you should be okay to reboot then

ok,here goes

pray for me

bootcode would have nothing to do with it. EFI looks for the GPT header at 0x1000

kernel booting...

there's still the pMBR in case somebody tries to legacy boot the disk to tell the user that they goofed up

ivy: you are my hero!

ring0_starr: i think the freebsd bootloader is too large to fit into a single FAT sector, it's meant to be installed on a dedicated partition

when you say bootloader you mean stage 3 zfsloader

ivy: you have literally saved my weekend

thank you, thank you

look at stand/i386... there's many bootloaders in various stages

ring0_starr: no, boot2

oh that's for ufs i think

wow bootloader situation on fbsd is confusing

there's so many versions of approximately the same thing

right, i think for zfs you use zfsboot instead of boot2? i try to avoid CSM boot as much as possible so i'm not 100% sure how this works

and uefi was supposed to take over as the bootloader

ok, i can now go back to what i was initially trying to fix all along

i have a directory on a zfs partition that is empty, but cannot be deleted

input/output error

zpool status lists this file as being corrupt also

what are my options?

i believe the hardware is OK, nothing in dmesg about failing disks

it's on a zfs mirror

you may need to ask fs@ about that, but zfs errors are generally fairly opaque and difficult/impossible to fix

yikes

i suppose i can start by scrubbing to see if anything else is busted

although, on the off chance you use zfs native encryption, this is expected due to a long-standing zfs bug that no one knows how to fix

oh, ez, just learn all about ZFS internal structure, do a deep dive into ZFS code, set up a kernel debugging environment, break on that error log, and get to it! so easy

so ez

shit's overcomplicated well past the point an actual human being can troubleshoot an issue like this

this specific zfs issue might be rare, but the probability of encountering any opaque issue while interacting with some kind of deep system is high

and they all require you to drop everything and go learn about some specific advanced niche topic

if i don't care about the corrupted file (i have a backup), can i just use `zfs destroy`?

it sounds like that would delete a whole filesystem, not the individual file

more info: forums.freebsd.org/threads/zfs-remove-a-stubborn-directory.97097

oh, it's waiting approval of course...

is there any way i can install unmet dependencies of a port using pkg, and install the port itself that i wish to make using ports?

somehow i always end up compiling perl

i'm really sick of those cutesy little p5- packages, by the way

pkg install <package>; pkg remove <package> and then build the port?

oh, scripting language dependency can get installed through the scripting language's manager

but there's system versions of the same thing

and mixing the two cause problems

i don't know how anybody has a functional computer

this chaos is unsustainable

agh

datasets in zfs are pretty cheap, so you can create them instead of directories and then zfs destroy would be the tool to delete it, but not sure if it works with datasets containing corrupted files, I guess it should, but that would have needed to decide on that setup before hand and probably fideling with zfs allow, if you want to do that for your user and not only for system directories

nimaje, that would work but the deps for building are different for runtime deps

use the build-depends-list target of the port

wait how would i get a list of deps from that

oh it literally does that nevermind

thanks

nimaje: i'd just really like to be able to delete that directory

like i say, i have a backup, but i don't want to restore the whole fs if only that dir is bust

I find edge cases. On an scp over a VPN, I get: ssh_ssh_dispatch_run_fatal: Connection to 10.1.0.17 port 22: message authentication code incorrect

Hi, I'm trying to find which program opened the port 2222 on one of my jails. If from the host I do a: sockstat -4 -l|grep 2222 I don't get anything. But I can do a telnet <IP> 2222 and it responds, but I don't know which protocol it uses, because I cannot ssh, nor http to that IP:Port

martinrame: hello, try "sockstat -l -j jid

on the host

Hi mzar I found that my host is connected to that IP:PORT, now I need to find which program is connected.

Hey there! I wonder, when using blacklistd with a VNET jail, I presume I should have blacklistd running in the jail too? As opposed to jails that share networking with the host system and can just link up to the host system's blacklistd socket

OK, then check just "sockstat -j jid"

Dooshki: so it is jail without vnet, with firewall on the host and blacklistd running on the host ?

mzar: No, a jail with vnet, with a firewall in each jail

Dooshki: if the jail has own TCP stack and firewall (VNET jail), then you manage this firewall from the jail and you have to run blacklistd inside this jail

Alright, thanks for the pointer!

jail with VNET adds overhead, has slower network performance and consumes more CPU cycles

do you really need this VNET Dooshki ?

I'm not about to spend the entire weekend re-doing my entire home server :P

OK

I am running a lot of jails, and only few are VNET ones

dvl: I responded via e-mail, I hope it will help you to troubleshoot this issue

Happy International Women’s Day, for those who celebrate !

mzar: I believe the reason why I went for VNET jails when I set them up a year ago was to have fine-grained control over what sort of networking capabilities compromised applications within the jails could have

Phew, it works! Didn't need to spend the entire weekend on it :)

ivy: about that file i can't delete. Do you think making a new dataset and copying everything over, then restoring the corrupted file from backup will work?

Hi again mzar, it looks like there are no jails opening the 2222 port. I wonder why when I call telnet <IP> 2222 I get a response, can that be the firewall, pf in this case.

I stopped the jail with problems and now there's no telnet to it. Now I'm running Sniffnet in the host and I see there's still showing connections to the IP of that jail...

maybe it's something listening on the host on port 2222

with telnet I cannot reach that port on the IP (nor the loopback) of the host.

if the jail shares TCP stack with the host (non-VNET jail), then it can behave this way

interesting story

yes I wonder if Sniffnet caches resultsd

mzar: I restarted it and that IP is no longer showing

OK

now there's a jail connected to the outside, how can I know which program is doing that?

Excellent, it's working even after a reboot :D

why it wouldn't ?

mzar: most likely something wrong with the three different instances of /etc/rc.conf

(one host, two jails)

that's 100% correct

And yeah, thanks again for pointing me in the right direction!

mzar: email regarding openvpn and scp issues?

hmmm so after playing with the installer it appears for FDE with keyfile on a usb you got to drop into the shell

and also manually partition

orrr I believe instead could set the keyfile after install

that would likely be the better option

I am going to just do the latter, although I would be curious to do an install without bsdinstall to learn more indepth the freebsd system, anyone got any guides/articles on bsdinstall-less install?

arch linux style :P

mzar: I updated one client from FreeBSD 14.1 to FreeBSD 142 - problem seems to have gone away.

mzar: I'll work on upgrading the other hosts soon. It would be good to know WHY this upgrade fixes the issue.

do we have "gretap" (Ethernet encap over GRE) in FreeBSD?

looks like no, i guess vxlan(4) is the alternative

I'm trying to secure one internet exposed jail with pf and found when I check a port using: telnet <ip> <port> it shows the message "Connected to ....Escape character is..", how can I block that?

It doesn't matter which port I point, even non existent ports, it always returns that connection message.

I have a block in rule and only allow access (tcp and udp) to two specific ports.