00:00:46 https://github.com/clasp-developers/clasp/blob/a902594fbc1b5c40e2b82d45a6782ef33f8c6c47/src/core/debug_unixes.cc#L225 00:02:01 I wonder whether I should try putting my own mutex around that line. 00:03:30 there's the problem, I bet 00:03:49 if you trace back through call sites, you land at elf_startup_loaded_object_callback 00:04:03 that itself is the result of startup_register_loaded_objects() calling dl_iterate_phdr 00:04:24 so you end up recursing on the lock, which we don't allow (but Linux does) 00:04:42 Ah. 00:05:16 There are 3 call sites in that file alone. I didn't notice that before. 00:05:31 Thank you, I know what to try. 00:05:44 yup, good luck 00:15:19 This exposes that I don't understand what the nested walking is supposed to accomplish... 00:46:25 the last time I had to deal with recursing in dl_iterate_phdr(), it's because they wanted the outer layer to ensure that nothing else was simultaneously loading objects 00:46:39 but that made more sense because it was a sanitizer runtime that knows nothing about what it's being used in 08:54:56 morning 08:56:02 a fantastic way to start my weekend -- broke my freebsd server 08:56:42 ran 'zpool upgrade' but forgot to upgrade the efi bootcode 08:57:23 system didn't boot, saying something about unsupported zfs capabilities 08:57:52 vext01: you need to boot from external media (e.g., installer), mount the EFI msdos partition and copy the new /boot/loader.efi into it 08:57:55 so booted a live usb stick and ran the 'gpart bootcode ...' command from the manpage 08:58:07 no don't do that! 08:58:22 am i hosed? 08:58:37 well, no, but gpart bootcode won't fix your problem, that's for old BIOS/CSM boot method 08:58:47 for UEFI the firmware loads the loader from the msdos partition 08:59:37 2!hemlock ~# ls -l /boot/loader.efi /boot/efi0/efi/boot/bootx64.efi 08:59:38 -rwxr-xr-x 1 root wheel 663040 Feb 17 11:00 /boot/efi0/efi/boot/bootx64.efi* 08:59:38 -r-xr-xr-x 2 root wheel 663040 Feb 28 17:08 /boot/loader.efi* 08:59:47 ok, before we continue, should i be worried that 'zpool status' dosn't show anything? 09:00:00 you need to copy the second file to the first file, basically 09:00:09 vext01: that's normal, you probably need to run 'zpool import' 09:00:23 oh phew 09:00:26 (maybe need zpool import -f) 09:00:27 i see my pools 09:01:02 right so i need to locate the msdos partition, is that with gpart? 09:01:05 yeah, so import your root pool, you need that to get the new loader.efi, but import it with an altroot: zpool import -f -R /mnt zroot 09:01:39 and yes, use gpart to locate the efi partition, it will look like this: 09:01:42 4!hemlock ~# gpart show ada0 09:01:42 => 40 1953525088 ada0 GPT (932G) 09:01:42 40 2008 - free - (1.0M) 09:01:42 2048 532480 1 efi (260M) 09:01:59 if your zroot is on ada0 (or whatever) then the EFI partition is probably the first partition on the same disk 09:02:43 then mount it: mkdir /efi; mount -t msdos /dev/ada0p1 /efi # replace ada0p1 with the actual device 09:03:16 read only filesystem apparently 09:03:24 hm, try mount -u -orw / 09:03:26 ah, ill mount on /mnt 09:03:41 you can mount it on /mnt but then don't mount the zpool on /mnt as well :-) 09:04:20 if you want to do that, import the zpool first, then mount the msdos filesystem on /zroot/ROOT/default/boot/efi or something 09:04:36 (you may need to manually 'zfs mount zroot/ROOT/default' first) 09:04:56 i won't mount the zpool at all for now 09:05:02 hey, https://svnweb.freebsd.org/ports/head/sysutils/ cuts off after a certain number of ports. https://cgit.freebsd.org/ports/tree/sysutils works fine though 09:06:07 ivy: so the efi partition is index 1 of ada0, i ran 'mount -t msdos /dev/ada0p1 /mnt' and got 'mount_msdosfs: /dev/ada0p1 :invalid argument' 09:06:21 (having to type all this my hand :P ) 09:06:43 could be because the fs is ro still? 09:07:02 it shouldn't, if /mnt exists you can mount on it 09:07:19 hrm 09:07:45 maybe my 'gpart bootcode' thing broke the partition? 09:07:47 vext01: maybe you need to 'kldload msdosfs'? 09:07:59 although that should be in GENERIC 09:08:07 says its already loaded 09:08:19 what gpart bootcode command did you run exactly? 09:08:30 the one from the manpage, let me type it out 09:08:58 also check dmesg, there may be a more informative error there 09:09:15 gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0 09:10:22 well, since your system is broken anyway and the EFI partition only contains a single file, you can just create a new filesystem there: newfs_msdos /dev/ada0p1 # make sure this is really your EFI partition before you do this, and not (say) your zroot 09:10:44 right 09:11:13 1 efi (260M) 09:11:17 ^ sounds like the one 09:11:19 yeah, that seems right 09:11:36 ivy: by the way, i really appreciate this help 09:11:45 you are rescuing my weekend as we speak 09:12:28 wouldn't it be a better idea to figure out what's wrong with the current partition first 09:12:42 ok, new fs is mounted on /mnt 09:12:52 ring0_starr: i assume it was broken by installing BIOS bootcode, you're meant to install that on the freebsd-boot partition, not a FAT filesystem 09:13:10 makes sense 09:13:13 vext01: ok, mkdir /mnt/efi/boot; cp /boot/loader.efi /mnt/efi/boot/ 09:13:34 my big fear about switching to freebsd for my server is that if i break thing, i'll have no idea how to fix them 09:13:34 assuming you're using installer media that's new enough 09:13:56 ive been using Macs for 35 years but freebsd, all i can really do is follow instructions 09:14:12 ivy: file copied 09:14:20 unmount and reboot? 09:14:37 huh 09:14:38 vext01: no, i gave you the wrong filename, sorry 09:14:46 vext01: it should be called /mnt/efi/boot/bootx64.efi 09:15:06 bootcode goes in the first sector of a FAT filesystem 09:15:09 (some firmwares might be able to load it from loader.efi but bootx64.efi is the standard name) 09:15:23 the first three bytes are defined by the FAT standard to be an x86 jmp instruction 09:15:26 lol 09:15:49 ok, renamed 09:15:56 vext01: you should be okay to reboot then 09:16:00 ok,here goes 09:16:04 pray for me 09:16:27 bootcode would have nothing to do with it. EFI looks for the GPT header at 0x1000 09:16:36 kernel booting... 09:17:10 there's still the pMBR in case somebody tries to legacy boot the disk to tell the user that they goofed up 09:17:23 ivy: you are my hero! 09:17:32 ring0_starr: i think the freebsd bootloader is too large to fit into a single FAT sector, it's meant to be installed on a dedicated partition 09:18:01 when you say bootloader you mean stage 3 zfsloader 09:18:18 ivy: you have literally saved my weekend 09:18:29 thank you, thank you 09:18:51 look at stand/i386... there's many bootloaders in various stages 09:18:53 ring0_starr: no, boot2 09:20:16 oh that's for ufs i think 09:22:16 wow bootloader situation on fbsd is confusing 09:22:36 there's so many versions of approximately the same thing 09:22:57 right, i think for zfs you use zfsboot instead of boot2? i try to avoid CSM boot as much as possible so i'm not 100% sure how this works 09:23:17 and uefi was supposed to take over as the bootloader 09:23:48 ok, i can now go back to what i was initially trying to fix all along 09:24:03 i have a directory on a zfs partition that is empty, but cannot be deleted 09:24:35 input/output error 09:24:47 zpool status lists this file as being corrupt also 09:25:02 what are my options? 09:25:20 i believe the hardware is OK, nothing in dmesg about failing disks 09:25:30 it's on a zfs mirror 09:26:11 you may need to ask fs@ about that, but zfs errors are generally fairly opaque and difficult/impossible to fix 09:26:32 yikes 09:26:46 i suppose i can start by scrubbing to see if anything else is busted 09:27:25 although, on the off chance you use zfs native encryption, this is expected due to a long-standing zfs bug that no one knows how to fix 09:27:58 oh, ez, just learn all about ZFS internal structure, do a deep dive into ZFS code, set up a kernel debugging environment, break on that error log, and get to it! so easy 09:28:40 so ez 09:29:28 shit's overcomplicated well past the point an actual human being can troubleshoot an issue like this 09:30:20 this specific zfs issue might be rare, but the probability of encountering any opaque issue while interacting with some kind of deep system is high 09:30:54 and they all require you to drop everything and go learn about some specific advanced niche topic 09:41:56 if i don't care about the corrupted file (i have a backup), can i just use `zfs destroy`? 09:43:46 it sounds like that would delete a whole filesystem, not the individual file 09:51:45 more info: https://forums.freebsd.org/threads/zfs-remove-a-stubborn-directory.97097/ 09:53:49 oh, it's waiting approval of course... 11:15:00 is there any way i can install unmet dependencies of a port using pkg, and install the port itself that i wish to make using ports? 11:15:19 somehow i always end up compiling perl 11:15:42 i'm really sick of those cutesy little p5- packages, by the way 11:16:05 pkg install ; pkg remove and then build the port? 11:16:27 oh, scripting language dependency can get installed through the scripting language's manager 11:16:35 but there's system versions of the same thing 11:16:44 and mixing the two cause problems 11:16:57 i don't know how anybody has a functional computer 11:17:06 this chaos is unsustainable 11:18:32 agh 11:18:49 datasets in zfs are pretty cheap, so you can create them instead of directories and then zfs destroy would be the tool to delete it, but not sure if it works with datasets containing corrupted files, I guess it should, but that would have needed to decide on that setup before hand and probably fideling with zfs allow, if you want to do that for your user and not only for system directories 11:18:51 nimaje, that would work but the deps for building are different for runtime deps 11:19:53 use the build-depends-list target of the port 11:22:31 wait how would i get a list of deps from that 11:23:02 oh it literally does that nevermind 11:23:05 thanks 11:53:22 nimaje: i'd just really like to be able to delete that directory 11:53:59 like i say, i have a backup, but i don't want to restore the whole fs if only that dir is bust 12:09:00 I find edge cases. On an scp over a VPN, I get: ssh_ssh_dispatch_run_fatal: Connection to 10.1.0.17 port 22: message authentication code incorrect 16:15:45 Hi, I'm trying to find which program opened the port 2222 on one of my jails. If from the host I do a: sockstat -4 -l|grep 2222 I don't get anything. But I can do a telnet 2222 and it responds, but I don't know which protocol it uses, because I cannot ssh, nor http to that IP:Port 16:21:59 martinrame: hello, try "sockstat -l -j jid 16:22:13 on the host 16:23:16 Hi mzar I found that my host is connected to that IP:PORT, now I need to find which program is connected. 16:25:33 Hey there! I wonder, when using blacklistd with a VNET jail, I presume I should have blacklistd running in the jail too? As opposed to jails that share networking with the host system and can just link up to the host system's blacklistd socket 16:25:51 OK, then check just "sockstat -j jid" 16:27:09 Dooshki: so it is jail without vnet, with firewall on the host and blacklistd running on the host ? 16:28:31 mzar: No, a jail with vnet, with a firewall in each jail 16:29:05 Dooshki: if the jail has own TCP stack and firewall (VNET jail), then you manage this firewall from the jail and you have to run blacklistd inside this jail 16:30:11 Alright, thanks for the pointer! 16:30:28 jail with VNET adds overhead, has slower network performance and consumes more CPU cycles 16:30:59 do you really need this VNET Dooshki ? 16:31:01 I'm not about to spend the entire weekend re-doing my entire home server :P 16:31:08 OK 16:31:29 I am running a lot of jails, and only few are VNET ones 16:32:26 dvl: I responded via e-mail, I hope it will help you to troubleshoot this issue 16:33:07 Happy International Women’s Day, for those who celebrate ! 16:43:56 mzar: I believe the reason why I went for VNET jails when I set them up a year ago was to have fine-grained control over what sort of networking capabilities compromised applications within the jails could have 16:47:45 Phew, it works! Didn't need to spend the entire weekend on it :) 17:08:13 ivy: about that file i can't delete. Do you think making a new dataset and copying everything over, then restoring the corrupted file from backup will work? 17:14:17 Hi again mzar, it looks like there are no jails opening the 2222 port. I wonder why when I call telnet 2222 I get a response, can that be the firewall, pf in this case. 17:16:57 I stopped the jail with problems and now there's no telnet to it. Now I'm running Sniffnet in the host and I see there's still showing connections to the IP of that jail... 17:17:15 maybe it's something listening on the host on port 2222 17:18:10 with telnet I cannot reach that port on the IP (nor the loopback) of the host. 17:18:15 if the jail shares TCP stack with the host (non-VNET jail), then it can behave this way 17:18:31 interesting story 17:18:42 yes I wonder if Sniffnet caches resultsd 17:19:32 mzar: I restarted it and that IP is no longer showing 17:19:48 OK 17:22:43 now there's a jail connected to the outside, how can I know which program is doing that? 17:25:48 Excellent, it's working even after a reboot :D 17:27:13 why it wouldn't ? 17:31:18 mzar: most likely something wrong with the three different instances of /etc/rc.conf 17:31:29 (one host, two jails) 17:32:46 that's 100% correct 17:51:25 And yeah, thanks again for pointing me in the right direction! 18:11:58 mzar: email regarding openvpn and scp issues? 18:34:49 hmmm so after playing with the installer it appears for FDE with keyfile on a usb you got to drop into the shell 18:35:13 and also manually partition 18:36:43 orrr I believe instead could set the keyfile after install 18:36:49 that would likely be the better option 18:44:36 I am going to just do the latter, although I would be curious to do an install without bsdinstall to learn more indepth the freebsd system, anyone got any guides/articles on bsdinstall-less install? 18:44:44 arch linux style :P 20:36:53 mzar: I updated one client from FreeBSD 14.1 to FreeBSD 142 - problem seems to have gone away. 20:52:38 mzar: I'll work on upgrading the other hosts soon. It would be good to know WHY this upgrade fixes the issue. 20:58:11 do we have "gretap" (Ethernet encap over GRE) in FreeBSD? 21:03:48 looks like no, i guess vxlan(4) is the alternative 22:56:59 I'm trying to secure one internet exposed jail with pf and found when I check a port using: telnet it shows the message "Connected to ....Escape character is..", how can I block that? 22:57:47 It doesn't matter which port I point, even non existent ports, it always returns that connection message. 22:58:30 I have a block in rule and only allow access (tcp and udp) to two specific ports.