02:44:25 jclulow: Regarding what veg brought up a few days ago according to the channel logs, I am of the opinion that illumos and the broader OpenZFS community do not do enough networking at things like Illumos Day (when was the last one?), the OpenZFS developer summit, etcetera. Some networking would probably help, especially since there were some patches done specifically for the sake of improving compatibility 02:44:31 with illumos a few months back. 02:44:33 https://github.com/openzfs/zfs/commit/0e4c830bc19766e860e760e10e0d59250f12cced https://github.com/openzfs/zfs/commit/cf66e7e594fc7063db8050f2b7c718ae3f94641b 02:44:55 We don't really hold regular events 02:45:16 And I'm not planning to go to any in person conferences any time soon 02:46:23 jclulow: Also, were any of my follow-up emails explaining my analysis of that thing that I suspect people here would rather I say less about helpful? 02:46:40 I'm still reading through them, I'll get you a response soon 02:46:49 Okay. 02:47:19 wrt. to the ZFS summit I was somewhat surprised it was in person this year -- I would have tried to attend if it were online 02:47:31 But I've also been astoundingly busy at work 02:48:31 I haven't been intentionally skipping the monthly call, but I also don't feel like I'm in a position to offer much guidance there because we're so far apart at this point 02:48:57 Regarding the monthly call, I keep forgetting due to being busy too. I need to add it to my schedule. 02:50:59 As for getting closer, I have made a few efforts to reduce differences on the OpenZFS repository side. Beyond those two patches, there was also this one that removed a bunch of ZoL specific changes from the code: https://github.com/openzfs/zfs/commit/fdc2d303710416868a05084e984fd8f231e948bd 02:51:23 Wait... that might be the wrong patch. One moment. 02:53:36 This is it: https://github.com/openzfs/zfs/commit/ab8d9c1783c4fe51f3ef8518f560fedd2c6fab19 02:53:48 Many ZoLisms were removed by that. 02:55:07 That might make porting more painful due to the historical patches not being fixed, but at some point, it should start making things easier. 02:57:55 Some feedback on this would also be nice: https://github.com/openzfs/zfs/commit/303678350a7253c7bee9d6a3347ee1bcdf9cc177 02:59:54 The naming is bad, but it fixed a potential problem. I was told to name it kmem_scnprintf() and given the lack of disenting opinions that name won. It could still be changed. In both XNU and Linux, a functionally identifical function is named scnprintf(), but I did not want to cause confusion if illumos were to begin using that name to mean something else... 03:00:31 I also did not know that the function was called that at first. I made it independently and then was told that I had reinvented scnprintf(). ^_^;; 03:01:19 I also gave the wrong link for that one. This is the patch introducing it that I meant to link: https://github.com/openzfs/zfs/commit/97143b9d314d54409244f3995576d8cc8c1ebf0a 03:01:35 I have written so many patches over the past 4 months that I am losing track of which is which. 03:08:05 The sheer quantity of patches in ZOL is part of what makes it hard to keep up to be honest 03:08:34 I expect if we're going to try to get up to date we'll need to re-port the whole codebase rather than trying to add patches piecemeal 03:08:59 Which obviously is a lot of work and presents tremendous risk in a critical component 03:10:41 jclulow: Well, there is always the possibility of doing what FreeBSD did so that the development tree and stable releases always support illumos and you can grab the latest driver with little to no effort on your part, after the initial big effort in getting it working in the first place. 03:11:01 Yes I expect we would look at something like that 03:11:14 The big effort is the hardest part haha 03:11:39 Once it is working, we would be running the test suite on illumos on every PR, so support should stay working. 03:12:03 Well, you are not alone. I am sure myself and others would be willing to do some pair programming on it. 03:14:20 The biggest headache with a port, is that you kind of need another filesystem in tree to use to avoid having ZFS be the rootfs, since doing surgery on the rootfs driver is a painful experience when the inevitable mistake occurs that compiles, but utterly breaks it. I hope UFS is still in decent shape. 03:16:07 That could just be my Linux experience speaking since Linux will keep going even if a filesystem has effectively crashed via a NULL pointer dereference or similar issue that would panic another kernel. 03:17:01 That's absolutely terrifying 03:20:40 Well, it makes debugging easier in some cases since the thread where the bad thing happened basically is frozen in time and I can attach a debugger to the system from userland to examine it. 03:21:11 No need to deal with crash dumps, or do special configurations to attach a debugger to the machine over a serial line. 03:21:37 s/thing happened/thing that happened/ 03:22:32 There's literally no way to trust the computer though 03:23:17 Anyway this is obviously a cultural difference haha 03:23:19 Well, it works when it is inside the filesystem driver and the mount points involved are not related to your rootfs where your tools are. It might be less applicable outside of that area. 03:24:39 I heard that it was a result of Linus Torvalds not liking debuggers. Anyway, as long as it works... 03:25:57 Also, the technique was useful for a "hang" that was not really a bug. I had a pool that was so severely damaged by my testing that involved a corruption issue due to bad hardware, that async destroy stopped working. Examining the machine live let me realize that I could set a certain module parameter and it would unstick itself. 03:26:03 Which I got to see right in front of my eyes. 03:26:52 My goal there was to make ZFS even more resilient against bad hardware... it has been a slow and painful process, although I am getting closer. 03:27:56 Note that this is a special kind of bad hardware that just returns errors on flushes in a single massive event because it killed all of the in-flight data and not a "I am consistently malfunctioning" kind of bad hardware. 03:32:43 That case is rare, but it can break the expectation that fsync returning okay means the data is safely on disk because the txg_sync commit's references to the fsynced data got destroyed, yet we have enough information that we can handle that situation in theory with some changes to track what was written since the last successful flush and re-execute it when we know that there was a problem with a flush. At 03:32:49 least that is the idea. 03:52:53 jclulow: This mailing list thread where Linus Torvalds is vehmently against things like kernel debuggers and crash dumps might explain some things: https://lkml.org/lkml/2000/9/6/65 03:53:08 I am familiar with it haha 03:54:45 Linux eventually got them in crippled forms and the end result is that I debug Linux via its own userland since dealing with the headaches from trying to get the stars to align properly from poorly maintained debug tools are more trouble than attaching a mdb-like tool Delphix made to the running kernel to be able to poke around. 03:56:12 In the past, I would attach gdb via QEMU after reproducing the issue, but gdb took many hard to get right commands to do what Delphix's sdb enables me to do with ease. 11:35:45 Seems that RTL8153 (USB3 to Ethernet) is newish chip and that support is not included out of box in illumos.. 11:57:43 nikolam: That is correct. There is currently no support for the device. 15:05:35 Something seems very wrong with /opt/zfs-tests/tests/functional/removal/remove_mirror 15:05:44 https://paste.ec/paste/76W-5Ilv#Uc16iwZyQ3l5ffhJI+kSCuE-95PqSus0dPieuvkWiIM 15:08:48 Has anyone run this recently? 15:16:10 oh, well this explains why it has stopped 15:16:12 zfs_removal_suspend_progress: 1 18:21:38 ohhhh boi, I just was playing around with https://github.com/authzed/spicedb and found out about this new thing called 'xDS' https://www.envoyproxy.io/docs/envoy/latest/api-docs/xds_protocol and now I'm so confused 18:22:52 Anyone using xds? trying to understand what was broken / lacking with load balancing that makes this a requirement for my brain cycles 20:27:47 tsoome_: do you need me to do anything about #pragma ident, or are folks going with it? 20:28:05 andyf: did dan hit that problem too? 20:28:22 the zfs tests are so spotty :* 20:28:24 uh, :( 20:28:36 you mean 15280? 20:28:40 I feel like I need a "learning a weird keyboard" warning sticker 20:29:02 tsoome_: yeah, I didn't know I need to be loudly in favor of it or not :) 20:29:05 (I am) 20:29:21 richlowe - I don't know, but it looks like a previously aborted test run left that variable set in the kernel 20:29:25 I havent seen feedback yet:) 20:29:31 disabling -zcompstrtab by default would make me happy, and things faster 20:29:34 Once I reset it, things ran better 20:29:55 but if I do it with idents around, .comment can get mindbogglingly huge 20:30:01 headers, 20:30:12 but, the good thing is, I should have cleanups done and I'm doing verification builds atm 20:30:37 but for eg oxide have had problems were rust-y things make compressing the compressable string sections the slowest thing we do 20:30:41 haskell also can get into that place 20:30:57 How weird a keyboard? 20:31:00 so you might be enabling a nice perf win by getting rid of the only downside :) 20:32:16 current nightly is running with "cw: remove -fident flag"