00:00:00 but with that it will save a few hundred million syscalls 00:00:26 I think it dropped something like 34 seconds on one of my machines 00:02:56 I think tsoome_ was working on getting us upgraded? I forget. I worked on this a bit ago but life stuff has been happening and I didn't follow up. 00:03:14 But I also wonder if newer versions will just add more checks and negate any wins, haha. 00:15:47 is that the bug where sometimes it starts looking at like every file on your build system? 00:16:32 no 00:16:37 I don't think I ran across that one 00:16:49 i've yet to be able to reproduce it consistently 00:17:02 but sometimes if there's a bug, it gets into this thing where it looks at like every header file it can find 00:17:09 or something like that 00:17:20 which of course means the build just sits there for hours 00:17:28 [illumos-gate] 16485 smbios chassis SKU logic is accidentally required -- Robert Mustacchi 00:17:37 I have another patch I want to make, it's a follow up to one pmooney made years ago, but we missed a few spots. That also would reduce a bunch of redundant syscalls IIRC. But I don't think I've seen the one you are talking about. 00:17:39 e.g. if there's a bug in the code you're building 00:52:48 while we're talking about build perf: I'm still tinkering with it but it looks the changes I'm doing for 16497 is saving me 2-3% wall clock time on builds while keeping the load average relatively low. 01:09:25 is that the dmake M2 change? 01:10:56 Yeah. 01:11:24 Fixed the algorithm to more closely resemble what I remember seeing GNU make and the NetBSD make do. 01:12:20 nice.. i had discovered it way back and tried it, but didn't have the time to dig into the problem 01:12:33 after it blew up rather spectacularly 01:13:16 I haven't root-caused the failure - just changed the M2 limiter to interact with the rest of make the way the M1 limiter does. 01:14:14 yeah, when it starts trying to build things in the midst of make clobber it goes off the rails very quickly. 01:27:42 What is "M2"? 01:31:49 never mind, read the issue 01:55:14 i have to wonder how much of the time savings is just from not going nuts under usr/src/man :) 02:34:00 Agnar : we have a build of minio for SmartOS that jperkins worked on recently.. He can fill in further details. https://us-central.manta.mnx.io/pkgsrc/public/test-packages/minio-20240321.tgz for reference 02:34:28 s/jperkins/jperkin/ 06:49:16 also I've been upstreaming the fixes required to golang so hopefully you should be able to build minio without patching on your distribution at some point 10:19:00 hm, 504 Gateway Time-out from src.illumos.org .( 14:23:32 [illumos-gate] 16484 libnvme should take care of unaligned log page sizes -- Robert Mustacchi 15:10:25 [illumos-gate] 16387 CLSET_NODELAYONERR not set if udp transport is used. -- Toomas Soome 18:55:42 @jbk ==> fenix illumos 16502 18:55:43 BUG 16502: panic in zfs:abd_iter_map (New) 18:55:43 ↳ https://www.illumos.org/issues/16502 18:56:09 (I can't believe your small change would've caused this, but since you were the last person in abd-land, thought you should know.) 19:02:21 it's unfortunate there's no dump there 19:03:28 i can't see anything obvious from the stack that'd suggest it could be related 19:04:02 i'd imagine if there was any problems w/ that change, it'd be freeing into the wrong arena... 19:10:50 I _think_ https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/zfs/abd.c#L773 is the line that caused the panic 19:10:52 Yeah... why I can't believe it. 19:11:35 which suggests the abd is bad (would a use after free potentially cause a page fault like that?) 19:11:40 2nd panic is different, but also called by abd_iterate_func(). 19:14:21 given the pc and fault addr in nils' report you should be able to pin it down to what it's trying to read at least 19:14:32 if someone has a dump 19:14:45 also I had no idea nils was around, that's cool 19:15:36 Yeah... look at both dumps. They are called from vdev_indirect_io_done() through a checksum error path and erport. 19:15:43 jbk: agree with you re: the line at issue. 19:15:48 SOrry... both panic stacks. 19:16:02 I think jbk's theory that his work is safe is not necessarily true, iff we got corruption onto the disk that nils is now choking on 19:16:11 but I want to be clear I'm not sayying it's _unsafe_ 19:16:24 the second looks like https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/fs/zfs/abd.c#L838 -- I think the first arg to memcpy is probably bogus 19:17:58 jbk: yeah, that's why I was saying about the fault addr and the dump, given the arguments, you can look at the fault address in terms of its offset from them 19:18:08 like, private->arg_buf there 19:31:59 unfortunately, it doesn't appear there's a dump 19:32:32 It's hard to get, but I may be able to work to get one. 19:35:14 I think from what I can understand easy to get into kmdb and get data, but not a dump. 19:39:41 I think we were able to get a dump. 19:41:58 rmustacc: you mean you can't dump raw memory from your firmware, and produce a valid dump offline? 19:42:00 amateurs ;) 19:42:10 It's on a commodity PC. 19:42:18 I was just joking anyway 19:42:24 All good. 19:46:19 I'll see if we can get it somewhere. 19:46:36 But I'm a bit backed up personally. 19:54:10 yeah, i'd like to take a look if I can... i really tried to go through and verify that either you allocated a new abd_t w/ the metadata flag set appropriately, or if you (more or less) got an abd_t from another abd_t, that the metadata flag would match the source -- and couldn't find any way the flag could 'flip' 19:54:19 so if I did miss something, I want to know what :) 19:55:14 Yeah, just figuring out upload mechanics. I miss manta. 19:57:59 looking further up the stack, it's logging an error after repairing a checksum error encountered while handling i/o to a indirect vdev (which is something that is created when you remove a top-level device from a pool). 19:58:59 if Nils can recall it would likely be good to capture the history of the pool configuration. 19:59:52 was there an accidental "zpool attach" when "zpool add" was meant? 19:59:58 err, other way around 20:00:04 zpool add when zpool attach was meant. 20:01:06 rmustacc: me too.. i wish we had it here for uploading diag stuff (incl dumps) from customers 20:07:44 the thing we use now really can't handle large files well at all 20:07:55 (e.g. crash dumps) 20:13:05 The dump file has been uploaded to the URL Dan provided. 20:27:49 thanks nils 20:45:08 Nice when vmdump.N is < 1GB large. :) 20:45:47 I have a copy in my dump area on kebecloud, and I'll put it somewhere for public download unless you're worried about making it public @NilsN 20:46:16 I wouldn't make it _public_ either way, but maybe letting "us" see if nils is ok 20:47:31 I can put it somewhere for specific values of, "us". :) 20:49:56 I doubt there is anything sensitive in the dump, since it's coming from an ISO-booted system, but it's probably not ideal to make it world readable.  Feel free to give access to whoever ends up digging into the bug. 20:55:48 Given the pool history (removed one of two striped disks) I'd be looking at the vdev_indirect code very close. 20:55:50 closely 20:59:36 https://github.com/openzfs/zfs/pull/14804 might be relevant 21:04:10 Corrupted abd map for sure. 21:04:11 fffffe007cac84b0 abd_copy_to_buf_off_cb+0x30(dc6003a8fb040179, e9f, fffffe007cac8550) 21:04:24 That first parameter is supposed to be a pointer and it sure the &*%^^$ isn't. 21:10:52 So in the dump I have... theres' a call to annotate_ecksum() 21:11:33 It takes a "goodabd" and a "badabd" and the "badabd" is so bad mdb has "no mapping for address" for its pointer. 21:12:18 well at least the name is right then :) 21:13:47 if i'm looking at the right stack in the ticket, 21:14:02 the 'bad' abd should be coming from vdev_indirect_checksum_error 21:14:14 ic->ic_data 21:18:40 I think sommerfeld might be on to something... 21:18:59 https://github.com/illumos/illumos-gate/blob/a87f5a443d0dfc960a9aa1d5c5781c0afe353c41/usr/src/uts/common/fs/zfs/vdev_indirect.c#L1402 21:19:25 so I have a couple thoughts - 1) I wonder if this particular sequence of events has been tested (zpool remove in a non-redundant pool) 21:20:15 2) the failure happens when it's trying to log data around the corruption. if we just turn off the ereport generation can it import the pool w/o crashing? 21:21:46 Happy to give that a try. 21:24:34 Put some data in the ticket. @NilsN I've at most an hour until dinner time here. Anything you want me to do with the dump, please let me know. 21:32:30 i need to dig in more, but IIUC (from what I have looked at so far), for the code in vdev_indirect.c to have identified on of the indirect_child_t's as bad, it would have to actually attempt to checksum the 'bad' ic_data 21:33:21 oh wait, duh 21:33:28 I don't see any easy way to suppress ereports from ZFS, but I think I could do it by setting vdev_checksum_rl to 0. 21:33:30 danmcd: the no mapping should actually be expected 21:33:41 because it's zfs data 21:33:46 Ahh. 21:33:49 That would trigger the ratelimitter. 21:33:59 by default, that's explicitly excluded from dumps 21:35:01 that was in fact part of the motivation for my change -- since we (already) exclude zfs data from dumps normally (IIRC, you can modify the behavior w/ dumpadm to include it) 21:35:13 we were seeing 200+gb dumps 21:35:21 with most of it in abd_ts 21:35:33 well or at least large percentages 21:36:30 since abd's came later, the thought was that since i think only illumos cares about such distinctions, not using the data arena on them was just an oversight 21:36:42 (on abds that weren't marked as metadata) 21:40:53 though.. it's the _good_ abd data that seems like it's the problem 21:41:07 unless the mdb output dan pasted is off w/ args 21:41:25 look at the value to abd_borrow_buf_copy 21:41:51 and then look at where it is in the call to zfs_ereport_post_checksum 21:44:49 though either way, i'm not sure how it could (prior to trying to generate the ereport) determine which abd was bad and which was good without accessing both of them 21:46:18 which suggests (need to look through the code to confirm this) that the contents of both should have been mapped/readable/accessible at the time it figured out which was good and which was bad 21:48:05 (let me note some of that in the ticket) 22:00:49 ok, I think I reproduced something similar in a VM. 22:01:05 with the advantage of not having done it to the root pool. 22:02:04 Go ahead.  Rub it in. 22:02:38 i think both stacks (in the ticket) are related 22:03:01 sorry... 22:03:12 if it's any consolation, the vm is boot looping 22:03:23 not exactly the same traceback 22:03:32 abd_iter_map() is what computes the address prior to invoking the callback (abd_copy_to_buff_off_cb) which in the second stack gets a bogus address (created by abd_iter_map) 22:04:25 since the abd_t is segmented, it's trying to grab linear chunks of it to copy into the new buffer (to concatenate the bits into a linear buffer) 23:01:36 NielsN: I was thinking something more along the lines of commenting out/patching out the call to vdev_indirect_checksum_error() 23:22:08 patching out the call to vdev_indirect_checksum_error in vdev_indirect_repair (overwriting the 5 bytes of the call instruction with 0x90 (amd64 NOP)) lets me import the pool.