11:41:29 I'm sure I've seen dcmds that consume output from others. 12:38:38 if nothing else, i would think we could add something to do that 13:08:11 It looks fairly straightforward to use `mdb_eval()` - xhci.c has an example of the kind of thing. 14:00:32 yeah.. 14:00:54 if i can ever get some free time, i'm going to try to do something.. if only so I don't have to remember it :) 14:01:33 we had a dump where (among other things) txg syncing was taking a long time as it was waiting to acquire a rrwlock as writer 14:03:23 and was able to track it down to a zfs diff operation holding things up 14:03:59 (and was reminded about the general unfortunate lack of backpressure in the zio pipeline) 14:27:49 congrats to the folks from oxide: https://oxide.computer/blog/our-100m-series-b 14:31:24 jbk: so much for "Hence, we do not starve writers"... 14:31:58 (found at the end of the big theory statement in rrwlock.c) 14:32:31 i guess it depends if holding up a txg sync for 4+ minutes is 'starving' or not :) 14:33:21 we've had a lot of issues that ultimately come back to 'the zio pipeline doesn't do backpressure' 14:34:34 though in this case I think it's zfs diff is going full throttle and just holding the lock for a long time 14:35:57 yeah, the assumptions about forward progress all assume relatively short hold times on read locks. 14:37:05 in other cases, you can basically create zio_t's and stuff them into the zio pipeline until it's _completely_ exhausted all of your kernel memory (even if you have hundreds of GB of ram) 14:37:26 fixing that unfortunately is a big lift 17:57:29 Seems like Oxide will 'premiere' some/new - Speaking Loudly in the Datacenter - Oxide Tour on youtube. https://youtu.be/iBHos6ZRrIg I hope I don't get kicked for sharing youtube video.. :P 18:00:49 Oh it is about Oxide 100M series B.. https://oxide.computer/blog/our-100m-series-b 18:16:22 I wonder what happens if you shout into an Oxide ¿whatever they call their units. 18:18:51 I believe they're all SSD storage so .. nothing? 18:22:58 Maybe those enlarged fans will change their RPM a little? I was kind a hoping someone would shout again. It was Brendan Gregg originally. 18:46:58 run a big hollywood wind machine at the rack to spin the fans backwards and really upset the service processors? 19:02:26 Long, long ago, early in my career, management decided to cancel support for one of the main department computers. 19:02:38 A few weeks later a lightning strike spun all the platters backwords. 19:03:10 They decided to surplus the system instead of paying to repair it. 19:03:14 I'm still sad about that. 19:23:43 there's an old saying 'the bitterness of poor quality remains long after the sweetness of low price is forgotten'... while not quite the same thing, a similar sentiment probably applies... 20:36:24 at a previous job, one of the managers had that printed out and on the wall in a conference room frequently used by our group... as a big of a passive aggressive prod to our IT director :) 20:36:50 he had a compulsory need to feel like he was always getting a deal 20:37:11 so no matter how good of a price it was, he would _always_ refuse the first quote from a vendor 20:37:23 even if the resulting delay ended up costing more than whatever the savings was 23:37:59 [illumos-gate] 17519 nvme panics on device with no namespaces -- Robert Mustacchi 23:38:37 no names man, no names