16:54:07 [illumos-gate] 15640 name resolution failure on destination hosts with multiple AAAA records -- Andy Fiddaman 17:09:13 i was told to come here for this question: if you cover UNIX, what UNIX version did the NeXTCube run? (NeXTCube as in Steve Jobs's NeXT) 17:16:18 that's not really on-topic here, but the answer is their own custom Unix, named NeXTSTEP, based on BSD & Mach: https://en.wikipedia.org/wiki/NeXTSTEP 17:17:04 it evolved to become MacOS now 17:19:58 alanc: is there a channel that i can get more info on that? 17:21:57 I really don't know 17:31:16 There's a whole thread of history @cosmicwonder you can unravel starting with that NeXTSTEP article. 19:23:40 [illumos-gate] 15687 Remove compiler flags that only mean something to Sun Studio -- Richard Lowe 19:23:41 I'm scared of this one, back it out if there's trouble. 19:23:51 (I know there's a process, but consider "talking to me about it" dealt with) 19:28:54 Building -gate locally, and running a not-yet-pushed SmartOS merge, richlowe. 19:29:25 You said you generated no wsdiff... this pleases me and makes me thing we shouldn't be too worried. The proof, however, IS in the pudding, so I'll let you know. 19:29:37 s/thing/think/^^ 19:51:55 whete is this pudding making feature in illumos? 19:52:01 *where 19:53:14 richlowe - I'll get that pulled into helios in a bit which will give it a spin 19:53:38 we mostly make sausage 19:59:04 we're trying to setup a new system and it fails to boot SmartOS because services timeout when they execute devfsadm 19:59:09 Booting it with milesystone=none gets us a shell and it looks like it might be related to ahci timeouts: https://frupic.frubar.net/shots/43465.png 19:59:22 There seem to be three ahci controllers (1x pci1b21,1062 ASMedia + 2x pci197b,0 JMicron) about which I haven't found out much yet 19:59:43 Any suggestions for debugging this further? If it might be just one bad disk, isolating that would be cool. If it's related to a controller that needs a workaround I can try to dig in and test things 20:09:59 boot the kernel debugger and load one module at a time maybe? 20:15:42 There are at least two loops that drv_usecwait() in the ahci_software_reset(). 20:16:06 If you have a known-good disk, boot this new system with only the disk (and the USB or ISO) to see what happens. 20:16:27 sjorge: sadly, it only makes blood pudding, not sweet dessert pudding 20:17:38 @richlowe --> both my local -gate build (on OmniOS '046, both -nd and -DEBUG) plus the illumos part of smartos-live (non-DEBUG only, but can try later on DEBUG) build. 20:23:29 @alanc & @sjorge ==> My dogs both perked up also when I said pudding, FWIW. 20:24:08 (Normally one dog is with W and one with me but W was down here so we had both dogs, and I was literally quoting proof-in-the-pudding about the Rich push to her...) 20:26:49 Your dogs would probably enjoy a good blood pudding though 20:35:38 wiedi: I think you can blacklist ahci from attaching to things? 20:35:41 but I'm not certain anymore 20:35:56 can I do that only for one of the controllers? 20:36:35 I thought so, but now I'm not sure. 20:37:18 it might be I was thinking of the sata blacklist, but I remember when ahci was new and spotty 20:37:23 and I could swear we had a way to make it go away and see what happened. 20:37:44 wiedi: My general approach when that happens is if this is rerpdoucible, when I hit the state where it's been going on for a non-trivial time is to inject an NMI and look at the system in kmdb. 20:38:32 maybe we just told people to turn it off in the bios, back when that was a thing. 20:38:36 based on the manual page being awful, but saying that 20:39:40 https://frupic.frubar.net/shots/43477.png sadly looks like it's not isolated to one controller or disk 20:44:40 freebsd seems to have a quirks thing for jmicron: https://github.com/freebsd/freebsd-src/blob/fe590ffe40f49fe09d8275fbf29f0d46c5b99dc7/sys/dev/ahci/ahci_pci.c#L427 20:46:13 @richlowe -DEBUG SmartOS passed through its illumos portion with your Makefile fixes w/o issue as well. 20:46:38 And yes @alanc they probably would (in spite of their respective sweet-tempered, but distinct, natures.) 20:47:08 wiedi: I have a disturbing workaround cribbed from linux for a jmicron controller in one of my machines. 20:47:37 sounds promising, I'll take it :D 20:48:25 the freebsd quirks thing seems to be "ew, don't touch it" rather than anything functional. 20:49:21 they have a few more further up (AHCI_Q_NOFORCE | AHCI_Q_1CH) and throughout the file, not sure yet what they all do yet 20:50:07 ah, I already shared it -- https://www.illumos.org/issues/1648 20:50:08 → BUG 1648: JMicron JMB363: switch from RAID to AHCI mode (New) 20:50:21 that might not be your issue. 20:51:13 if your machine catches fire from it don't blame me :-) 20:51:49 hm yep sounds a bit different, here it already shows up as ahci 20:52:25 probably not the same underlying issue. 21:25:34 is it an actual bad disk? 21:26:06 the sd default timeouts + retries are pretty ridiculous for anything made in this millennium 21:26:23 (IIRC, it will try for around 20 minutes per disk before giving up) 21:29:55 now it seems unlikely to be a bad disk since the timeouts happen with both jmicron controllers and on different ports 21:32:34 https://www.illumos.org/issues/14693 seems similar 21:32:35 → BUG 14693: JMicron JMB585 AHCI SATA controller: System panics when trying to write data on pool (New) 21:34:40 that might also be worth lodging an issue 21:35:34 basically sd will keep retrying for so long that it can trip the deadman timer 21:35:42 or it can 21:41:18 (this can be annoying for problematic disks since it'd probably be better to try to just fault the disk if we can) 21:43:10 yep, I've had this in the past where pulling a disk made things better so this was a first thought. Unfortunately this time it looks like it won't be that easy ;) 21:43:33 there's some other fun things i'm trying to fix 21:44:16 like if you resilver a disk in a large pool (in terms of devices)... if one of the top level vdevs is slower, it can cause some 'fun' pathologies 21:45:15 (the scrub prefetch can end up causing an absolutely massive backlog of I/O on a disk) 21:45:31 that itself can trigger the deadman timer due to some other pathologies 22:48:13 wiedi: are you able to run 'mdb -k' or are things far too bad for that? 23:29:14 @richlowe very few diffs from wsdiff and NONE of them are your fault (some/most of them are OUR fault... looking at you libzdoor!!!!). I'm pushing this merge of fenix illumos#15687 into illumos-joyent. 23:29:15 BUG 15687: Remove compiler flags that only mean something to Sun Studio (Closed) 23:29:15 ↳ https://www.illumos.org/issues/15687 | https://code.illumos.org/c/illumos-gate/+/2876