00:24:31 <LXGHTNXNG> richlowe: it's insufferable
00:25:27 <LXGHTNXNG> one thing I want to do, if I fork a unix, is engage in convergent evolution
01:14:49 <gitomat> [illumos-gate] 15152 ctfdump member output could be easider to decode -- Robert Mustacchi <rm⊙fo>
02:24:29 <LXGHTNXNG> Hello, how do I get svc.startd to reap a zombie child? (serious)
02:32:43 <LXGHTNXNG> youse with jaws ajar, this is serious. I have four zombie processes walking across the process table in my illumos system. three of them are the sons of weechat, an IRC client which I can script to do this. one of them is the son of svc.startd.
03:16:12 <LXGHTNXNG> kernel core dumped again
03:16:27 <LXGHTNXNG> what is happening here
03:18:15 <LXGHTNXNG> image uuid: 8bd71bc4-50d1-e8d9-fdb5-ae5ca5c5fdc6
03:18:17 <LXGHTNXNG> panic message: pageout_deadman: stuck pushing the same page for 90 seconds (freemem is 0)
03:18:19 <LXGHTNXNG> dump content: kernel pages only
03:18:21 <LXGHTNXNG>   (curproc requested, but a kernel thread panicked)
03:19:51 <LXGHTNXNG> https://chatspeed.net/~lightning/20221109-morepanic.txt
03:20:21 <LXGHTNXNG> and then when my system came back up, it was not happy as I have /root on a separate filesystem which it refused to mount
03:39:42 <LXGHTNXNG> this is very weird...
03:50:09 <LXGHTNXNG> hello chamkila
04:03:37 <jclulow> LXGHTNXNG: https://smartos.topicbox.com/groups/smartos-discuss/T604b18d9e24014b6-M3ace9a7ff9614e84247c1ec8/does-this-compute-node-panic-callstack-ring-any-bells
04:05:27 <jclulow> It looks like your system only has 512MB RAM, which is definitely tight.  Figuring out what's blocking pageout from working will be critical to fixing it for you
04:05:43 <jclulow> It seems like you had plenty of swap to decant those anonymous pages into
04:06:44 <jclulow> Oh no it just wrapped strangely on my phone, 2GB
04:07:39 <jclulow> Anyway same advice applies 
04:07:45 <jclulow> pageout should not be stuck 
04:09:10 <LXGHTNXNG> jclulow: right...
04:11:42 <LXGHTNXNG> But yes my system has 2048mo of RAM and a single core CPU. On a sunny day, it's plenty, but if I'm whacking the system, e.g. running cc, it can lock up this way. Should I append my mdb session to the thread, even though I am not on SmartOS?
04:12:15 <jclulow> No I only linked it so I don't have to type all that out again 
04:12:28 <jclulow> If you pastebin what you get back I'll look at it here
04:13:19 <LXGHTNXNG> ok
04:19:59 <LXGHTNXNG> I have appended the result of fffffe0004551c20::findstack -v to the .txt file I linked earlier in the conversation.
04:23:22 <jclulow> Hmm
04:23:26 <jclulow> This is complicated 
04:23:36 <jclulow> Can you upload the dump for me to look at?
04:25:05 <LXGHTNXNG> Right... What format? Plain vmcore, or the compressed-up vmdump thing?
04:25:55 <jclulow> The compressed vmdump is fine
04:25:59 <jclulow> I can unpack it myself 
04:26:25 <jclulow> Put the URL in a DM if you've got any sensitive information in there
04:26:44 <LXGHTNXNG> I don't know that I necessarily do but I'll do that
04:26:59 <jclulow> (dumps can contain all kinds of memory, like passwords or keys or whatever)
04:27:43 <LXGHTNXNG> right
05:30:15 <LXGHTNXNG> i decided to aleatorily inflate some virtual memory tunables
07:42:57 <LXGHTNXNG> I wonder if I can make it deadlock with a large compile
07:43:04 <LXGHTNXNG> which was the usual trigger
08:24:48 <LXGHTNXNG> beep :3
08:27:42 <LXGHTNXNG> ... maxpgio. Diskrpm. Wut?_?
08:40:32 <LXGHTNXNG> wonder if it'll survive the night. recall that it fell over while trying to compile gcc.
08:41:34 <LXGHTNXNG> you won't recall that as until now that was a fact not in evidence.
09:39:24 <andyf> yuripv, richlowe - I added a note to the mount_media.c PR - seems that gcc11 and 12 both accept this which is why the omnios build works. It's definitely a bug though. https://github.com/omniosorg/kayak/pull/236#issuecomment-1308473469. Thanks yuripv 
09:39:51 <yuripv> ah, i tried gcc7, gcc10, and recent clang
14:08:57 <LXGHTNXNG> it deadmanned again.
14:15:04 <andyf> LXGHTNXNG - that's on OmniOS?
14:15:17 <LXGHTNXNG> yes...
14:15:36 <LXGHTNXNG> are they behind the times?
14:15:44 <andyf> No, they're ahead in this case
14:16:02 <LXGHTNXNG> ¿_?
14:16:29 <andyf> Both OmniOS and SmartOS try much harder (than stock illumos-gate) to reclaim pages when memory is low.
14:17:36 <andyf> What you're seeing is due to a low memory condition, so the crash dump that jclulow asked for will hopefully shed some light on what is happening.
14:18:16 <LXGHTNXNG> I sent Josh the crash dump, now the one previous to the most recent one.
14:18:34 <andyf> I used to hit that deadman a lot when doing daft things like bulding 5 copies of nodejs at once
14:18:45 <andyf> but since OmniOS changed the reclaim code, I haven't seen it since.
14:20:13 <LXGHTNXNG> I was building gcc with make -j1 on a system vith 2GB of RAM and with 3GB of swap, regrettably on ZFS. The system fell over with only ca. 700MB of svap in use.
14:21:37 <andyf> It's not generally about how much swap is in use - more about how much there is in total. https://man.omnios.org/man7/swap has more detail about why illumos is different to, say, Linux.
14:22:19 <LXGHTNXNG> good grief.
14:22:38 <andyf> What seems to be happening in your case is that the system can't evict pages out to swap.
14:23:04 <andyf> most likely because whatever needs to write it to swap can't allocate memory to do so..
14:23:30 <andyf> there are guards that are supposed to reserve memory for this, the dump should give some clues about what is happening.
14:25:12 <LXGHTNXNG> hm.
14:36:09 <LXGHTNXNG> there are snakes on this plane
14:36:37 <andyf> One thing worth gathering from the dump is '::stacks -c pageout', in case anyone has time to look.
14:37:05 <LXGHTNXNG> I was linked a Topicbox thread on the matter.
14:37:21 <LXGHTNXNG> It instructed doing exactly that.
14:38:56 <LXGHTNXNG> andyf: do you swap to a zvol? How much RAM do you have that you can exceed building a single copy of Node at once?
14:40:05 <andyf> Sorry, missed that, I just reloaded your text file and see it there now. Looks like a classic case of it trying to eject a page to disk but failing because it needs to allocate memory to do so. There is a tuneable on OmniOS you can set to increase the number of page scanners, that might help.
14:40:26 <andyf> My build VMs have 32G usually, yes they have a swap zvol for paging.
14:41:10 <andyf> If you want to try the tuneable and see if it improves things, put 'set despagescanners=5' in a file like '/etc/system.d/paging'
14:45:08 * LXGHTNXNG puts a pin in that
14:49:23 <LXGHTNXNG> In the meantime, I need to spin up a build VM with a more respectable amount of RAM.
16:51:33 <gitomat> [illumos-gate] 15149 pcieadm help messages should include field list -- Robert Mustacchi <rm⊙fo>
17:40:21 <gitomat> [illumos-gate] 15101 WPTS FSA QueryDirectory test cases with ShortName errors -- Gordon Ross <gwr⊙rc>
17:40:42 <gitomat> [illumos-gate] 15108 SMB option to enable short names -- Gordon Ross <gwr⊙rc>
18:19:42 <yuripv> andyf: also noticed that build/miniroot.pkglist still lists system/library/processor, which was obsoleted (after noticing it, check that all other entries are ok); i can create a pr or you can take care of it yourself? :)
18:23:36 <yuripv> (err, that should read data/miniroot.pkglist)
18:24:35 <andyf> If you have time, please go for it, thanks.
18:24:45 <andyf> Otherwise I can do a PR later tonight
21:47:43 <LXGHTNXNG> andyf: would it help if I moved swap to a partition or slice? This would require extended downtime, but as I said, my clients are tolerant of that and none of them are paying me.
21:48:16 <LXGHTNXNG> (sorry, I'm being rude aren't I. I should shut up.)
22:51:05 <LXGHTNXNG> what does "rebooting.." - twice - mean in a serial console? I'm on omnios '43 and the machine appears stunned.
22:57:52 <jclulow> Is it a physical machine or a QEMU guest
22:58:26 <LXGHTNXNG> QEMU guest.
22:58:47 <LXGHTNXNG> (this isn't directly related to the deadman issue fwiw)
22:59:22 <jclulow> https://www.illumos.org/issues/14526
22:59:23 <fenix> → BUG 14526: illumos guest hangs on reboot under QEMU 6.0.0 (New)
23:03:41 <LXGHTNXNG> right, but I'm wondering why it would progress to reboot in the first place; there's nothing suggesting that the machine has cause to reboot at this stage
23:11:34 <jclulow> Dunno -- something must have asked it to
23:12:21 <LXGHTNXNG> is it some kind of panic thing maybe
23:12:39 <LXGHTNXNG> maybe I am mistaken in installing bloody\
23:13:33 <jclulow> If it panicked I would expect to see panic messages and other junk on the console before rebooting
23:15:50 <LXGHTNXNG> I moved to using uefi rather than bios, and I was able to capture a panic message on the serial console
23:16:09 <LXGHTNXNG> shall I reboot into kmdb mode?
23:16:46 <LXGHTNXNG> https://chatspeed.net/~lightning/20221109-okok.txt (named after the words I said after it did that)
23:16:58 <LXGHTNXNG> (I probably shouldn't be using the 'bloody' media...)
23:19:45 <LXGHTNXNG> oh no, it can't even do kmdb?
23:20:14 <LXGHTNXNG> just, œ
23:28:18 <LXGHTNXNG> hm, I guess I should pull down a recent OI and see what happens
23:41:03 <LXGHTNXNG> is it unable to use virtio main hard drives?
23:44:46 <jclulow> I really don't know what's going on there sorry
23:44:50 <jclulow> it seems pretty far off the rails
23:45:58 <LXGHTNXNG> yeah...
23:46:06 <LXGHTNXNG> now I have to clean my keyboard. sec
23:48:50 <LXGHTNXNG> I did manage to pull a stack trace out of Omnitribblix. Same file.
23:53:57 <LXGHTNXNG> it seems successfully booted with a SATA hard disk and an SCSI CD. This suggests, to my weary eyes, something about virtio disks being wonky?