richlowe: it's insufferable

one thing I want to do, if I fork a unix, is engage in convergent evolution

[illumos-gate] 15152 ctfdump member output could be easider to decode -- Robert Mustacchi <rm⊙fo>

Hello, how do I get svc.startd to reap a zombie child? (serious)

youse with jaws ajar, this is serious. I have four zombie processes walking across the process table in my illumos system. three of them are the sons of weechat, an IRC client which I can script to do this. one of them is the son of svc.startd.

kernel core dumped again

what is happening here

image uuid: 8bd71bc4-50d1-e8d9-fdb5-ae5ca5c5fdc6

panic message: pageout_deadman: stuck pushing the same page for 90 seconds (freemem is 0)

dump content: kernel pages only

(curproc requested, but a kernel thread panicked)

and then when my system came back up, it was not happy as I have /root on a separate filesystem which it refused to mount

this is very weird...

hello chamkila

LXGHTNXNG: smartos.topicbox.com/groups/smartos…node-panic-callstack-ring-any-bells

It looks like your system only has 512MB RAM, which is definitely tight. Figuring out what's blocking pageout from working will be critical to fixing it for you

It seems like you had plenty of swap to decant those anonymous pages into

Oh no it just wrapped strangely on my phone, 2GB

Anyway same advice applies

pageout should not be stuck

jclulow: right...

But yes my system has 2048mo of RAM and a single core CPU. On a sunny day, it's plenty, but if I'm whacking the system, e.g. running cc, it can lock up this way. Should I append my mdb session to the thread, even though I am not on SmartOS?

No I only linked it so I don't have to type all that out again

If you pastebin what you get back I'll look at it here

ok

I have appended the result of fffffe0004551c20::findstack -v to the .txt file I linked earlier in the conversation.

Hmm

This is complicated

Can you upload the dump for me to look at?

Right... What format? Plain vmcore, or the compressed-up vmdump thing?

The compressed vmdump is fine

I can unpack it myself

Put the URL in a DM if you've got any sensitive information in there

I don't know that I necessarily do but I'll do that

(dumps can contain all kinds of memory, like passwords or keys or whatever)

right

i decided to aleatorily inflate some virtual memory tunables

I wonder if I can make it deadlock with a large compile

which was the usual trigger

beep :3

... maxpgio. Diskrpm. Wut?_?

wonder if it'll survive the night. recall that it fell over while trying to compile gcc.

you won't recall that as until now that was a fact not in evidence.

yuripv, richlowe - I added a note to the mount_media.c PR - seems that gcc11 and 12 both accept this which is why the omnios build works. It's definitely a bug though. omniosorg/kayak #236#issuecomment-1308473469. Thanks yuripv

ah, i tried gcc7, gcc10, and recent clang

it deadmanned again.

LXGHTNXNG - that's on OmniOS?

yes...

are they behind the times?

No, they're ahead in this case

¿_?

Both OmniOS and SmartOS try much harder (than stock illumos-gate) to reclaim pages when memory is low.

What you're seeing is due to a low memory condition, so the crash dump that jclulow asked for will hopefully shed some light on what is happening.

I sent Josh the crash dump, now the one previous to the most recent one.

I used to hit that deadman a lot when doing daft things like bulding 5 copies of nodejs at once

but since OmniOS changed the reclaim code, I haven't seen it since.

I was building gcc with make -j1 on a system vith 2GB of RAM and with 3GB of swap, regrettably on ZFS. The system fell over with only ca. 700MB of svap in use.

It's not generally about how much swap is in use - more about how much there is in total. man.omnios.org/man7/swap has more detail about why illumos is different to, say, Linux.

good grief.

What seems to be happening in your case is that the system can't evict pages out to swap.

most likely because whatever needs to write it to swap can't allocate memory to do so..

there are guards that are supposed to reserve memory for this, the dump should give some clues about what is happening.

hm.

there are snakes on this plane

One thing worth gathering from the dump is '::stacks -c pageout', in case anyone has time to look.

I was linked a Topicbox thread on the matter.

It instructed doing exactly that.

andyf: do you swap to a zvol? How much RAM do you have that you can exceed building a single copy of Node at once?

Sorry, missed that, I just reloaded your text file and see it there now. Looks like a classic case of it trying to eject a page to disk but failing because it needs to allocate memory to do so. There is a tuneable on OmniOS you can set to increase the number of page scanners, that might help.

My build VMs have 32G usually, yes they have a swap zvol for paging.

If you want to try the tuneable and see if it improves things, put 'set despagescanners=5' in a file like '/etc/system.d/paging'

In the meantime, I need to spin up a build VM with a more respectable amount of RAM.

[illumos-gate] 15149 pcieadm help messages should include field list -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 15101 WPTS FSA QueryDirectory test cases with ShortName errors -- Gordon Ross <gwr⊙rc>

[illumos-gate] 15108 SMB option to enable short names -- Gordon Ross <gwr⊙rc>

andyf: also noticed that build/miniroot.pkglist still lists system/library/processor, which was obsoleted (after noticing it, check that all other entries are ok); i can create a pr or you can take care of it yourself? :)

(err, that should read data/miniroot.pkglist)

If you have time, please go for it, thanks.

Otherwise I can do a PR later tonight

andyf: would it help if I moved swap to a partition or slice? This would require extended downtime, but as I said, my clients are tolerant of that and none of them are paying me.

(sorry, I'm being rude aren't I. I should shut up.)

what does "rebooting.." - twice - mean in a serial console? I'm on omnios '43 and the machine appears stunned.

Is it a physical machine or a QEMU guest

QEMU guest.

(this isn't directly related to the deadman issue fwiw)

→ BUG 14526: illumos guest hangs on reboot under QEMU 6.0.0 (New)

right, but I'm wondering why it would progress to reboot in the first place; there's nothing suggesting that the machine has cause to reboot at this stage

Dunno -- something must have asked it to

is it some kind of panic thing maybe

maybe I am mistaken in installing bloody\

If it panicked I would expect to see panic messages and other junk on the console before rebooting

I moved to using uefi rather than bios, and I was able to capture a panic message on the serial console

shall I reboot into kmdb mode?

chatspeed.net/~lightning/20221109-okok.txt (named after the words I said after it did that)

(I probably shouldn't be using the 'bloody' media...)

oh no, it can't even do kmdb?

just, œ

hm, I guess I should pull down a recent OI and see what happens

is it unable to use virtio main hard drives?

I really don't know what's going on there sorry

it seems pretty far off the rails

yeah...

now I have to clean my keyboard. sec

I did manage to pull a stack trace out of Omnitribblix. Same file.

it seems successfully booted with a SATA hard disk and an SCSI CD. This suggests, to my weary eyes, something about virtio disks being wonky?