16:16:37 <pjustice> Had a panic this morning.
16:16:37 <pjustice> 2024-11-18T15:40:32.932053+00:00 copacabana savecore: [ID 570001 auth.error] reboot after panic: I/O to pool 'zones' appears to be hung.
16:16:40 <pjustice> 2024-11-18T15:40:32+00:00 copacabana savecore: [ID 315959 auth.warning] incomplete dump on dump device
16:16:52 <pjustice> savecore doesn't like the resulting dump file:
16:17:26 <pjustice> hmm, lost that bit, something about tag 1082 and invalid magic number
16:17:35 <pjustice> mdb doesn't like the extracted vmcore
16:18:02 <pjustice> METRICS says the dump was written ~crash time, 15:18 UTC today
16:18:10 <pjustice> compressed file is ~10 GB.
16:21:37 <pjustice> copacabana 9 $ mdb unix.0 vmcore.0
16:21:37 <pjustice> mdb: vmcore.0 is not a kernel core file (bad magic number 0)
16:21:37 <pjustice> mdb: failed to initialize target: No such file or directory
16:23:39 <pjustice> platform joyent_20241017T041739Z
16:26:39 <pjustice> sata ports from C620 chipset
16:30:43 <bahamat> pjustice: You should first look into hardware failure. The "pool appears to be hung" error is almost always hardware related.
16:31:23 <bahamat> If savecore is still running it may still be extracting the dump, but with a hung pool I wouldn't expect there to be a full dump anyway.
16:31:46 <pjustice> It's interesting that it managed to write _some_ with a hung pool.
16:32:11 <pjustice> Or else something else initiated the panic, and the hung pool during dump save concealed the tracks of that.
16:32:45 <pjustice> Since the pool doesn't claim to have any problems and was recently scrubbed, I'm guessing a controller or sata backplane/expander issue.
16:33:14 <pjustice> Nothing useful in the SEL log.
18:21:39 <bahamat> Well the kernel message explicitly says that it detected the pool as hung.