14:17:29 <warden> Hi to all. The first OmniOS server I put in production is rebooting from time to time... I suspect that the cause lies in ipfilter, which I have enabled and configured, but I'm not able to track the issue to the root cause.
14:17:53 <warden> Here is the fmdump output and a basic analysis of the core dump file: https://paste.omnios.org/?eb3d39a5705b95cf#HQ47BknR62LgPPKsa5BJqCVMELdmj1UJ46Y1JVwftz6N
14:18:06 <warden> I'd be very grateful if someone more skilled than me can suggest how to further analyze the issue. Thanks! :)
14:49:40 <rzezeski> warden: running `::status` in mdb would be helpful
14:50:39 <rzezeski> It appears to be a problem with grabbing a mutex, but I see nothing obvious in fr_tcp_age(). Makes me wodner if a stack frame is missing because it does call fr_movequeue(), which does try to enter a mutex.
14:50:59 <rzezeski> Also, wow, didn't know we had some K&R C still floating around.
14:57:29 <warden> Thank you rzezeski, here it is the ::status output: https://paste.omnios.org/?d6088f55f52d9cb7#G5zJtmePLhu91NQpiRgRkTCzXWRqsTdhZ8qKWvMJcAcS
14:57:39 <warden> please note that all these debug commands are arcane for me... threat me like a baby! :)
14:58:46 <warden> s/threat/treat/
15:12:59 <rzezeski> Okay, so a null pointer. Let's look at the tqe argument passed to fr_movequeue(). Run this in mdb: `fffffe599ee636a0::print -t ipftqent_t`
15:13:37 <rzezeski> It also might be nice to just keep adding to the same paste if that's possible so everything is in one spot.
15:21:26 <warden> Thanks. PrivateBin seems not to allow me to modify existing paste, so I merged all text here: https://box.messagecloud.it/index.php/s/2Agx3n367zmi6M8/download/OmniOS_coredump.txt
15:32:28 <rzezeski> Okay, now let's look at the tqe_ifq member. Run this in mdb: `fffffe599ee636a0::print -t ipftqent_t tqe_ifq |::print -t ipftq_t`
15:35:05 <warden> great, I've just updated the text file (also readable inside the browser window at this URL: https://box.messagecloud.it/index.php/s/2Agx3n367zmi6M8 )
16:23:49 <rzezeski> warden: weird, it seems a NULL pointer was passed to mutex_enter(), but the code (albeit very confusing) doesn't seem like it should do that. So I might be making the wrong assumption. We are missing a stack frame because mutex_enter is optimized and doesn't store one. We should be able to look at it by doing something like: `fffffe007d7fcbd0,30/nap`
16:34:29 <warden> rzezeski: I barely understand what you mean, but I've just updated the text file with the last output. Thanks!