Smithx10: pfiles $PID_OF_PROCESS will give you ports, which you can then use as arguments to snoop/tcpdump/whatever.

(Unless your NAT is on the same machine, in which case you'll have to be careful to snoop on the process's zone's netstack)

I noticed that a few machines rebooted unexpectedly but nothing in /var/crash/volatile

Anything else to check to maybe get a hint at what happened?

maybe /var/adm/messages -- if it's a matter of the dump volume being too small, that should get spit out to there during boot

as long as it hasn't been too long

2 days

if you can still see the normal boot output in one of the messages files, it'd be sometime shortly after that

Only time I've seen unexpected machine boots are when I'm doing disk-intensive ops on my HDC (which I now suspect to be a cooling problem with the HDDs).

[<- this guy, I know] Going back to messing around with this machine. I hardware booted a opnsense on it, and it ran into roughly the same limit of routed traffic at ~700Mbit. I've got a replacement on order, and then I'm going to figure out what's up with this hardware

I realize it's old, but this CPU should be able to route 1Gb of traffic intel.com/content/www/us/en/product…-cache-2-40-ghz/specifications.html

jbk: danmcd think this might be something "_" gist.github.com/Smithx10/5a740c3e7022a099fb79d78acac9fa99

This looks very familiar. Hang on...

jinni illumos#13700

Smithx10: ^^^

First fixed in 20220811

well, that one and 14982

I believe postgres was specifically tripping over 14982

Added another, dump..

looks like its the same thing on the other node

jinni illumos#14982

Smithx10: is that a post 14982 PI?

pmooney: got the bug id right?

14982 is a ZFS fix.

14892

Stack shows pollhead_delete issues.

sorry

jinni ilumos#14892

jinni illumos#14892

(should be marked as related to 13700)

one of those "A problem, but not _the_ problem" situations

so... this and the nvme bug the latest platform I should be good to pull in so if / when they fault again

the CN will boot with the fixes*

Smithx10's dump is from 202205xx which is pre-13700

*phew*

Smithx10: don't forget we drop 20221215 this week.

(Which is ALSO a Triton release.)

I didn't immediately recognize 13700 as the pollhead stuff, but the number raised my hackles for sure

joyent_20220505T001410Z says the gist.

Alright Ill just wait for 20221215

Thanks.

Cool, glad these are already known

./back to b33rs

Smithx10: note that 14892 is NOT YET FIXED.

:(

What's pmooney waiting for :P

If you see PG invoking a panic after you jump to a post-13700-fixed PI, we'll need to deep-dive into 14892.

I think 13700 closes the 14892 window BY A LOT.

oh, bother

I'm totally mixed up about what I've fixed, and what I haven't

This bug is actually helping all of the folks using our PG service to learn how to handle failover

Bonus

lol

I should charge extra for that

You know you could do this? psql 'postgres://postgres:$password⊙1928,10.91.197.149,10.91.209.157/postgres?target_session_attrs=primary&sslrootcert=root.crt&sslkey=server.key&sslcert=server.crt&sslmode=verify-ca'

target_session_attr along with multiple peer addrs

so, I have a branch drafted for 14892

but it involved a bunch of changes, which spurred me to write more tests

and that's where it stalled

Smithx10: This is good to knwo :)

toasterson: yeesss

Not every psql library tho has support

yep, sadly

toasterson: tho, github.com/jackc/pgx/blob/v3.6.2/conn.go#L144 this pgx lib does

I think I recall you doing stuff with the go