09:40:07 nomad - if you get a moment, could you try the updated `dump_spares`? 09:42:01 There is also a hotfix for you from danmcd_ at https://hf.omnios.org/r50/mpt_sas.p5p 09:47:55 (that one is signed :p ) 15:16:34 andyf: : || lvd@chrufs ~ [520] ; ./dump_spares 15:16:34 Segmentation Fault (core dumped) 15:18:38 That's a shame! 15:19:47 Can you share the core? 15:20:22 as long as it doesn't contain any of the data in the zpool, sure. 15:20:49 That host has PII on it. 15:21:11 It should not, but the output of `pstack ` is probably enough for me to track this down 15:21:41 : || lvd@chrufs ~ [523] ; pstack core 15:21:41 core 'core' of 1345: ./dump_spares 15:21:41 0000000000401528 sort_spares () + 76 15:21:41 fffffc7fef0bcba6 qsort_r_wrapper (9ecfb0, 9ecfb8, 4014b2) + 16 15:21:41 fffffc7fef0bd1cc qsort_r (9ecfb0, 2, 8, fffffc7fef0bcb90, 4014b2) + 61c 15:21:42 fffffc7fef0bd647 qsort (9ecfb0, 2, 8, 4014b2) + 27 15:21:44 0000000000401643 dump () + e0 15:21:46 fffffc7feed1bc7d zpool_iter (918550, 401563, 0) + 9d 15:21:48 00000000004016b4 main () + 27 15:21:50 00000000004013f7 _start_crt () + 87 15:21:52 0000000000401358 _start () + 18 15:22:50 I can't test the hotfix on these prod hosts. Unfortunately, my test host is currently in a bit of a disfunctional state while we deal with the 9500 HBA driver question. 15:34:06 oh, the hot fix was for the driver. Right, that I can test. 15:59:17 IIRC, there's some special magic for getting a hot fix recognized in a test be, right? 15:59:22 hey there - I'm noticing that the omnios.org website & pkg updates have been slow/unreachable for the past couple days. Is it me, or is there something else going on? 16:00:21 swinokur - I'm not aware of anything. The US mirror was down for update yesterday, but everything is there now. 16:00:39 nomad - You should just be able to `pkg apply-hot-fix --be-name= ` 16:01:51 hmm... omnios.org (pkg.omnios.org, www.ominos.org) aren't pingable right nwo 16:02:17 I'm not sure if they ever are pingable 16:03:11 thanks andyf 16:04:21 okay, and my traceroute stops here  13 168 ms 167 ms 169 ms rou-fw-rz-rz-gw.ethz.ch [192.33.92.169] 16:05:03 That's close enough - the server is hosted at ETH Zürich 16:10:55 looks like the new driver is attaching and drives are visible. 17:06:07 sorry for the delay -- too many phone calls this morning! If I surf to https://omnios.org/releasenotes - the page just times out - and the pkg updater runs very very slowly (if at all). If I go to the web page on my mobile phone (not connected to wifi, just to cellular) - that page does load fine. Is it possible that my ip/network is being blocked? 17:15:31 swinokur, I just loaded https://omnios.org/releasenotes successfully 17:15:39 there was a very brief delay before it loaded. 17:16:19 thanks! yeah I'm thinking there's something amok goin on over here on the wired network... 17:43:19 okay I have figured it out - my omnios server and windows desktop machine both have jumbo frames (mtu 9000) enabled. When I set the interface on the omnios machine back to MTU = 1500 then pkg started to work again. This used to work fine so I'm wondering if someone changed something on the omnios networking side? 17:46:43 swinokur: likely to be MTU discovery issues. 17:46:56 path mtu discovery, that is 17:47:42 at some point in your network, there is a point where the link MTU changes from 9k to 1500 17:48:30 the router straddling that boundary needs to be able to send ICMP errors that the packet is too big back to the host sending it. 17:48:50 nod - although nothing has changed on my network in quite a while, and pkg not working seems quite recent 17:48:52 or all hosts inside need to be aware of the boundary (perhaps by setting the MTU on routes) 17:51:42 could also be path mtu discovery issues on the other end (you're advertising a TCP MSS based on an MTU of 9k, other side sends jumbograms and they don't get through). 17:53:43 path mtu discovery via ICMP is very often broken because of either busted firewalls not letting it through, or busted routers failing to send the error 17:54:20 its very strange, because omnios.org is the only website that I've found that's behaving this way (and as I mentioned pkg) 17:57:04 hm, you mentioned ICMP - would omnios.org not responding to pings be related to that? 17:57:18 yes 17:57:31 if all ICMP is blocked, path mtu discovery can't work 17:59:28 swinokur, are you using IPv6 perchance? 18:00:15 just for yucks: route add 192.132.2.0/24 -mtu 1400 18:00:29 and reenable jumbograms 18:01:10 not using IPv6 myself (who knows what comcast is doing in the middle) 18:06:48 okay, jumbograms are back on. did the add net command 18:06:59 pkg stops working again 18:07:22 oh -- the route command has a typo! 18:08:19 omnios is 129.132 ;-) 18:08:53 okay with the route command correct, and jumboframes enabled pkg update -v does work 18:10:52 so you're probably better off in the long run setting that MTU value on your default route. (you said comcast, I'm assuming residential cable modem...) 18:12:42 yeah, residential cable modem 18:12:50 yeah that's probably safest 18:15:59 and -mtu 1500 will almost certainly work, barring some odd PPPoE config (unlikely on comcast), but I suggested 1400 for the test for the sake of eliminating that variable 18:21:04 one of the more impactful things I did at google was as part of a project that let jumbo(ish)grams be incrementally deployed in their datacenters. If you have authoritative info on where in your network topology the MTU steps are, route-based constraints on MTU work better than MTU discovery. 18:22:51 yeah that makes sense - I only need jumbo frames inside the house to speed big copies of files around on the 40G & 10G links 18:23:04 unfortunately doesn't look like windows supports route based MTU hahahah 18:27:53 so your best bet is isolating the windows box via some sort of NAT proxy that does MSS clamping. (see ipnat.conf "mssclamp" option if you want to do it with illumos) 18:37:15 it turns out that pfSense has this built in. there's a "MSS" setting in the interfaces section. I put that at 1500 and omnios website now loads again from windows 18:37:45 (pfsense does the header size subtraction for us) 18:42:23 Thanks so much for your help sommerfeld! 18:48:40 glad to be able to help.