09:49:53 @andyf maybe we can look at the man page for viona at FOSDEM? 09:49:59 Or at least get me started. 15:14:56 This is starting to happen on a pretty routine basis: https://www.illumos.org/issues/14730 last night it was due to the switches having a blip for 2 minutes 15:15:00 → BUG 14730: When only site DC reboots, chaos ensues with LDAP and SMB (New) 15:15:15 is there anything that can be done for this? 15:16:13 Like, why does it never attempt to reach the DC again after those 2 minutes? 15:25:54 KungFuJesus: I don't know about the idmapd things but if unix login is not possible it might be that pam settings prevent local logins completely 15:26:23 And also the order of services in nsswitch.conf matters. Make sure files are first 15:27:13 In general all of the user account stuff is very configurable so we will need to know the configs to be able to help 15:31:29 sure, what do you need? 15:31:41 the idmap service is configured in IDMU mode 15:33:01 nssswitch: https://pastebin.com/gspn0pi5 15:34:16 The hanging login bit I'm almost certain is this one: https://www.illumos.org/issues/15024 15:34:20 → BUG 15024: NFS can exhaust pool threads getting RPCSEC_GSS credentials (In Progress) | https://code.illumos.org/c/illumos-gate/+/2402 15:34:34 sort of a separate issue, but...related. I'm really really hoping that one gets merged soon, it's been a big problem for us 15:34:37 KungFuJesus: Well the only place would be in pam.conf where the local login thing could be fixed. As said I have no clue about the idmap issues or modes 15:35:10 local login sometimes works afterward, if a ton of threads aren't being consumed by that bug 15:35:25 and sometimes then, restarting the smb and ldap client services fixes it 15:35:58 Yeah then I won't be able to help 15:36:25 COnfig is working but it' can't launch bash I assume 15:36:38 essentially, yes 15:36:56 the not being able to launch processes is likely the thread pools being exhausted in the kernel 15:37:10 very 15:37:58 but that's only a symptom of the general bug of smb and ldap services not being able to recover when a KDC is unreachable 15:38:52 e.g. LDAP logins, smb accesses and nfs access would still be broken without manually kicking over those services, regardless 15:39:38 I don't understand why the smb and ldap services don't have a retry path 16:31:16 toasterson this needs AD setup with at least 2 AD nodes:D 16:31:55 unix local login? 16:32:13 fixing nss 16:32:59 tsoome_: We do but...for a long time the secondary DC in smbd would have all sorts of bizarro issues 16:33:19 I dunno is gordon fixed that yet or not but I'm very hesistant to put in more than one DC 16:33:47 however in the last particular outage it wouldn't have helped, since the switch may have lost power / rebooted 16:33:54 is it ldap or kerberos issue? 16:33:56 both ports on the link aggregation were down for ~2 minutes 16:33:58 In general one DC should not become lockup reason for the OS 16:34:24 the lockup reason is triggered by the kerberized NFS access 16:34:29 true, but then again, we do not even have decent DC cache... 16:34:52 however, even with that absent, we still have an issue where the SMB _and_ ldap client services never try to talk to the DC again after the timeout 16:35:01 That part of nss probably needs a redo 16:38:43 well, if the issue is about single kdc, then ofc kdc needs backup, with ad, thats of course second dc. 16:39:36 I mean we have that and I think my ldapclient command even specifies the second site. However in this case, it wouldn't have helped. The 2 minute network outage made both DCs unreachable 16:40:02 but if you also have ldap config, then ldapcachemgr is also from years ago and does not cope with ms ad well... 16:44:03 -a defaultServerList="et-cin-dc01.INTERNAL.ETEGENT.COM et-day-dc01.INTERNAL.ETEGENT.COM" 16:44:46 I can add a secondary DC to the smb service properties as well, assuming it doesn't cause issues anymore. Can anyone confirm that those issues went away? 16:46:15 I have no facilities to test ad atm:) 16:49:44 and I have https://code.illumos.org/c/illumos-gate/+/2584 which would be really nice to get tested against ad... 16:49:45 → CODE REVIEW 2584: 15296 libsmb: array subscript is of type 'char' (NEW) | https://www.illumos.org/issues/15296 17:11:18 Hmm, that one seems like it might be _marginally_ ub but not likely to cause issues with most compilers 17:12:24 actually no, the subscripts aren't characters at all. That should not be triggering that warning 17:13:37 Or is it just not happy with the bit shifts that happen to the values? I guess that could be some form of UB depending on if it's a logical or arithmetic shift 17:13:48 interpretation of sign could matter a bit, there 17:18:45 about that warning? array subscription must be int (size does not matter), char is not int, but uint8_t is int. 18:50:28 right but from the MR it looked as though the index was always integer but I didn't see prior versions of it 18:50:47 I've definitely seen issues with signed v unsigned char on big endian platforms 18:51:53 but looking at the comments, it's also been raised that the bitshifts of the values in that array could be problematic if signed and that's why you're using uint8_t 19:39:16 are any of you (hadfl? toasterson? tsoome?) using clang 15? 19:39:58 wacki is currently trying to compile it. Firefox seems to need it 19:40:38 omnios-extra is using it for some packages 19:43:31 yeh there are a few things that hard require it now 19:48:18 to be sure, did you all spot I was asking about 15 specifically? 20:10:11 yes, 15.0.7 currently in pkgsrc 20:12:04 great, but also shit. 20:12:34 richlowe: yep 20:12:38 are you running into the problem where it doesn't build with gcc10? 20:13:12 see DM, but maybe? 20:27:26 [illumos-gate] 15284 tools: remove pragma ident -- Toomas Soome 20:33:11 richlowe I'd use but I have no time to build it:D 20:38:48 tsoome_: well, sounds like andyf has one for you :) 20:38:52 and jperkin 20:38:57 :) 20:39:10 tsoome_: offer to access to the OI build server still stands :) With dedicated user 20:39:24 I can take no credit for the omnios packages 20:39:40 and fritz :) 20:41:11 :) 20:43:08 well. but now there is this issue that https://code.illumos.org/c/illumos-gate/+/2577 and https://code.illumos.org/c/illumos-gate/+/2578 but there are many files..... I have feeling, the webrev would be better option:D 20:43:09 → CODE REVIEW 2577: 15289 cmd: remove pragma ident (NEW) | https://www.illumos.org/issues/15289 20:57:39 KungFuJesus sounds very similar to https://www.illumos.org/issues/13326 20:57:40 → BUG 13326: nscd stops returning results from LDAP until it is restarted (GSSAPI) (New) 21:03:48 tsoome_: being able to sort on that "Size" column would help 21:04:16 tsoome_: or having separate diffs for "just remove it" and "clean up nits" 21:05:49 mm, I guess thats a point where .editorconfig did backfire a bit:D 21:06:06 I have to admit at this point I thought doing that diff wise was easy, but the manual page isn't helping me 21:06:13 I thought there was a negative -G or a --diff-filter thing 21:06:21 so you could say "give me all the diffs that aren't ident" 21:06:34 oh, git diff -I 21:06:46 might let you say "give me diffs that aren't ident", and then git apply them to a scratch branch to show us 21:06:50 without it being way too much work 21:12:50 it doesn't, because of the blank line removal too 21:17:07 tsoome_: I figure these `git diff --numstat -w illumos/master...remotes/gerrit/changes/78/2578/4 | awk '$1 != 0 || $2 > 2'` are the interesting ones 21:17:54 and possibly only those with additions, really, if wsdiff is clean too. 21:18:27 there's like 100 files with additions, and 600 if you include more than expected delitions 21:18:44 (and I think that's across all your branches, because of how they're stacked) 21:19:15 dunno how easy it would be to pull those out to let people only view the surprising ones. 22:28:20 danmcd: let me know if you are interested in working together to get that 14892 stuff over the line 22:28:32 sorry I sorta dropped the ball on it once it came time to write tests 22:28:45 I will let you know. I have to look at what's here, and what I'm seeing in the customer dump. 22:29:11 Expect to hear from me, but maybe not until later this week? Lots of shit flying, this is only some of it. 22:29:24 that's fine, I have plenty on my plate 22:30:00 if it's a burning issue for y'all, it might be worth trying to replicate the postgres reproducer on some other machine 22:30:12 if you get it blowing up in the same manner, then it'd be at least one smoke-tests for the new bits 22:31:23 it's not like I had a whole suite of signalfd consumers lined up when I did the overhaul 22:33:31 AndyF is going to provide me a VM image with his I-think-reliable reproducer. 22:33:55 Also I had a merge conflict with LX stuff in signalfd. Two lines, but they need to move in your new world order. 22:34:07 (OmniOS will have the same problem because of LX.) 22:42:43 which lines, out of curiosity? 22:46:35 Around here: /* Convert k_siginfo into external, datamodel independent, struct. */ 22:46:50 We have a brand-check and a brand-call for signal number conversion. 22:47:13 if you diff signalfd.c between illumos-{joyent,omnios} and -gate you'll see. 22:47:36 It didn't merge easily because of your rewrite (not a shock), but I need to make sure I moved the check-and-conversion prior to USAGE. 22:48:27 So if you diff between the just-now-created illumos-joyent:14892 branch's signalfd.c and your one for -gate, you'll see a similar diff as comparing -{joyent,omnios} to -gate. 22:48:43 * danmcd is kicking off a smartos-live build. 22:50:14 I'm back home and on with building the reproducer now 22:51:47 If you look at https://github.com/TritonDataCenter/illumos-joyent/pull/432 it should drop right into illumos-omnios since I did the merge-heck for the LX changes in signalfd.c 22:52:28 ah, that makes sense 22:55:16 Thank you andyf for reproducer building. It's almost dinnertime US/Eastern, so I'll let the smartos-live reality-check build finish in the bg. 22:58:52 is our 'upstream' for mandoc the openbsd one? 23:05:10 I don't know if there's a portable- like for other things, but yes. 23:05:20 yuripv is the main person to talk to about it though. 23:15:27 jbk: There's a regular release tarballs they put out. 23:15:36 I've done recent syncs there. What's up? 23:54:49 hallo! 23:55:07 yes, i know, we speak english here :) 23:55:34 if you spoke complicated german to me i would be lost anyway 23:56:36 I'm actually finally transfering my dev box to a vm, i really need the machine 23:56:58 had one donated especially for illumos but he's apparently busy :( 23:57:48 and i know the sys v way is to use find and cpio but i'm used to tar so i'm using that 23:59:20 oh, i should use p option with tar though... it's my home dir right now so i don't care