18:03:30 I have a weird issue in a multiDC deployment...I setup the ufds replication 2 years ago and it was working fine. At somepoint the admin network routes on our dark-fiber got AFU'ed in our IGP and AZ2 couldn't contact AZ1. It seems that all of the sdcperson objects have propogated from AZ1 to AZ2 previously but for some reason the only user that isn't propogating to AZ2 is the poseidon user for manta. 18:03:37 Although all is synced fine in AZ3. Has anyone ever seen this behavios in UFDS? Or could it be that I since I haven't run the manta setup steps in AZ2 that the posiedon user simply isn't going to be replicated yet? I do not remember running anything in AZ3 besides installing the manta zone on the headnode. So I am not sure why it is replicating there. 18:04:34 Side note: /opt/smartdc/ufds/bin/ufds-reconcile-data reports that all data is in sync on all 3 AZ's in the ufds0 zone on each headnode. 18:04:56 But `sdc-useradm search poseidon` only returns results in AZ1 and AZ3 18:05:28 This is `barfield` BTW. For some reason the nickServe isn't letting me update my nick today. 18:05:56 Yeah, there's a bug (maybe fixed now, I don't remember) where the manta install script would add poseidon to the local UFDS instead of to the primary. 18:06:27 If you're only missing it in one DC, then you're going to have problems. 18:06:40 I see, so maybe I did run it in AZ3 and it is pulling from the local primary? 18:06:58 I guess UUID's would identify this 18:07:05 The way UFDS replication (well, *all* ldap replication) works is by sending differential changes. There's no "reconcile everything" feature in LDAP. 18:07:26 right that makes sense 18:07:38 The only way to fix it is by using ldap commands directly in the local UFDS. 18:07:57 the UUID is the same in both AZ1 and AZ3 it just doesn't exist in AZ2 18:08:08 The local UFDS in the replication zone in AZ2? 18:08:20 In the DC where it's missing. 18:08:26 Thanks man! 18:08:29 Thats easy enough. 18:08:41 I was worried I had a bigger issue then that 18:09:02 BTW I found the issue with my rabbitmq zone 18:09:50 Was the external NIC in sdc-napi causing the issues. I had to delete it and unreserve the IP address. Then when I bounced the headnode net-agent updated napi and rabbit came right up 18:09:56 I suspect that someone was aware of the UFDS bug at install time and attempted to correct it by removing the offending users, but then forgot to add the fixed one in to AZ2. 18:10:09 That would make sense 18:10:16 It would've been me but I've slept a lot since then 18:10:26 Drank alot too :) 18:11:06 Is the Linux CN project still active by chance? 18:11:17 As for rabbit, ISTR there being a discussion about the extra IP being an issue. 18:12:05 We're working on a new milestone release for LinuxCN. 18:12:06 I updated all of the SAPI VM UUID's in /instances ... serveral had params.server_uuid set to 2 different missing server ID's. 18:12:35 Things are a bit in flux because of the LXD announcement. We haven't decided what to do about that yet. 18:13:53 there was a 0.1 incus release earlier, I think 18:14:43 Have you guys looked into `harvestor` as a potential pi? 18:14:56 What LXD announcement? I must have missed it 18:15:54 nueroserve: is this on github? 18:26:57 No. I don't know what harvestor is. 18:27:25 Its essentially a minimal container OS designed to back k8s 18:27:56 Then it's unlikely to fit well with Triton 18:28:12 We'd likely have to remove a ton of stuff until it resembles what we need. 18:28:14 I figured as much 18:28:51 I was just curious...seemed like it might make a good candidate 18:29:02 What distro currently backs the linux CN? 18:31:15 Debian 18:31:22 oh nice 18:31:26 my favorite anyway 18:31:32 well for linux that is 19:33:39 https://github.com/lxc/incus 19:33:55 thansk 19:51:36 I've got poseidon recreated in AZ2 but I cannot seem to add the "dclocalconfig" attr so that it is displayed when you run sdc-useradm get UUID 19:52:02 Is this something that should be populated from sapi instead? 19:52:22 replace-attr, add-attr etc completes successfully. but get never shows that aspect 19:55:59 oh wait I may have just figured it out 20:18:29 Nah I didn't. I thought that I could add it with a json manifest to sdc-ufds. 20:18:58 So I am assuming that dclocalconfig on a ufds object comes from the local ufds master and not from the remote 20:19:34 The remote appears to nest all of the localconfigs on a user as a json array [] 20:19:41 *dclocalconfigs