07:52:43 <gonzosysadm[m]> any freebsd13 bhyve vms floating around?
07:52:46 <gonzosysadm[m]> sorry, images
08:23:01 <neuroserve> haven't seen one (only 12)
14:37:03 <bahamat> gonzosysadm[m]: I don't understand the FreeBSD release cadence very well, so it's hard for me to pick which releases I should make images of.
18:07:31 <tealirc> Hi there. We are still struggling with Manatee. manatee0 primary and manatee1 sync are ok, but manatee2 async is still in failed state. We created the maantee3 zone with sapiadm command, but this did not help.
18:08:02 <tealirc> Do you have any tips? Can we destroy manatee1, 2 and 3 and run command “sdcadm post-setup ha-manatee -s SERVER1_UUID -s SERVER2_UUID” again?
18:08:23 <tealirc> Is this possible?
18:43:01 <bahamat> I need more information than just "did not work". What does the sitter on the new node report in the log?
19:07:24 <tealirc> bahamat: The logs are the same in the new manatee3 zone. Lots of "zfs recv" commands but nothing happens. The dataset does not exist. https://pastebin.com/g8KszyUa
19:24:05 <bahamat> that log output doesn't show anything about progress, and that's what I need to see.
19:30:52 <tealirc> Sorry for my ignorance, but where can I find the detailed information? Headnode/manatee0/sitter, etc.
19:32:28 <bahamat> In the peer that's attempting to rebuild
19:32:31 <bahamat> the manatee-sitter log
19:34:00 <bahamat> Like yesterday the log you showed me had "body: {"restore": null}" in it. It's not supposed to say null there.
19:34:16 <bahamat> It's supposed to say the total size and the size transferred so far
19:34:22 <bahamat> But null means something is broken.
19:34:52 <bahamat> I need to see what that's doing on the new peer.
19:35:15 <bahamat> If it's null again, then you need to figure out what's getting in the way of your zfs recv.
19:35:56 <bahamat> That's not normal, and nothing about Triton would cause that to happen, so I don't have a solution for you.
19:36:09 <bahamat> Maybe it's networking? Maybe your filesystems are full?
19:36:17 <bahamat> Anyway, you'll need to figure that out.
19:36:55 <bahamat> Once you figure that out, I can guide you through the rest of restoring manatee.
19:57:49 <akole> ZfsClient.postRestoreRequest is the function that initiates the zfs send process?
20:31:50 <akole> ok, might've solved the issue... saw a whole lotta ECONNREFUSED to the manatee0 port 12345 on that function... and sure enough for some reason the backup and snapshotter services were disabled there
20:32:14 <akole> now restore is happening happily
20:54:17 <neuroserve> tealirc : have you checked the diskspace of the manatee instance? I had a similar issue two years ago -> https://bit.ly/3MOaaAJ
21:01:54 <tealirc> neuroserve: We did found issue (more precisely, akole found it). Key services on the manatee0 zone were down (backup and snapshotter).
21:03:20 <neuroserve> cool
21:06:14 <akole> that said, we have had the diskspace issue in the past as well :)
21:07:25 <neuroserve> :)
21:20:24 <tealirc> akole: Then it was a broken hard drive.