ll

whoops wrong widnow

But I do have a question I'm seeing an NTP issue between a routed network (RAN_Admin) and the headnode using NTP. Wondering if I missed something

joysetup keeps failing

nodes can ping each other fine

netboot works fine

I disabled ipfilter on the headnode but ntp refuses

compute node PI 20230727T000733Z and 20230824T000638Z

nvmd

looks like bios clock off by 6 minutes

nope. Not it

IIRC, joysetup creates a log file in /tmp

on the CN

that might be worth looking at if you haven't yet

(ISTR a few times having to add some extra debugging to it, but my memory is hazy now)

I'm tailing it one sec I'll send some output

10:18 < jbk> IIRC, joysetup creates a log file in /

whoops

[2023-08-29T15:19:49Z] ./joysetup.sh:244: check_ntp(): ntp_wait_seconds=90

[2023-08-29T15:19:49Z] ./joysetup.sh:246: check_ntp(): [[ 90 -gt 600 ]]

[2023-08-29T15:19:49Z] ./joysetup.sh:233: check_ntp(): true

[[2023-08-29T15:19:49Z] ./joysetup.sh:234: check_ntp(): ntpdate -b 192.168.35.254

[2023-08-29T15:19:57Z] ./joysetup.sh:234: check_ntp(): output='29 Aug 15:19:57 ntpdate[4842]: no server suitable for synchronization found'

[2023-08-29T15:19:57Z] ./joysetup.sh:235: check_ntp(): retval=1

[2023-08-29T15:19:57Z] ./joysetup.sh:236: check_ntp(): (( retval == 0 ))

I ran a snoop

I'm thinking hwclock issue on the headnode. Log says "TIME_ERROR" clock unsynchronized on the headnode

29 Aug 15:18:31 ntpd[31385]: Listening on routing socket on fd #54 for interface updates

29 Aug 15:18:31 ntpd[31385]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

29 Aug 15:18:31 ntpd[31385]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

29 Aug 15:18:33 ntpd[31385]: Soliciting pool server 142.147.88.111

29 Aug 15:18:33 ntpd[31385]: Soliciting pool server 217.180.209.214

29 Aug 15:18:34 ntpd[31385]: Soliciting pool server 45.55.58.103

What is the equivilant `hwclock` command on smartOS?

like on linux

seems like only "date" is available

so `ntpdate -b 0.pool.smartos.ntp.org` returns "no server suitable". digging into this a bit further.

29 Aug 15:23:40 ntpd[31385]: Soliciting pool server 162.159.200.123

29 Aug 15:25:34 ntpd[31385]: receive: Unexpected origin timestamp 0xe8988c6e.9602e113 does not match aorg 0000000000.00000000 from server⊙1121 xmt 0xe8988c6e.96501509

I wonder if this has anything to do with the `smartos.ntp.org` vanity name

29 Aug 15:38:16 ntpdate[32564]: ntpdate 4.2.8p15⊙1 Thu Dec 15 04:20:11 UTC 2022 (1)

Guess I'll update the headnode PI and see if it gets resolved by a newer version of ntpdate

jbk: thank you for helping

Wonder if I should report a smartOS bug in PI:

offline 15:09:34 svc:/system/fm/smtp-notify:default

ignore that^ PI joyent_20221215T000744Z

hmmm... last time we did any serious updates of NTP was late 2021. I wonder if smartos.ntp.org has an issue?

That was my original thoughts

4.2.8p15⊙1

Yeah... updated in 2021.

Hmmm

Is pretty close to 4.2.8p7.1 as mentioned in the suse bug report I referenced earlier

But of course, I updated the headnode image, rebooted, and it can't find the pi. So now to fix the USB key or determine if I set it up with piadm

Guess I'll have some manatee work to do too lol

do you happen to have the loader `boot /os/X` command handy so I can tell this thing to boot from the previous image?

0.smartos.pool.ntp.org that's the correct host.

We still have a problem, however.

nevermind I'll check another headnode

Yeah I hadnt even noticed this until I tried setting up new compute nodes. What problem are you seeing?

29 Aug 15:59:30 ntpdate[1805]: no server suitable for synchronization found

but I see traffic. It's like our NTP doesn't like the answers for some bizarre reason.

reference clock is 00:00:00\

NTP: ----- Network Time Protocol -----

NTP:

NTP: Leap = 0x3 (alarm condition (clock unsynchronized))

NTP: Version = 4

NTP: Mode = 3 (client)

NTP: Stratum = 0 (unspecified)

NTP: Poll = 3

NTP: Precision = -6 seconds

NTP: Synchronizing distance = 0x0001.0000 (1.000000)

NTP: Synchronizing dispersion = 0x0001.0000 (1.000000)

NTP: Reference clock =

NTP: Reference time = 0x00000000.00000000 ()

NTP: Originate time = 0x00000000.00000000 ()

NTP: Receive time = 0x00000000.00000000 ()

NTP: Transmit time = 0xe8988a86.754192e1 (2023-08-29 15:17:26.45803)

29 Aug 16:02:31 ntpd[69188]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

29 Aug 16:02:31 ntpd[69188]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized

from /var/log/ntp.log

very strange. We had a bad a NTP host in the ntp.org pool affect all of the AP-North and Southeast regions for samsung back in december.

On the non-triton platform

1 bad pool member wrecked havoc on everything, including SSL certs, causing "untrusted site" errors across all of the api-gateway hosts. It was pretty terrible. I found the source host from the folks over at: teamcymru

Well, let me read back to catch up...

Welcome to the party :)

So barfield --> Two things:

1.) bahamat reminded me to disable the NTP service (so I uttered `svcadm disable -t ntp`)

2.) I tried once after that and failed, but tried again and succeeded. I have packet captures for both the failure and the success.

Shit. I knew that. Then manually update

So when you re-enabled did it come back online?/

worked like a champ. What a rookie mistake.

Considering I was the person who fixed this 9 months ago LMAO

What through me off was another user I saw on the mailing list having NTP issues as well. Made me go down a rabbit-hole that was more complex than needed

Thank you guys.

however, let me back track a bit, after re-enabling ntp service, I'm still getting "clock unsynchronized" kernel messages in the logs.

Yeah, so IIRC, what's happening is that because ntp is over UDP, if you have the ntp service online then it's already bound listening on port 123.

So then when ntpdate runs, it sends the request and fails to bind to port 123 (because ntpd is already there). Then ntpd gets all the replies and ntpdate thinks it got no response.

Yeah I totally knew that. Clearly haven't had enough caffeine today.

So if you're doing it by hand you need to make sure to disable ntp first.

But are the kernel error logs normal? I can't say I have spent a lot of time monitoring them in the past

And the -b is important with ntpdate. That says "update the clock no matter how far off it is"

I ran `svcadm disable ntp` then ran `ntpdate -b 0.smartos.pool.ntp.org` then ran `svcadm enable ntp`

The default is that it only adjusts if it's within like 2m or 5m or something.

Aha

Thank you for clarifying

It all makes sense now.

Because it assumes that the clock is mostly right, and that a response that's too far off must be malicious.

You guys are the best.

And -b says to ignore all that and just take whatever you get.

Probably not a bad assumption. I thought that my IPS may be dropping the traffic as well. Because we drop DDOS on port 123 consistently.

reflection attacks happen all day long against our networks. Now if I can just get my headnode to boot :)

Or in other words "I as an operator know this clock is very wrong, so no matter how bad someone else's clock is, it's better than us"

Again, sorry for the rabbit hole. I should've been able to fix this quickly. ...sigh...

Another indication that something was off, was that ntp had no logs since december of 2022.

Now I see it logging consistently again.

danmcd: bahamat: jkb: thank you all for assisting with this jr admin problem lol

On another note, this is the first zone that I enabled rack-aware networking in. I've some hurdles to work through. so it played into the complexity as well. I found a way to get past the old dell/intel pxe boot issues we used to see, when a node didn't have a USB key to load ipxe

I didn't want 1000 USB key's like JPC/SPC had.

If you guys are interested in how I finally got around it I'll send you a write up

Well, I always recommend booting the first time with a USB, then you can switch to pool booting.

That takes a while though if you're preflighting 100+ nodes

I never fully trust hardware pxe. And if there are issues with it, we can't fix them.

Very true

Depends on your HW>

The answer to "my hardware pxe isn't working with Triton" is always "well that's what the USB is for".

Does piadm support compute nodes yet?

Yes

My HN boots from disk. My 3 supermicro CNs boot three different ways: BIOS PXE, EFI Netboot, and iPXE-from-disk

barfield: piadm doesn't per se.

piadm can, for disk-booting-to-iPXE only.

Oh hell

Thats genius

Hang on...

Yeah, it manages just the boot bits, but the PI still comes from Triton.

I've actually had some issues with these OCP blades booting from the NVME's on board

Also why the USB exists :-D

So I'm still letting the NIC PXE boot first because of that. I leave a USB key in the headnode for emergencies.

See "TRITON COMPUTE NODES and iPXE" in the piadm(8) man page.

I read that a while back. I'll read it again.

Also, it doesn't need to be a USB. You can use SD/CF, etc. cards.

The old "compute node wont boot" from support tickets because of a bad usb drive is what I was trying to avoid.

Yeah, the SPC datacenters had issues with heat management that caused an excessive number of USBs to fail.

Regardless the USB booting is still awesome if you ask me.

So did JPC not see the same issues as often?

I came in after JPC was EOL'ed

But I dont remember it happening as often in JAC

Yeah, JPC had like one, maybe two USBs fail in the time I was there, which was like 6 years.

I don't know how many there were before that, but my understanding is that it was not many.

We also had very high end and low capacity drives.

I think the drives we had were physically only 4G.

You can't even find those these days.

Yeah, I noticed 64-128GB drives at bestbuy the other day for like 10 bucks.

And the higher the capacity in the same space, means you've got more electricity flowing through smaller circuits and that increases heat.

so last question

My HN cannot find the latest PI that I assigned

I am stuck in the loader menu

I use these: amzn.to/47VJwQP

I copied and pasted the var's from another headnode for bootfile bootarchive etc

but its still saying can't find unix, etc, when I type boot

WTH am I doing wrong now

I use the tiny metal kingston 16GB drives and I have never had one go bad.

And they're not even in stock anymore. I'm glad I bought a 5 pack when I could.

I'll find the model but I do not know if you can still get them.

I think I bought like 50. a couple of years ago lol. I'll ship them to you if you want them. I intend to go usb-free all the way

copy/pasting from another node is not recommended...

There should be an option to boot the old PI, unless you broke it buy hand.

I didn't break it by hand. But in EFI mode the loader menu doesn't come up

It just tries to boot, fails, and drops to prompt.

I guess I'll switch it bios, fix it, and switch back

Too bad this damn intel VMD cannot just be removed from the firmware.

danmcd: do you know if there is any illumos/smartos work going on for Intel VMD support?

Not that I care about using it

That's probably a better question for #illumos

I figured

I've not heard of any, but that doesn't mean there isn't.

okay well I got the loader menu up in legacy mode and chose option 3 to rollback. Seems to be working. Thanks for everything.

well back to square 1.

ntp -b 192.168.35.254 (headnode ip in another rack) still fails on the new compute nodes

It's ntpdate -b <address>

sorry

that is what I meant

ntpdate -b

so the next step is to snoop the packets.

got it

Are they sent? Is the headnode receiving them? Is it sending replies? Are the replies received?

yep

one second let me check something

The reference time being zero seems wrong.

yeah, that is what I was referencing earlier in the chat

I suppose it could be a hardware clock issue on the headnode

So check that it matches what's being sent by the headnode.

Maybe the headnode's clock needs to be force set.

compute node snoop

The first was the headnode

Oh, the reference time 0 is the request.

> IP: Destination address = 192.168.35.254, 192.168.35.254

If that's the only thing you're seeing, then you're not getting responses.

oh hell

let me check that out