Hi.  I am having issues with omnios in AWS.  Running omnios-r151046be on t2.large instance.  I have 2 instances, pretty much exactly the same.  One has been turned off for a while and when started it is getting the clock speed completely wrong.

root@ip-10-130-1-149:/localhome/aslate-local# psrinfo -pv

The physical processor has 2 virtual processors (0-1)

  x86 (GenuineIntel 406F1 family 6 model 79 step 1 clock 112 MHz)

        Intel(r) Xeon(r) CPU E5-2686 v4 @ 2.30GHz

This is causing the machine to have very strange issues

root@ip-10-130-1-149:/localhome/aslate-local# time sleep 10

real 0m0.817s

user 0m0.017s

sys 0m0.046s

The other machine is fine and continues to be after rebooting.  So seems to be an issue after stopping and restarting.

Any ideas?

al_s: we occasionally see AWS giving us a very strange clock, see for example a discussion (and potential resolution) a little while ago

Thank you ptribble I am giving that a go now

Hmm, now the instance does not start at all, seeing this in the system log

OmniOS r151046 Version omnios-r151046-8ab991ea831 64-bit

Copyright (c) 2012-2017 OmniTI Computer Consulting, Inc.

Copyright (c) 2017-2024 OmniOS Community Edition (OmniOSce) Association.

panic[cpu0]/thread=fffffffffbca07a0: Failed to calibrate TSC

Warning - stack not written to the dump buffer

fffffffffbca6620 unix:tsc_calibrate+3a5 ()

fffffffffbca6640 unix:startup_tsc+1b ()

fffffffffbca6650 unix:startup+4a ()

fffffffffbca6690 genunix:main+36 ()

fffffffffbca66a0 unix:_locore_start+88 ()

That's a known problem. The easiest fix is to force use of a different time source. Let me look it up.

It's fixed in r151050, fwiw

echo set pit_is_broken = 1 > /etc/system.d/pit

and reboot

Oh, that's what Peter linked to, sorry

Amother option is to stop and start the EC2 instance, so AWS brings it up on a different physical host which might have a better behaved environment

I have done that at least 5 times already today!  Strangely the instance starts fine when changed to a t3 type.  I have other issues then though, such as "Instance reachability check failed", despite ping and login working.  I have yet to determine if that is an actual problem, but may trigger the instance status check alarm configured to reboot

it on hang.  We will see.

Yep, it does.

Now I am running on a t3 instance, I am worried about illumos.org/issues/16615  Do we know what is likely to provoke the panic?

→ BUG 16615: ena assertion failure in ena_tx_intr_work() (Closed) | code.illumos.org/c/illumos-gate/+/3553

A lot of network traffic - enough so that upwards of 1024 packets are buffered waiting to be transmitted.

There is a hotfix available for r46 at hf.omnios.org/r46/ena-16615.p5p (install with "pkg apply-hot-fix <url>")