rdma implies infiniband.

but as we can read, rdma over ethernet is also around... .

[illumos-gate] 15202 nvme should recognise Format-in-progress generic status -- Andy Fiddaman <illumos⊙fn>

Yeah it's RDMA over Ethernet it seems, well at least what we have at work and was wondering if we had it too

No big, performance is decent untill viona locks up :D

jbk I was trying to boot smartos in oracle cloud, but seems stuck at calibrating TSC ( my guess), how I could debug this further ? I booted in debug I see this pastebin.com/Xrfdne19

yeah -- IIRC, the PIT calibration _should_ print out something.. if it's not, then something's happening there..

what might be useful is seeing how other OSes do their calibration on Oracle cloud (I'm not sure what they're using for their hypervisor)

jbk I'll check what OpenBSD is doing as that is working there and is not supported, I could set kmdb and break on one of the pit calibration routines and start stepping

jbk thanks!

jbk this seems my first guess src.illumos.org/source/xref/illumos…s/i86pc/os/tscc_vmware.c?r=04ddedda

since there's AFAIK no way to tell if a system actually has a PIT or not, we try to see if it counts down at all

it tries them in order of preference

jbk I'm reading src.illumos.org/source/xref/illumos…/uts/i86pc/os/tscc_pit.c?r=236cb9a8 comments now to understand this , as I have 0 experience

that the sanity check doesn't even appear to work is somewhat frustrating

you could try an /etc/system module and try the HPET

or I think you can set it from the bootloader

jbk oh, the thing that smartos is not supported on oracle cloud so I just dd the image to the disk of the free instance. if it's possible in the boot loader would be great

aha! i left comments on how to do it (I'm not at home due to the upcoming US holidays)

set tsc_calibration=HPET

then boot

though it still might hang since it still tries to see what value the PIT would produce when using something else to measure the TSC frequency

that unfortunately i think would require a boot module since that'd require /etc/system to explicitly set 'pit_is_broken' to just bypass it completely

jbk thank you very much, I'll try that

i tried to put in as many escape hatches as i could since there isn't really much in the way of 'official' guidance from Intel or AMD on how you're supposed to determine the TSC frequency

we've historically used the PIT, but then some Intel systems (recent NUCs) don't have one

and then some environments such as Hyper-V gen2 vms don't emulate one either

some virtual environments (e.g. hyper-v and VMware) will just tell you the value

but bhyve doesn't (currently)

I think KVM _might_, though there's currently nothing that uses it

I would suggest looking at what source another guest used to determine what you may want to look at.

That will help you see what virtual devices are present and what things to experiment with.

jbk, rmustacc thanks I'll do that

Is it possible to set on kmdb to reboot if it's stuck ? I remember the deadman timer, it will just save me a couple of clicks on the web interface as I need to reboot through there when it hangs

neirac: Am I understanding your question correctly: that sometimes kmdb gets stuck and you'd like it to automatically reboot based on that situation?

Can you describe what happens when it gets stuck?

Basically if you're at the kmdb prompt and can type, then you can instruct kmdb to reboot the system.

However, otherwise kmdb is a single threaded environment. If it is executing an instruction on that CPU, then it will not be able to do anything else until it resumes execution. If an instruction hangs, then there isn't much it can do.

There is no notion of an asynchronous timer in kmdb. It really is just all synchronous and when you single step there's nothing else running unti it returns and hits a breakpoint.

rmustacc thanks, its not possible then, the last thing that's executed is tsc_calibrate_pit+0x4d: call -0x66002 <freq_tsc_pit> after that kmdb is not responding

hmm.. that's interesting.. that means it called pit_sanity_check() and it succeeded

jbk I'm stepping into freq_tsc_calculate

tsoome_: you're exorcising some weird demons from dladm's code

jbk seems I'm stuck in freq_tsc_calculate

I'm trying to print the value of the esi register in mdb is <esi=D correct?

I think so

anyway with dladm modify-aggr to put 1 of the interfaces into standby or not use it anymore?

Smithx10: Just to clarify you don't mean the LACP active vs. passive logic, but instead basically just saying don't use this interface unless the other one fails?

rmustacc: we have some cabling issues with the people in the DC and I believed 1 of the members of the aggregate were trying to join another aggregation

The cables were cross connected

neirac: yeah, that'll do it

so I was hoping to be able to stop it from poisoning the other aggregation / port channel

because the ToR I believe suspends the entire thing

Smithx10: Remove it from the aggr?

Doesn't that require all the VMs on that box to be downed

or how does that work moving the nic tags over ?

Sorry for my ignorance :(

So the aggr has two devices in it right?

you can have an aggr with only 1 device?

Pretty sure.

ahh hawesome

Ill give that a go

so dladm remove-aggr -l <nic to remove> <aggr name>

We got the team to finally run new cables, outside of the nested cabling

but having this trick, will be useful

jbk pretty sure is freq_tsc_pit I just set pit_is_broken to 1 in kmdb and now it the vm boots, I'll try research more what's happening on that one

if there's a way to detect when we're running on oracle cloud (cpuid leaf, etc), we could add something to make it the default in that case

rmustacc: that would be useful too, because I could take the 1 interface out of the aggregation and run lldp on it

jbk I'll check that but now I don't know which is the root password for the latest smartos image, something else is failing at least now boots

Seems it does not detect the nic

Yes, that's the problem checking now svcs logs

According to dmesg this is the cpu that free instances are using cpu1: x86 (chipid 0x0 AuthenticAMD 800F12 family 23 model 1 step 2 clock 2000 MHz)

I'll print dmesg to see if there is something that helps to detect if illumos is running in oracle cloud and use hpet

smbios(8) output is a good place to look too

(that's how we detect similar GCE weirdness, at the moment)

jbk richlowe this is what I have at the moment, I don't see why it does not detect a nic , dladm show-phys is empty pastebin.com/qhaH97Lj

neirac: So the two things to do are smbios as richlowe said. If you do an smbios -w /path/to/file and upload that we can inspect that.

rmustacc let me try now

neirac: Run and share the contents of /usr/lib/pci/pcieadm show-devs

rmustacc, jbk, richlowe here it is smbios + pcieadm outputs pastebin.com/cWNKnL4h

The chassis asset tag is definitely something we could use.

neirac: Can you run /usr/lib/pci/pcieadm show-cfgspace -d 0/3/0 header0

rmustacc running now

What's likely happened is that they have the newer virtio type there which is why vioif doesn't attach.

rmustacc here it is pastebin.com/dL5t1TCe

OK, it's what I just mentioned.

That should be adding the new pcids ?

No, it requires work in the virtio core.

probably similar for the scsi device too i'm guessing

Yes. If one is using the new format, they all are.

(I think that's upstream)

I'm still so confused about the state of virtio

and desperately hoping someone who isn't gets less busy before I have to try to stop being confused.

richlowe: which aspect?

what it would take to use the non-legacy interfaces on pci

what the compatibility situation is with hypervisors and using them at all

some other stuff that's harder to articulate

rmustacc what needs to be done in virtio core?

neirac: if you read the spec, you'll see talk of "legacy" v. not, we only do the legacy

richlowe oh ok

i suspect most of it is how you interact with the device capabilities

my memory is fuzzy on it though, i'd have to go back and re-read things

richlowe I thought it was less work, but the problem is on the device initialization that legacy is not required to do (I'm just reading the spec 1.2 ) or is like creating the driver again?

it shouldn't be that bad, I don't think, but I'm not an expert and that's one of the things that confuses me

and we need to support the legacy interfaces still too

jclulow I remember worked also on virtio stuff

I do the rock!

We could totally add the new interface to the existing framework

It's all done with that in mind already

We would do the negotiation for you, like we do with features

And you, the driver, would continue to ask for queues and interrupt handlers etc probably without noticing

We might add some new hints or whatever for the newly flexible queue stuff, and possibly there are more feature bits available, but yeah, I don't expect it to be a whole new driver

It's possible hypervisors are generally doing the new stuff more now, but when I was doing the new framework I remember trying to test it in GCE and ... I think they were legacy only at the time so I just did legacy

lol..... what is "I do the rock"? lol

It's a Tim Curry song

Ohhhh boi

jbk Would it be too bad to read Asset Tag: OracleCloud.com from smbios and just set pit_is_broken? To let it boot hpet

The 70s hair... so currrrllly

neirac: If pit_sanity_check() succeeded what then fails later?

This song is crazy

I need to grow a strong moustache

jclulow: freq_tsc_pit hangs (or loops forever)

Seems pretty weird!

yeah

either the pit test is flawed in some way, or it's happening to not hit whatever freq_tsc_pit is hitting

though i don't think the pit sanity check is doing anything particularly strange

I am reminded of issuetracker.google.com/issues/130531009

the only thing with the smbios is are we able to access the info that early in the boot process?

(with using it)

though i'll have to check (if no one else does) after dinner..

If we do, we'll want to be very careful not to then depend on SMBIOS tables for startup

not all systems have them etc

Also I think it would be good to make it possible to force the use of the PIT on these systems even if we believe it's broken, to make it easier to try it out later

recall also that we did illumos.org/issues/14554 but I have not had a chance to do illumos.org/issues/14556 myself

→ BUG 14554: TSC calibration is the PITs (Closed) | code.illumos.org/c/illumos-gate/+/2062

(TSC calibration from HPET includes too much error)

Quick question. Is there a way to cause a disk target going away and being rediscovered in qemu or bhyve? The reason I am asking is, on a Customer Linux system with ZFS we had a disk go byebye and sd sent a scsi redicover and readded the disk and the pool then started to have performance problems which where hard to debug. AFAIK in illumos fmd would mark that disk as faulty. I would like to simulate that.