08:05:04 rdma implies infiniband. 08:07:35 but as we can read, rdma over ethernet is also around... . 09:23:38 [illumos-gate] 15202 nvme should recognise Format-in-progress generic status -- Andy Fiddaman 12:28:12 Yeah it's RDMA over Ethernet it seems, well at least what we have at work and was wondering if we had it too 12:28:42 No big, performance is decent untill viona locks up :D 13:33:04 jbk I was trying to boot smartos in oracle cloud, but seems stuck at calibrating TSC ( my guess), how I could debug this further ? I booted in debug I see this https://pastebin.com/Xrfdne19 13:37:22 yeah -- IIRC, the PIT calibration _should_ print out something.. if it's not, then something's happening there.. 13:37:59 what might be useful is seeing how other OSes do their calibration on Oracle cloud (I'm not sure what they're using for their hypervisor) 13:40:10 jbk I'll check what OpenBSD is doing as that is working there and is not supported, I could set kmdb and break on one of the pit calibration routines and start stepping 13:40:20 jbk thanks! 13:42:52 jbk this seems my first guess https://src.illumos.org/source/xref/illumos-gate/usr/src/uts/i86pc/os/tscc_vmware.c?r=04ddedda 13:44:06 since there's AFAIK no way to tell if a system actually has a PIT or not, we try to see if it counts down at all 13:46:23 it tries them in order of preference 13:48:03 jbk I'm reading https://src.illumos.org/source/xref/illumos-gate/usr/src/uts/i86pc/os/tscc_pit.c?r=236cb9a8 comments now to understand this , as I have 0 experience 13:48:30 that the sanity check doesn't even appear to work is somewhat frustrating 13:49:38 you could try an /etc/system module and try the HPET 13:52:30 or I think you can set it from the bootloader 13:53:13 jbk oh, the thing that smartos is not supported on oracle cloud so I just dd the image to the disk of the free instance. if it's possible in the boot loader would be great 13:54:39 aha! i left comments on how to do it (I'm not at home due to the upcoming US holidays) 13:54:51 set tsc_calibration=HPET 13:54:53 then boot 13:56:44 though it still might hang since it still tries to see what value the PIT would produce when using something else to measure the TSC frequency 13:57:53 that unfortunately i think would require a boot module since that'd require /etc/system to explicitly set 'pit_is_broken' to just bypass it completely 13:58:44 jbk thank you very much, I'll try that 13:59:48 i tried to put in as many escape hatches as i could since there isn't really much in the way of 'official' guidance from Intel or AMD on how you're supposed to determine the TSC frequency 14:00:43 we've historically used the PIT, but then some Intel systems (recent NUCs) don't have one 14:00:59 and then some environments such as Hyper-V gen2 vms don't emulate one either 14:01:47 some virtual environments (e.g. hyper-v and VMware) will just tell you the value 14:02:01 but bhyve doesn't (currently) 14:02:34 I think KVM _might_, though there's currently nothing that uses it 14:03:29 I would suggest looking at what source another guest used to determine what you may want to look at. 14:03:46 That will help you see what virtual devices are present and what things to experiment with. 14:04:53 jbk, rmustacc thanks I'll do that 15:01:39 Is it possible to set on kmdb to reboot if it's stuck ? I remember the deadman timer, it will just save me a couple of clicks on the web interface as I need to reboot through there when it hangs 15:03:14 neirac: Am I understanding your question correctly: that sometimes kmdb gets stuck and you'd like it to automatically reboot based on that situation? 15:03:31 Can you describe what happens when it gets stuck? 15:05:35 Basically if you're at the kmdb prompt and can type, then you can instruct kmdb to reboot the system. 15:06:15 However, otherwise kmdb is a single threaded environment. If it is executing an instruction on that CPU, then it will not be able to do anything else until it resumes execution. If an instruction hangs, then there isn't much it can do. 15:06:52 There is no notion of an asynchronous timer in kmdb. It really is just all synchronous and when you single step there's nothing else running unti it returns and hits a breakpoint. 15:30:07 rmustacc thanks, its not possible then, the last thing that's executed is tsc_calibrate_pit+0x4d: call -0x66002 after that kmdb is not responding 15:53:45 hmm.. that's interesting.. that means it called pit_sanity_check() and it succeeded 16:25:33 jbk I'm stepping into freq_tsc_calculate 16:26:56 tsoome_: you're exorcising some weird demons from dladm's code 16:37:02 jbk seems I'm stuck in freq_tsc_calculate 16:47:46 I'm trying to print the value of the esi register in mdb is I think so 17:10:10 anyway with dladm modify-aggr to put 1 of the interfaces into standby or not use it anymore? 19:13:42 Smithx10: Just to clarify you don't mean the LACP active vs. passive logic, but instead basically just saying don't use this interface unless the other one fails? 19:15:06 rmustacc: we have some cabling issues with the people in the DC and I believed 1 of the members of the aggregate were trying to join another aggregation 19:15:11 The cables were cross connected 19:19:41 neirac: yeah, that'll do it 19:40:08 so I was hoping to be able to stop it from poisoning the other aggregation / port channel 19:40:29 because the ToR I believe suspends the entire thing 19:48:09 Smithx10: Remove it from the aggr? 19:48:32 Doesn't that require all the VMs on that box to be downed 19:48:48 or how does that work moving the nic tags over ? 19:49:00 Sorry for my ignorance :( 19:49:56 So the aggr has two devices in it right? 19:50:11 you can have an aggr with only 1 device? 19:50:27 Pretty sure. 19:50:31 ahh hawesome 19:50:34 Ill give that a go 19:50:50 so dladm remove-aggr -l 19:50:56 We got the team to finally run new cables, outside of the nested cabling 19:51:13 but having this trick, will be useful 19:51:41 jbk pretty sure is freq_tsc_pit I just set pit_is_broken to 1 in kmdb and now it the vm boots, I'll try research more what's happening on that one 19:55:19 if there's a way to detect when we're running on oracle cloud (cpuid leaf, etc), we could add something to make it the default in that case 20:04:54 rmustacc: that would be useful too, because I could take the 1 interface out of the aggregation and run lldp on it 20:09:26 jbk I'll check that but now I don't know which is the root password for the latest smartos image, something else is failing at least now boots 20:10:12 Seems it does not detect the nic 20:12:57 Yes, that's the problem checking now svcs logs 20:14:50 According to dmesg this is the cpu that free instances are using cpu1: x86 (chipid 0x0 AuthenticAMD 800F12 family 23 model 1 step 2 clock 2000 MHz) 20:16:02 I'll print dmesg to see if there is something that helps to detect if illumos is running in oracle cloud and use hpet 20:16:23 smbios(8) output is a good place to look too 20:16:48 (that's how we detect similar GCE weirdness, at the moment) 20:21:04 jbk richlowe this is what I have at the moment, I don't see why it does not detect a nic , dladm show-phys is empty https://pastebin.com/qhaH97Lj 20:21:06 neirac: So the two things to do are smbios as richlowe said. If you do an smbios -w /path/to/file and upload that we can inspect that. 20:21:23 rmustacc let me try now 20:21:26 neirac: Run and share the contents of /usr/lib/pci/pcieadm show-devs 20:26:27 rmustacc, jbk, richlowe here it is smbios + pcieadm outputs https://pastebin.com/cWNKnL4h 20:27:03 The chassis asset tag is definitely something we could use. 20:27:37 neirac: Can you run /usr/lib/pci/pcieadm show-cfgspace -d 0/3/0 header0 20:28:08 rmustacc running now 20:29:33 What's likely happened is that they have the newer virtio type there which is why vioif doesn't attach. 20:30:20 rmustacc here it is https://pastebin.com/dL5t1TCe 20:30:39 OK, it's what I just mentioned. 20:30:57 That should be adding the new pcids ? 20:31:06 No, it requires work in the virtio core. 20:31:44 probably similar for the scsi device too i'm guessing 20:31:57 Yes. If one is using the new format, they all are. 20:31:58 (I think that's upstream) 20:31:59 I'm still so confused about the state of virtio 20:32:21 and desperately hoping someone who isn't gets less busy before I have to try to stop being confused. 20:32:54 richlowe: which aspect? 20:33:12 what it would take to use the non-legacy interfaces on pci 20:33:31 what the compatibility situation is with hypervisors and using them at all 20:33:49 some other stuff that's harder to articulate 20:34:47 rmustacc what needs to be done in virtio core? 20:38:04 neirac: if you read the spec, you'll see talk of "legacy" v. not, we only do the legacy 20:42:57 richlowe oh ok 20:46:47 i suspect most of it is how you interact with the device capabilities 20:47:03 my memory is fuzzy on it though, i'd have to go back and re-read things 20:48:51 richlowe I thought it was less work, but the problem is on the device initialization that legacy is not required to do (I'm just reading the spec 1.2 ) or is like creating the driver again? 20:49:27 it shouldn't be that bad, I don't think, but I'm not an expert and that's one of the things that confuses me 20:49:45 and we need to support the legacy interfaces still too 21:01:05 jclulow I remember worked also on virtio stuff 21:01:27 I do the rock! 21:02:18 We could totally add the new interface to the existing framework 21:02:26 It's all done with that in mind already 21:02:39 We would do the negotiation for you, like we do with features 21:03:07 And you, the driver, would continue to ask for queues and interrupt handlers etc probably without noticing 21:03:58 We might add some new hints or whatever for the newly flexible queue stuff, and possibly there are more feature bits available, but yeah, I don't expect it to be a whole new driver 21:08:19 It's possible hypervisors are generally doing the new stuff more now, but when I was doing the new framework I remember trying to test it in GCE and ... I think they were legacy only at the time so I just did legacy 21:11:22 lol..... what is "I do the rock"? lol 21:12:54 It's a Tim Curry song 21:15:17 Ohhhh boi 21:15:41 jbk Would it be too bad to read Asset Tag: OracleCloud.com from smbios and just set pit_is_broken? To let it boot hpet 21:15:41 https://www.youtube.com/watch?v=E1gzt4yU0T4 21:16:08 The 70s hair... so currrrllly 21:26:41 neirac: If pit_sanity_check() succeeded what then fails later? 21:28:59 This song is crazy 21:29:48 I need to grow a strong moustache 21:56:36 jclulow: freq_tsc_pit hangs (or loops forever) 21:57:06 Seems pretty weird! 21:57:09 yeah 21:57:32 either the pit test is flawed in some way, or it's happening to not hit whatever freq_tsc_pit is hitting 21:58:21 though i don't think the pit sanity check is doing anything particularly strange 21:58:36 I am reminded of https://issuetracker.google.com/issues/130531009 22:00:55 the only thing with the smbios is are we able to access the info that early in the boot process? 22:00:59 (with using it) 22:02:02 though i'll have to check (if no one else does) after dinner.. 22:02:59 If we do, we'll want to be very careful not to then depend on SMBIOS tables for startup 22:03:04 not all systems have them etc 22:04:50 Also I think it would be good to make it possible to force the use of the PIT on these systems even if we believe it's broken, to make it easier to try it out later 22:06:29 recall also that we did https://www.illumos.org/issues/14554 but I have not had a chance to do https://www.illumos.org/issues/14556 myself 22:06:30 → BUG 14554: TSC calibration is the PITs (Closed) | https://code.illumos.org/c/illumos-gate/+/2062 22:06:39 (TSC calibration from HPET includes too much error) 23:57:32 Quick question. Is there a way to cause a disk target going away and being rediscovered in qemu or bhyve? The reason I am asking is, on a Customer Linux system with ZFS we had a disk go byebye and sd sent a scsi redicover and readded the disk and the pool then started to have performance problems which where hard to debug. AFAIK in illumos fmd would mark that disk as faulty. I would like to simulate that.