16:37:56 does anyone built oxidecomputer’s omicron? I hit issue on omnios, and hit issue, https://pastebin.com/VHXHnCYr how to resolve the problem? does it support omnios or smartos ? best wishes 16:40:03 please point it out if anyone know the issue, thanks 16:55:11 have you tried doing what the error suggets and running 'tools/ci_download_dendrite_openapi' ? 17:40:50 tozhu: OmniOS is quite similar to our distribution, but with a few differences here and there. 17:42:28 There is some more detail on getting started with a simulated environment here: https://github.com/oxidecomputer/omicron/blob/main/docs/how-to-run-simulated.adoc 17:44:06 I would note two things though: It's a complex consolidation of software that is seeing a lot of churn -- we're actively working on it; and, it's really only targetted to run on Oxide hardware on top of our Helios distribution, in a rack with our specific switches and so on, it's not really a general purpose control plane you can just install on some Dell servers etc. 18:06:49 danmcd: did the i40e stuff ever get integrated? I sort of lost track of that a while ago. 18:11:13 It hasn't because it requires a LOT of regression tests. See https://illumos.topicbox.com/groups/networking/Te8c1f6fce26c8a54-M02e81fae4b931966e92d75bd for what I mean. 18:11:41 The fix should be straightforward, but making sure it doesn't break other promisc corner-cases is a nightmare requiring time and multiple i40e boards. 18:14:40 ah right we needed a range of hw to test 18:15:09 And a range of tests (per the email). 18:22:34 right, so it seems unlikely to get all that arranged 18:37:14 danmcd: is the patch small enough that you could hide it behind a configuartion in system.d like sjorge did for the bhyve viona stuff. Then at least people with the part could opt into the new behavior and provide feedback etc 18:40:14 IIRC it changes the way erm sole HW based mac filters are initialized, so not sure it's easy to toggle 18:40:19 If it is, that could indeed be an option 18:59:51 danmcd: I'm struggling a little to understand exactly what cards are what in that mail chain 19:00:00 Is there a concise list of the parts we need? 19:00:42 It's possible I can buy something, and we have a 40G-capable switch here, etc 19:01:55 Also, if this is the sum total of the change: https://kebe.com/~danmcd/webrevs/13230-newtry/usr/src/uts/common/io/i40e/i40e_gld.c.sdiff.html 19:02:17 I feel like we could parcel that up behind an /etc/system toggle tbh 19:02:43 So I've been working on building zig-0.11.0 on OmniOS (making use of the excellent omnios-extra facilities), and through a few small patches I made it to attempting to build stage3. However, as part of that process, the zig2 binary creates a bogus ELF executable where the header claims it's a 64-bit executable but the interpreter section points to 32-bit ld.so. 19:03:12 As I debugged this more I found the executable is built with lld which is baked into the zig2 executable. Here is how the zig2 executable is called: 19:03:12 https://gist.github.com/rzezeski/56a9d9c1255e47af35af5f544a0bd42d 19:05:05 So, I don't know much about linkers, but I do know the common refrain of "you should be using the system linker". However, zig appears to be taking the strong approach of controlling the linking itself. Right now they rely on lld, but they have already been working on their own linker and the plan is to eject LLVM from zig itself. 19:06:34 So I guess I'm wondering, what are the big reasons for insisting on the system ld(1)? I have notions of why we take this stance, but I'd love to hear specifics. If zig is going this route is it just doomed to never running on illumos? 19:08:07 One thing I need to ask the zig folks: would they allow an escape hatch to farm out to the system ld. That's code I could write. However, I think I've also heard that you typically want to use gcc as a front-end to ld for various reasons. So perhaps even directly invoking system ld(1) is a bit of work IF the zig folks are amenable to such a tactic on illumos. 19:09:15 I'm also wondering, if in the meantime, I can get zig's embedded ld.lld to build a proper executable on illumos. Is that possible? 19:09:59 for building, i suspect changing `-dynamic-linker /usr/lib/ld.so.1` to `-dynamic-linker /usr/lib/amd64/ld.so.1` would probably help 19:10:12 jbk: yea that's one think I wrote down in my notes 19:11:18 as for built-in ld vs. not.. rich is probably the best person to know (though I thought i saw he's going to be out of pocket for a bit).. if I had to guess, maybe similar to 'use libc' vs. 'issue syscalls directly' 19:11:21 I guess part of my ignorance here is that I've oft heard that we should always using the illumos ld(1). And I get why we might say that. But I'm also wondering if that's because other linkers just straight up cannot build a executable that runs on our system or if it's more for subtle reasons we says this. 19:11:45 in that it might allow us to insulate over some lower-level stuff that might otherwise cause issues 19:11:55 (but as I said, it's more of a guess than anything, so could easily have it wrong) 19:12:50 also i suppose, it removes some amount of potential versioning hell if worrying about specific versions of third party linkers 19:13:38 Yea, I undertand our stance around "we provide the system API/ABI for you", and I imagine part of that is the linker part. But zig seems to want to draw a hard line in the sand here, and I've also noticed an uptick of other languages (like Rust) that seem to want to experiment more with newer linkers. 19:14:50 I came across some issue in Rust that was basically "use lld everywhere", but obviously they haven't actually full committed to that path at this point 19:15:21 @jclulow THere are two big types to test: 19:15:53 1.) 710 family: X710 (10), XXV710 (25), XL710 (40) 19:16:34 2.) 722 family: X722 (I have this and THIS is the one that exhibits problems). THere might be other X722 actual boards, but I"m not sure. X722 is often found on motherboards. 19:18:23 could probably even do that with a dladm private property as well 19:22:22 * rzezeski tries to pretend not to see i40e talk 19:23:34 danmcd: that fix certainly feels vaguely familiar, many many years ago I remember discussing with Robert that I need a one-line change that Linux had made. But because of other work things I never got around to it. And that _might_ be the change I was thinking of. 19:23:58 rzezeski: another aspect might be that we might want extra 'stuff'.. e.g istr talk of eventually integrating the ctf stuff into ld so you don't need to do any post processing 19:24:40 rzezeski: It's not documented and Linux has this "if not VF do this simple thing". PITA. See the fix in https://kebe.com/~danmcd/webrevs/13230-newtry/ 19:27:39 danmcd: yea that's what I was referring to, though I feel like the change I wanted to make was during init, not during set promisc 19:27:50 but it had something to do with the default VSI 19:28:02 and I remember looking at Linux and being like "oh we have to do that too" 19:28:05 and I never got around to it 19:28:07 Eeesh I hope it's not a both thing? 19:28:31 well the problem is this was like 4-5 years ago? lol 19:28:31 X722 experiments show the extra dups disappear. The other cases need testing, though. 19:30:07 I have an X710, but that's just not something I'm willing to jump into right now, something like that would probably take me a week and I just don't have that desire right now. Maybe if I was getting paid for it. 19:32:23 jbk: yea CTF as part of ld would certainly be a good reason not to rewrite all this in zig, though they seem to be determined to own all codegen/linking... 19:36:13 since that also means things like dtrace SDT probes can work 19:36:41 (and for some reason, i wanted to typo that like it's an embarassing disease :P) 19:40:45 rzezeski: Basically, the system linker is the one we can actually control, support, improve, debug, etc 19:41:27 The suggestion to use the compiler driver (e.g., gcc) instead of ld directly is, I feel, mostly because it's probably easier when what you already have are C-like objects 19:41:43 There is no reason not to invoke ld itself, if you know what you're doing with it, etc 19:42:29 There's no reason other linkers can't be made to improve the fidelity and correctness of their outputs for illumos 19:42:46 Just ... it's a lot of work, and if we're going to improve a linker I would perseonally improve our own haha 19:43:19 Go also emits their own ELF binaries, and they basically mostly work as well as can be expected at this point -- Zig can certainly do the same if they want to 19:43:36 jclulow: makes sense, and yea, my gut feeling is: other linkers can probably build executables for illumos, but it might not always produce the best result and you should prefer the system's ld 19:43:43 I did NOT know that about Go 19:44:04 I mean, Go and Zig are philosophically quite close together as far as I can tell 19:44:35 We've also definitely had bullshit problems because of Go's DIY linker, but it's all just software. People can fix things. 19:44:44 Yes I think in some ways, and I knew go avoided libc, but I didn't realize it emits its own ELF. Though I guess I shouldn't be surprised by that. 19:44:52 i've not looked closely at zig, but for the love of god, i hope it has a better ffi story than go 19:44:56 I think they still get the OSABI value wrong for example? But it's also probably not _critical_ because they also don't do any of our extensions or features lol 19:45:16 jclulow: in any even, those are very helpful answers 19:45:19 jbk: It would be an impressive feat to come out worse FFI-wise 19:45:20 thanks 19:45:26 true 19:45:44 zig FFI is top-notch AFAICT 19:45:47 there was an article that described it as 'golang is an island'.. which is i think apt 19:45:50 ELF is also pretty bloody wishy-washy 19:45:52 rzezeski: oh thank god 19:46:48 rzezeski: Even if I'm not looking to write any Zig right now, I appreciate first-class support in up-and-coming runtimes and toolchains, so thanks for digging in there! 19:47:04 I know that, e.g., ncdu (a utility I love a lot) has been rewritten in Zig 19:47:11 I would be sad to not have ncdu 19:47:27 jclulow: Yea, this is stuff I don't know much about, and it made me wonder if the SYSV ABI covers some of this stuff, but not nearly enough...kind of like POSIX 19:47:50 What I'm definitely sure of is that we could make a lot of this clearer on https://illumos.org/docs 19:48:18 e.g., it would be good to refer to what actually constitutes our ABI etc 19:48:34 heh expecting useful changes to the SYSV ABI 19:48:35 I'm sure rmustacc would have thoughts about that 19:48:42 nowadays it is just legalizing GCCbugs 19:48:50 jclulow: haha, oh I'm sure he would 19:49:15 I think it will be even more important now that we're gradually minting a new platform (ARM) 19:55:54 rzezeski - did you look at how zig 0.9.1 is currently built from omnios extra? lld is pulled in there. 19:57:54 hadfl added that, he might now more. 19:58:02 speaking of languages, i should look at reporting pony again.. i almost had it working on illumos a long time ago, but ran into a cpp macro abuse from hell issue that i could never completely untangle.. though i think they (rightly) got rid of that 19:59:26 andyf: yea, I'm using a modified version of that to build 0.11.0 19:59:36 0.9.1 predates the self-hosted compiler 20:00:00 danmcd: Do you think _any_ single board in the 710 family is enough to make you happy? 20:00:04 That could well be why it's still on 0.9.1 20:00:30 andyf: yea, there has been at least one other person that tried to get self-hosted working and failed, we'll see what happens to me 20:00:56 maybe not "failed", but got stuck 20:01:39 I know some people have been experimenting with using the mold linker with rust for building illumos executables, so other linkers do work in some way or other. 20:32:23 jclulow: Honestly, the spanning-space for me would be: 20:32:27 - Any 710 20:32:36 - Any 722 with X557 PHY 20:32:46 - Any 722 standalone board (if it exists). 20:32:58 AIUI only the middle one manifests the problem. :upside-down: 20:33:56 as far as we know 😅 20:33:58 I have a 710 mobo edition (with X557 PHY) on Kebecloud. I know where I can find a 722 with X557 phy (either sjorge or there's an MNX box with it), the 722 standalone may be harder. 20:34:17 sjorge: I've tested 710 with X557 phy on Kebecloud.... doesn't exhibit the problem. 20:36:10 (But the combinatoric explosion per board... I've only done half of it (3/6 regression tests stated on that email) on the X710/X557 mobo edition. 20:37:09 I should update 13230 with explosion... 20:37:24 ebay seems to indicate there are X722-DA4's but those are spf not rj45 20:38:22 found a DA2 on ebay but at the 2nd hand price point it's a but steap for me 20:38:47 they are all spf+ it seems 20:38:58 I could use an SPF board anyway so it can speak with another SPF board (Mellanox ConnectX 6)... 20:39:14 I've updated fenix illumos#13230 20:39:15 BUG 13230: i40e has duplicate traffic when used with bhyve/snoop running (New) 20:39:16 ↳ https://www.illumos.org/issues/13230 20:39:22 See the test plan yourself. 20:42:27 ebay.com prices are tiny bit better, but 400-500 range is still a bit high for me to buy one 20:43:49 what's a LOM? those are way cheaper 20:44:22 lights out management? 20:44:36 aka ilom or ipmi or iDRAC or ... 20:44:57 that doesn't make sense for a nic perspective 20:45:10 it seems to be a formfactor 20:45:35 ILOM is frequently a dedicated NIC, sometimes on a daughter card. 20:45:58 * nomad has no further insight so shuts up again. 20:46:21 LOM = Lan On Motherboard ? 20:47:48 lights out management 20:52:25 I'm glad I'm not the only one who said that. :) 21:12:58 though if you've ever used intel's.. i think 'lots of misery' might also be appropriate 21:20:13 i've not looked to see what chip they actually use for it on their server systems 21:20:46 but if it turned out that they had a whole bunch of 8088s laying around and decided to repurpose them as their BMCs, it would not shock me 21:21:19 nah, they sold most of those to NASA already 21:22:09 browsing the internet over a 56k modem is generally more responsive than their BMCs 21:23:48 I was talking about this https://www.benl.ebay.be/itm/385571089437?hash=item59c5d3ec1d:g:6o0AAOSw0TNkR-IO&amdata=enc%3AAQAIAAAA4Mt%2FCmCAIRbv5mPoKpXgAFrdcldbG%2Fx8E1RusOCZxq4XQ1W6aRnpGm7GMXv6xxQOpbLmcxtUxUBgFDcNBrHhlfyDQxHVTOcwQGRD4%2BHmUqb1Dlp9%2BxjkbTiAVBdYmG2%2Ftyd5CNDPPiXmbzcnjWgz%2Bb1Stm9REFNy1oLcjXE%2FsF6UG5CqNR2B0DIJgVrNA74DOAnssl5Y0E0%2BF%2FLpRecyDkucBgxw75de%2FTzbsMqSAIFZQfU%2BbG6Tg6Fv02Vg%2Bh2G5y4IqWVJFaHlhPhLVxTNMQx4bh%2BLaajz5UPXn0Vu350 21:23:48 7%7Ctkp%3ABFBMspvW18li 21:23:52 off ok that links is horrible 21:23:53 sorry 21:24:10 Normal PCIe seems to be 4 or 5x that to start even 2nd hand 21:24:51 Perhaps it's some HP specific formfactor, wouldn't put it past them 21:30:40 Seems to indeed stand for Lan on Motherboard 21:30:42 TIL 21:37:32 sjorge: wow 21:38:23 Was kind of hoping it was just PCIe, as that price isn't too bad but a reg X722-DA2 or DA4, oh boy.