I like to imagine it like the beacons of Gondor being lit

is DilOS the only illumos/open-solaris based distro that is both a) still actively worked on, and b) uses debian packaging / apt ?

stevenm_ : were there other illumos distros using apt/debian packaging despite dilos?

well nexenta os was

aka illumian

it's an interesting question, but shouldn't IPS work fine too?

illumian was never a thing, and nexenta stopped using debian packaging with their 5.0 release five years ago. and the whole debian stuff never worked that well for it anyway.

Woodstock, according to this... illumos.org/docs/about/distro

NexenStor is still using APT

don't believe everything you read on the internet

shame if it's not true

tbh I think my dream scenario would be an official debian port ... like they do for the kfreebsd and hurd kernels

igork, ^ :P

stevenm_: dilos is active and many into at #dilos channel :)

Woodstock, strange I watched a whole presentation on youtube about illumian being a thing last night :P youtube.com/watch?v=P4SS3R_4oy8

the audience looked a little empty - and it was 9 years ago :P

stevenm_: it was a debian fork for illumos like kfreebsd but like kfreebsd it did not find traction and died about 3 years after

toasterson, so what'd be the proper term (if 'k' is 'kernel of') if debian did have a port? in the illumos/opensolaris sense?

kIllumos ?

or is the kernel still seen as the OpenSolaris kernel?

"OpenSolaris" hasn't been a thing for well over a decade, Solaris is very much not open, and if you find anything still referencing that name (other than historical) then it should be removed

stevenm_: no clue. we never gave the kernel a dedicated name. the source is illumos-gate but that also includes userland things not just the kernel.

toasterson, well you could say that the linux kernel 'project' hosts a few userland tools too

but no one calls the linux kernel (between your normal desktop/server gnu/linux distro's - nothing exotic like android) "forks" just because they might add/remove the odd patch or select/deselect things at compile time

so maybe 'illumos-gate' kernel is about the right name? doesn't uname still show SunOS for some reason!? :D

it does, but that has historical reasons. C programs still need it to detect things sometimes.

hmm, if I ask google to show me any pages mentioning "NexentaStor CE" OR "NexentaStor community edition" for the last 2 years...

I get 8 pages lol

I would just call it "the illumos kernel".

That's what `uname -o` shows, after all

certainly better than the threatening looking "killumos"

rmustacc: I've got a system getting an error creating overlays. Wondering if you've got any insight before I get lost in this stuff.

Error is: dladm: warning: failed to set property vxlan/listen_ip

The IP specified *is* plumbed (at least according to ipadm)

killumos sounds like what a terminator would run on

weren't they commodore 64 or something like that?

when you saw the code on screen

It was 6502 at least, IIRC

it was a couple ones depending on era. Terminator 4 ran linux. I don't know the older ones.

bahamat: It didn't tell you why it failed did it?

rmustacc: It says "dladm: overlay creation failed: <unknown error>"

Ah, that's too bad. I didn't remember how much additional info we had there.

There'll probably be a kernel ioctl that's issuing it.

Yeah, the dladm_overlay_setprop function does ioctl(dladm_dld_fd(handle), OVERLAY_IOC_SETPROP, &prop)

Yeah, I was hoping it would have given you a bit more there, but it's been 5+ years since i've looked at any of this.

Maybe grab a quick truss of it?

truss is good, and I think I have a good kernel place to throw downstack.d at.

It happens during vmadm start, so it's hard to grab.

OK, one min.

I can give you a dtrace thing for it and then I have to get back to something for danmcd.

Sweet, that'll help a lot. I was looking at writing one myself

dtrace -n 'fbt::vxlan_o_setprop:entry{ printf("%s\n", stringof(args[1])); }' -n 'fbt::vxlan_o_setprop:return{ trace(arg1); }'

Something like that.

Haven't built it.

ok, let me see what I can get out of that.

(That's the function I want to downstack...)

It doesn't call anything else.

It all returns directly.

Oh...

.. nothing that'll get reached at all on a failure. Gotcha.

I mean, you'll call bcopy, but that's not useful.

Or mutex calls.

But there are no return values generated from downstack stuff.

That's what that ioctl is doing?

Yes.

Eventually.

Hmm, crap. dtrace: invalid probe specifier

The CN I'm testing on has no vms with overlays currently. I tried on another one that does already have overlays and that worked.

Invalid probe specifier or doesn't match any probes?

I copied it

Hmm. modinfo says overlay is loaded

It comes from the vxlan module. If had said it didn't match any probes then you'd just toss in a -Z.

*If it had

Yeah, vxlan isn't loaded

Ok, even with -Z it was giving me an error about, but I modloaded it, and dtrace is happy now.

It probably would have complained about using args[1].

what is the address anyways?

of the module?

or the listen_ip?

The IP.

-p svp/underlay_ip=10.3.248.238 -p vxlan/listen_ip=10.3.248.238

Things that I don't think matter: They have aggrs, lldp falls to maintenance

We had both of that happening at Joyent and it never caused this issue.

So most of the checks are all about whether or not something is a multicast address.

There are also CNs running the same PI that aren't having this error, but the ones that do, it's consistent.

So I think the quesiton I'd ask is whether we're somehow not properly doing an endian encoding.

As 238.* is an IPv4 mcast address.

But that's why I'd confirm that we're getting EINVAL, that that was the property, and then we'll change it to print the data.

Ah...that may be.

I'll see if there are any other servers exhibiting this issue or just the one.

Well, we should go diagnose the bug.

I'll try mocking up this IP on a CN and see what I get out of that dtrace

Can't force the provision to that CN?

If that is right (and I sincerely hope not), then anything that ends in .224-.239 would fail.

It's a customer, so I'm preparing a set of things for them to get back to me with, but it'll probably be tomorrow at the earliest that I'd get a response.

You should be able to mock that up.

As long as you never start the overlay the set-prop will work.

Because we don't try to use that until we get further there.

So even on a system without that IP you can try it and see.

It'll fail in a different way later.

I'll check with Joyent to see if they have any underlay addresses with that host ID. If they do, and they have no issues there, that would seem to absolve the endianness

OK. Well if you have more info, let me know.

OK, so I could just do the dladm create-overlay even without plumbing it up and it should hit the same error, if it's an issue with the IP itself?

FWIW: vxlan_o_setprop:return 22

Hmm, just EINVAL.

So you'd want to modify the DTrace to then print the opening bits there.

EINVAL is the only error that whole chunk there.

So basically print the property name. If it maches what you expect, then you want DTrace to print the buffer as an in6_addr.

So I'm going to have to do that in the call to strcmp(), right?

O, no, actually I don't need to

No, you use the entry probe and predicate.

Yeah

And if it matches, then you case arg2 to the right type and print.

How do I make it print as a type?

Something like print((struct in6_addr *)arg2);

But you'd only want to do that if the predicate about the property name matches.

maybe: `print(*(struct in6_addr *)arg2)` (so you don't just print it as pointer)

ah yeah, thanks.

Ah, ok, here we go:

uint8_t [16] _S6_u8 = [ 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0xff, 0xff, 0xa, 0x3, 0xf8, 0xee ]

So that macro does this: #define IN6_IS_ADDR_MULTICAST(a) ((a)->s6_addr[0] == 0xff)

Now I want to get at that s6_addr

wait, we're probably falling all the way through to IN6_IS_ADDR_V4MAPPED so what I really want to look at is IN_MULTICAST

OK, so if I manually `0xeef8030a & 0xf0000000`, that does indeed equal 0xe0000000.

rmustacc: Thanks for your help here, you literally saved me days of trying to decipher all of this on my own.

Callers to IN_MULTICAST must normalize-to-host-order whatever it feeds. Other callers of IN_MULTICAST use ntohl() first, so should the overlay code in question.

(Don't ask me WHY it's that way, likely a 4BSD choice inherited from 2BSD.)