is there a modern recommended way of running the system without a usb stick?

yeh use piadm to install to disk

awesome, thx

what is the right thing to toggle in a svc manifest to not make a service start by default? i have a custom smf service but it should only be run manually and doesn't need to be run after booting

also, is there any built-in way for a(nother) service to detect that it's being invoked during boot? i'd like this service to run exactly once, after SMF comes up, and svcadm enable'ing it later shouldn't be possible. i'm sure i can do it manually by parsing some uptime-related thing, but i'm wondering if there's a built-in way

you can indicate in a manifest that it's disabled by default, so you'd have to explicitly enable it to run

and for most services, if they're enabled, they get run after any of their dependencies are satified during the boot process

w

<jbk> "and for most services, if they'..." <- ho?

you have to create a manifest, and then there's a spot in /opt (I tihnk /opt/custom, but i'd need to look) that'll import it during the boot process

yes, i know that. i meant how to make the service not autostart (i want the manifest to get imported (that's /opt/custom/smf), but the service to default to disabled)

Does Flexible Disk not work in the portal?

Any SmartOS C devs around that could take a look at this strace?

I wonder how github.com/TritonDataCenter/illumos…/lx/syscall/lx_socket.c#L3279-L3311 could return -1 EINVAL (Invalid argument)

This is the relevant line:

setsockopt(8, SOL_IP, IP_MTU_DISCOVER, [2], 4) = -1 EINVAL (Invalid argument)

Dumb question --> what numeric value is SOL_IP in Linux?

According to this:

It's "0" and that is an illegal value in general.

case LX_IPPROTO_IP:

that's how IP_MTU_DISCOVER gets shuffled.

One could argue we need to handle SOL_IP == 0 in LX (never mind the other definitions of SOL_* that Linux has done).

See lx_setsockopt()'s definition for where one *might* need to fix that.

I've not checked the latest POSIX or Austin defs, but SOL_IP seems.... Linux-y?

Regardless, that it's 0 is why you're getting EINVAL.

If you change SOL_IP to IPPROTO_IP that'll fix it in the source program.

danmcd (LIBERA-IRC): Thank you, the source program is caddy, a reverse proxy written in go

fwiw I built caddy natively, just needed a small patch to avoid SO_REUSEPORT

seemed to run fine

Sometimes I think go is a stealth project to force the LInux ABI on the world whether it likes it or not...

jperkin (LIBERA-IRC): Did you try to start caddy v2.6.x in an lx-brand zone? see smartos.topicbox.com/groups/smartos…-x-crashes-in-default-configuration

what's the proper way to set a v6 gateway for a zone? "gateways" doesn't seem to take ipv6

teutat3s: no in a native zone

jperkin (LIBERA-IRC): ah I didn't read that part well, gotcha

gonzosysadm[m]: That "gateways" doesn't take v6 is a bug, IIRC. You can, post-setup, use `route -p add -inet6 default <router-v6-addr>` which will have it persist across zone boots.

(I take it you don't wanna use NDP in this zone?)

(Sorry, Router Discovery?)

danmcd: noted .. seems the bug is open since 2016 too, heh

unrelated -- i need to use the same mac address for a zone as the one on the physical interface for the gz, but i get msg: dladm: vnic creation over i40e0 failed: MAC address reserved for use by underlying data-link when creating the zone

Yeah... "Make things v6-happy" is a longstanding issue that will require A Serious Effort.

multiple ips to a mac works, but the question is, does it work between a zone and the gz too?

No.

The ONLY way to have a physical NIC's mac address is to have that physical NIC itself assigned to the netstack (zone) you want.

If you have igb0 in the global zone, NO ZONE CAN ALSO HAVE igb0, for example.

On my home machine, I assign igb1 directly to the router zone (using "devices" in zonecfg(8)).

"MAC address reserved for use by underlying data-link" ==> means i40e0 owns that MAC, full-stop.

If you need it badly you'll need to assign i40e0 directly to the zone, and we don't allow that using the SmartOS vmadm(8) tooling.

yeah this is a machine on hetzner, they got some silly stuff going on with mac addresses.

problem is i also need the mac for the gz. i'm trying to avoid having to do the routing on the gz

Also of note, if you assign the physical NIC to the zone, any vnic on top of it can be snooped by root@zone.

unfortunately all packets going out of the machine need to have the right mac, or they get very unhappy

(My router zone is the ONLY one that uses igb1, in my example.)

danmcd (LIBERA-IRC): I'm confused, IPPROTO_IP looks like it's of the numeric value 0, too? github.com/TritonDataCenter/illumos…ommon/brand/lx/sys/lx_socket.h#L211

Shit... hang on pls.

Internally too...

sys/lx_socket.h:#define LX_IPPROTO_IP 0

Hmmm, I wonder...

Assuming you can just make this happen by running the program, I will find a DTrace script I'd like you to run.

Please grab this: kebe.com/~danmcd/webrevs/downstack.d

danmcd (LIBERA-IRC): sure, I did find this one and made it run before docs.smartos.org/lx-branded-zones/#probing-socket-options

chmod it to 0755, and run it as "./downstack.d lx_setsockopt"

It's a Big Hammer, and it will produce a lot of output.

You can also insert an "exit();" line on line 14 (in the $1:return probe) for a single capture.

So run downstack.d lx_setsockopt

once it shows N probes enabled (N will be large)

Run the LX program that's failing.

The D script will follow the bouncing callstack down and back up again. We can see which exact function returns an error.

Then one can source-dive in that function to figure out WTF.

(you can also edit the script to narrow its reach to, say, just the lx_brand module)

hmmm dtrace: failed to compile script ./downstack.d: line 3: probe description ::lx_setsockopt:entry does not match any probes

Ah, does it need to be run from the global zone?

Yes.

Sorry, should've been clear about that.

Oh shit, it may produce too much output if you have already-running other LX zones (or even processes).

(Like I said it's a big hammer.)

danmcd (LIBERA-IRC): bin.disroot.org/?6c8adc188510ff3d#5…yVpuUMAUtA8xMf2oYUVYx4uqg7XfPRsr3dc

Is that some useful output?

The CN seems to hang a bit, I might need to redirect it to a file to get all the output, if it looks like somethings missing

That shows a successful return of lx_setsockopt() (returns 0).

The CN hangs because that enables probes on ALL KERNEL FUNCTIONS>

Like I said it's a Big Hammer.

Lemme see if I can make some mods...

The last two probes, both have headers of:

::entry

and

::return

Change those to be:

fbt:lx_brand::entry

and

fbt:lx_brand::return

that'll lessen the impact

(Pardon latency... it's a Triton release week and I'm high-jitter Thursday afternoon and GONE until Monday when I'm at MNX.)

danmcd (LIBERA-IRC): No worries, thank you for looking into this with me

Found it.

One lx_setsockopt_ip call returns 0x16 == 22 == EINVAL.

Hard to figure out why right this second.

That function Should Work. I've followed the bouncing codepaths.

SOrry I can't help out more right this moment.

danmcd (LIBERA-IRC): thank you for your time (:

bahamat: The triton portal passes in options.dataset, and cloudapi doesn't know to put it on disks[0].image_uuid at github.com/TritonDataCenter/sdc-clo…pi/blob/master/lib/machines.js#L553

guessing triton portal should have options.image for input to api.Create or || params.dataset in cloudapi

Smithx10: What conditions trigger this? And what's the failure mode? I'm unfamiliar with this issue.

When a user tries to use a Flexible disk package I believe

And is this piranha, or adminui?

Piranha *

OK, we'll take a look at that and see if we can reproduce it.

Smithx10: Which version of Piranha?

Smithx10: And they're doing bhyve, right? flexible disk isn't supported for kvm, and makes no sense for lx/joyent brand

yea, bhyve

[root@5eaba95a-7637-6322-9d0a-9d08095d0db9 /opt/portal]# cat /opt/portal/package.json | json version; 6.0.1

OK, yeah, we'll try to see what's going on if we can.

Good timing, actually, because we're preparing a new release of piranha

yea, I just noticed that the create worked via node-triton, and not the portal with the same image and package. When I console logged the incoming requests into cloudapi from each, thats when I noticed github.com/TritonDataCenter/sdc-clo…pi/blob/master/lib/machines.js#L553 only acted on image. Piranaha was passing params.dataset.

Can you send me your package json out of papi?

Or to travisp

Just to make sure there's nothing we're missing in our testing.

package api? correct

Yeah

So from the headnode, sdc-papi /packages/<package uuid>

What you get from `triton package get -j` isn't exactly the same thing as what's in papi. CloudAPI will gloss over some of the details that end users mostly don't care about.

Smithx10: We'd also like to see your config.pro.json, but there's secrets in there so scrub them.

And probably give me an msigned link

PM it to me