08:22:50 <gonzosysadm[m]> is there a modern recommended way of running the system without a usb stick? 
08:24:08 <jperkin> yeh use piadm to install to disk
09:36:45 <gonzosysadm[m]> awesome, thx
13:04:32 <gonzosysadm[m]> what is the right thing to toggle in a svc manifest to not make a service start by default? i have a custom smf service but it should only be run manually and doesn't need to be run after booting
13:05:40 <gonzosysadm[m]> also, is there any built-in way for a(nother) service to detect that it's being invoked during boot? i'd like this service to run exactly once, after SMF comes up, and svcadm enable'ing it later shouldn't be possible. i'm sure i can do it manually by parsing some uptime-related thing, but i'm wondering if there's a built-in way
13:37:20 <jbk> you can indicate in a manifest that it's disabled by default, so you'd have to explicitly enable it to run
13:38:00 <jbk> and for most services, if they're enabled, they get run after any of their dependencies are satified during the boot process
13:46:21 <gonzosysadm[m]> w
13:46:22 <gonzosysadm[m]> <jbk> "and for most services, if they'..." <- ho?
13:57:06 <jbk> you have to create a manifest, and then there's a spot in /opt (I tihnk /opt/custom, but i'd need to look) that'll import it during the boot process
14:04:13 <gonzosysadm[m]> yes, i know that. i meant how to make the service not autostart (i want the manifest to get imported (that's /opt/custom/smf), but the service to default to disabled)
14:30:53 <Smithx10> Does Flexible Disk not work in the portal?
14:34:15 <teutat3s> Any SmartOS C devs around that could take a look at this strace?
14:34:18 <teutat3s> https://bin.disroot.org/?82515b9df256399e#2pCpo1kNEeNhDk2ocopEtbnWrTcevGxj5xE7B2WPzwYE
14:34:28 <teutat3s> I wonder how https://github.com/TritonDataCenter/illumos-joyent/blob/master/usr/src/uts/common/brand/lx/syscall/lx_socket.c#L3279-L3311 could return -1 EINVAL (Invalid argument)
14:35:52 <teutat3s> This is the relevant line:
14:35:53 <teutat3s> setsockopt(8, SOL_IP, IP_MTU_DISCOVER, [2], 4) = -1 EINVAL (Invalid argument)
14:39:30 <danmcd> Dumb question --> what numeric value is SOL_IP in Linux? 
14:39:32 <danmcd> According to this:
14:39:33 <danmcd> https://sites.uclouvain.be/SystInfo/usr/include/bits/in.h.html
14:41:42 <danmcd> It's "0" and that is an illegal value in general.
14:42:36 <danmcd>         case LX_IPPROTO_IP: 
14:42:46 <danmcd> that's how IP_MTU_DISCOVER gets shuffled.
14:43:32 <danmcd> One could argue we need to handle SOL_IP == 0 in LX (never mind the other definitions of SOL_* that Linux has done).
14:43:51 <danmcd> See lx_setsockopt()'s definition for where one *might* need to fix that.
14:44:15 <danmcd> I've not checked the latest POSIX or Austin defs, but SOL_IP seems.... Linux-y?
14:44:24 <danmcd> Regardless, that it's 0 is why you're getting EINVAL.
14:44:48 <danmcd> If you change SOL_IP to IPPROTO_IP that'll fix it in the source program.
14:48:34 <teutat3s> danmcd (LIBERA-IRC): Thank you, the source program is caddy, a reverse proxy written in go
14:49:10 <jperkin> fwiw I built caddy natively, just needed a small patch to avoid SO_REUSEPORT
14:49:14 <jperkin> seemed to run fine
14:49:17 <danmcd> Sometimes I think go is a stealth project to force the LInux ABI on the world whether it likes it or not...
14:50:24 <teutat3s> jperkin (LIBERA-IRC): Did you try to start caddy v2.6.x in an lx-brand zone? see https://smartos.topicbox.com/groups/smartos-discuss/Te8e92fc14c1b3f9d/lx-brand-caddy-2-6-x-crashes-in-default-configuration
14:50:32 <gonzosysadm[m]> what's the proper way to set a v6 gateway for a zone? "gateways" doesn't seem to take ipv6 
14:50:45 <jperkin> teutat3s: no in a native zone
14:51:09 <teutat3s> jperkin (LIBERA-IRC): ah I didn't read that part well, gotcha
14:53:15 <danmcd> gonzosysadm[m]: That "gateways" doesn't take v6 is a bug, IIRC.  You can, post-setup, use `route -p add -inet6 default <router-v6-addr>` which will have it persist across zone boots.
14:53:25 <danmcd> (I take it you don't wanna use NDP in this zone?)
14:53:32 <danmcd> (Sorry, Router Discovery?)
14:55:34 <gonzosysadm[m]> danmcd: noted .. seems the bug is open since 2016 too, heh
14:56:02 <gonzosysadm[m]> unrelated -- i need to use the same mac address for a zone as the one on the physical interface for the gz, but i get msg: dladm: vnic creation over i40e0 failed: MAC address reserved for use by underlying data-link  when creating the zone
14:56:29 <danmcd> Yeah... "Make things v6-happy" is a longstanding issue that will require A Serious Effort.
14:56:36 <gonzosysadm[m]> multiple ips to a mac works, but the question is, does it work between a zone and the gz too?
14:56:42 <danmcd> No.
14:57:11 <danmcd> The ONLY way to have a physical NIC's mac address is to have that physical NIC itself assigned to the netstack (zone) you want.
14:57:27 <danmcd> If you have igb0 in the global zone, NO ZONE CAN ALSO HAVE igb0, for example.
14:57:45 <danmcd> On my home machine, I assign igb1 directly to the router zone (using "devices" in zonecfg(8)).
14:58:20 <danmcd> "MAC address reserved for use by underlying data-link" ==> means i40e0 owns that MAC, full-stop.
14:58:41 <danmcd> If you need it badly you'll need to assign i40e0 directly to the zone, and we don't allow that using the SmartOS vmadm(8) tooling.
14:58:42 <gonzosysadm[m]> yeah this is a machine on hetzner, they got some silly stuff going on with mac addresses.
14:59:11 <gonzosysadm[m]> problem is i also need the mac for the gz. i'm trying to avoid having to do the routing on the gz 
14:59:27 <danmcd> Also of note, if you assign the physical NIC to the zone, any vnic on top of it can be snooped by root@zone.
14:59:35 <gonzosysadm[m]> unfortunately all packets going out of the machine need to have the right mac, or they get very unhappy 
14:59:38 <danmcd> (My router zone is the ONLY one that uses igb1, in my example.)
15:09:20 <teutat3s> danmcd (LIBERA-IRC): I'm confused, IPPROTO_IP looks like it's of the numeric value 0, too? https://github.com/TritonDataCenter/illumos-joyent/blob/master/usr/src/uts/common/brand/lx/sys/lx_socket.h#L211
15:09:43 <danmcd> Shit... hang on pls.
15:10:26 <danmcd> Internally too...
15:10:26 <danmcd> sys/lx_socket.h:#define LX_IPPROTO_IP           0 
15:10:39 <danmcd> Hmmm, I wonder...
15:11:22 <danmcd> Assuming you can just make this happen by running the program, I will find a DTrace script I'd like you to run.
15:11:52 <danmcd> Please grab this:  https://kebe.com/~danmcd/webrevs/downstack.d
15:12:19 <teutat3s> danmcd (LIBERA-IRC): sure, I did find this one and made it run before https://docs.smartos.org/lx-branded-zones/#probing-socket-options
15:12:54 <danmcd> chmod it to 0755, and run it as "./downstack.d lx_setsockopt"
15:13:03 <danmcd> It's a Big Hammer, and it will produce a lot of output.
15:14:13 <danmcd> You can also insert an "exit();" line on line 14 (in the $1:return probe) for a single capture.
15:14:24 <danmcd> So run downstack.d lx_setsockopt
15:14:35 <danmcd> once it shows N probes enabled (N will be large)
15:14:45 <danmcd> Run the LX program that's failing.
15:15:04 <danmcd> The D script will follow the bouncing callstack down and back up again.  We can see which exact function returns an error.
15:15:18 <danmcd> Then one can source-dive in that function to figure out WTF.
15:17:43 <danmcd> (you can also edit the script to narrow its reach to, say, just the lx_brand module)
15:19:46 <teutat3s> hmmm dtrace: failed to compile script ./downstack.d: line 3: probe description ::lx_setsockopt:entry does not match any probes
15:21:58 <teutat3s> Ah, does it need to be run from the global zone?
15:31:40 <danmcd> Yes.
15:31:45 <danmcd> Sorry, should've been clear about that.
15:32:14 <danmcd> Oh shit, it may produce too much output if you have already-running other LX zones (or even processes).
15:32:29 <danmcd> (Like I said it's a big hammer.)
15:35:43 <teutat3s> danmcd (LIBERA-IRC): https://bin.disroot.org/?6c8adc188510ff3d#5zUR1nRyTyVpuUMAUtA8xMf2oYUVYx4uqg7XfPRsr3dc
15:35:58 <teutat3s> Is that some useful output?
15:36:55 <teutat3s> The CN seems to hang a bit, I might need to redirect it to a file to get all the output, if it looks like somethings missing
15:37:42 <danmcd> That shows a successful return of lx_setsockopt()  (returns 0).
15:38:01 <danmcd> The CN hangs because that enables probes on ALL KERNEL FUNCTIONS>
15:38:05 <danmcd> Like I said it's a Big Hammer.
15:38:10 <danmcd> Lemme see if I can make some mods...
15:39:42 <danmcd> The last two probes, both have headers of:
15:39:51 <danmcd> ::entry 
15:39:53 <danmcd> and 
15:39:54 <danmcd> ::return
15:39:59 <danmcd> Change those to be:
15:40:01 <danmcd> fbt:lx_brand::entry 
15:40:02 <danmcd> and 
15:40:04 <danmcd> fbt:lx_brand::return
15:40:14 <danmcd> that'll lessen the impact
15:41:36 <danmcd> (Pardon latency... it's a Triton release week and I'm high-jitter Thursday afternoon and GONE until Monday when I'm at MNX.)
15:46:38 <teutat3s> danmcd (LIBERA-IRC): No worries, thank you for looking into this with me
15:46:40 <teutat3s> https://bin.disroot.org/?df3d3ba46ac994f5#G2bhFp1WqFmxpx1MfKLW7aBrm3Xe9FtYEZbLYV29gx5p
15:48:58 <danmcd> Found it.
15:49:22 <danmcd> One lx_setsockopt_ip call returns 0x16 == 22 == EINVAL.
15:56:11 <danmcd> Hard to figure out why right this second.
15:58:38 <danmcd> That function Should Work.  I've followed the bouncing codepaths.
15:59:06 <danmcd> SOrry I can't help out more right this moment.
16:08:03 <teutat3s> danmcd (LIBERA-IRC): thank you for your time (:
16:17:21 <Smithx10> bahamat:  The triton portal passes in options.dataset,  and cloudapi doesn't know to put it on disks[0].image_uuid   at https://github.com/TritonDataCenter/sdc-cloudapi/blob/master/lib/machines.js#L553
16:18:20 <Smithx10> guessing triton portal should have options.image for input to api.Create or || params.dataset in cloudapi
16:28:31 <bahamat> Smithx10: What conditions trigger this? And what's the failure mode? I'm unfamiliar with this issue.
16:31:35 <Smithx10> When a user tries to use a Flexible disk package I believe
16:31:58 <bahamat> And is this piranha, or adminui?
16:32:15 <Smithx10> Piranha *
16:32:34 <bahamat> OK, we'll take a look at that and see if we can reproduce it.
16:33:13 <travisp> Smithx10: Which version of Piranha?
16:33:32 <bahamat> Smithx10: And they're doing bhyve, right? flexible disk isn't supported for kvm, and makes no sense for lx/joyent brand
16:33:43 <Smithx10> yea,  bhyve 
16:34:27 <Smithx10> [root@5eaba95a-7637-6322-9d0a-9d08095d0db9 /opt/portal]# cat /opt/portal/package.json | json version;  6.0.1
16:37:05 <bahamat> OK, yeah, we'll try to see what's going on if we can.
16:38:14 <bahamat> Good timing, actually, because we're preparing a new release of piranha
16:42:22 <Smithx10> yea,  I just noticed that the create worked via node-triton, and not the portal with the same image and package.  When I console logged the incoming  requests into cloudapi  from each, thats when I noticed https://github.com/TritonDataCenter/sdc-cloudapi/blob/master/lib/machines.js#L553 only acted on image.  Piranaha was passing params.dataset. 
16:43:06 <bahamat> Can you send me your package json out of papi?
16:43:12 <bahamat> Or to travisp 
16:43:40 <bahamat> Just to make sure there's nothing we're missing in our testing.
16:43:44 <Smithx10> package api? correct
16:43:50 <bahamat> Yeah
16:44:10 <bahamat> So from the headnode, sdc-papi /packages/<package uuid>
16:45:11 <bahamat> What you get from `triton package get -j` isn't exactly the same thing as what's in papi. CloudAPI will gloss over some of the details that end users mostly don't care about.
16:45:48 <Smithx10> https://gist.github.com/Smithx10/1ad12898607fb2d56a84f8797a96e6dd
16:49:32 <bahamat> Smithx10: We'd also like to see your config.pro.json, but there's secrets in there so scrub them.
16:49:43 <bahamat> And probably give me an msigned link
16:49:55 <bahamat> PM it to me