00:06:59 <Smithx10> Is there a good doc for helping Linux folks learn about how to get around on Illumos?  Specifically trying to give good options for folks to inspect processes, what network, what files etc they are using?
00:44:36 <mxshift> https://wiki.smartos.org/the-linux-to-smartos-cheat-sheet/
00:44:51 <rzezeski> Smithx10: there is this resource: https://wiki.smartos.org/the-linux-to-smartos-cheat-sheet/
00:44:55 <rzezeski> lol
00:45:14 <rzezeski> so, there is that, but I'm not sure how much stuff like that really helps besides at a surface level
00:45:34 <rzezeski> I think what you also need to do, is collect the specific questions these folks are trying to answer
00:45:38 <rzezeski> and then pose them here
00:45:55 <rzezeski> and there might be other stuff we can point you to, or maybe someone could even write up a gist
00:54:11 <rzezeski> I think part of the problem in this area is that oftentimes you are just given a list of tools, but no real idea about how or when to use them, and to use the tools effectively you must have some knowledge of how the system works, because the tool is just answering questions about the system. In that regard, I think Gregg's DTrace book and the first edition of Systems Performance are both good in that regard.
00:54:23 <rzezeski> I think Solaris Performance and Tools is still pretty solid too
10:12:39 <ptribble> Any trick to getting a 9p mount to work in a bhyve guest?
10:13:12 <ptribble> I think I have the bhyve process with the right provileges and it's picked up the virtfs setup correctly.
10:13:25 <andyf> Shouldn't need a trick
10:14:09 <ptribble> The (Ubuntu 18) guest can see the virtfs device, but I get permission denied if I try and mount it.
10:14:13 <andyf> It should appear in the guest as a PCI device, so see if it's there with lspci
10:14:20 <andyf> ah, ok, I'll dig out the command I usually use
10:14:46 <andyf> mount -t 9p -o trans=virtio,rw,uname=root,dirsync <share name> /a
10:16:25 <ptribble> Aha! So the trick is to add uname=root
10:16:34 <ptribble> Thanks!
14:10:43 <KungFuJesus> Which OI/IPS package contains the kernel?  onbld? osnet? 
14:11:40 <yuripv> system/kernel?
14:11:52 <andyf> system/kernel/platform has `unix`, the first mach-dependent bit of the kernel
14:12:17 <KungFuJesus> hmm, well IPS then, while successfully completing update, is _not_ actually updating the kernel on OI right now
14:13:06 <KungFuJesus> I seem to be stuck on this commit hash at the moment: illumos-e4f7ccbe1e
14:14:45 <KungFuJesus> It also seems that most recent builds are only changing like 4 packages? https://hipster.openindiana.org/jenkins/job/illumos-gate/6862/console
14:15:35 <andyf> I've seen various reports of update problems on OI at the moment. I think it's because the osnet-incorporation needs rebuilding after the obsoleting of system/library/processor since older versions of that have been purged.
14:16:27 <KungFuJesus> damn, there's been a lot of useful fixes in smbd from Matt Barden that I'd like to test out.  The bug reports sound eerily like some of the oddities we've seen
14:16:47 <KungFuJesus> in particular I think we've hit the thread pool exhaustion one on multiple occassions for nfsd (though, that one isn't merged in yet)
16:21:09 <tsoome> aww, I can not test grub in this vm afterall... missing feature for read 'org.freebsd:zstd_compress'
17:30:15 <jclulow> andyf: Presumably it is not osnet-incorporation but rather one of their other metapackages
17:32:51 <andyf> I didn't spend a lot of time looking but it appears that osnet-incorporation has a lock for system/library/processor⊙0
17:33:01 <andyf> and I couldn't see that version of system/library/processor in the repository
17:36:31 <andyf> but you know how IPS error messages are.. :)
17:38:11 <jclulow> I suppose my assumption is that they get that package from the gate build like everyone else
17:38:17 <jclulow> But maybe that is unfounded
17:38:46 <andyf> It is, as far as I remember mostly auto-generated based on whether other manifests have the noincorp flag set
19:08:25 <richlowe> we often have no-incorped an obselete, for distro reasons
19:09:15 <richlowe> it is not in our package
19:09:27 <richlowe> so that seems like if not our fault, at least partially our fault
19:10:09 <richlowe> (I'm not sure how I have _not_ had this problem, to be honest, but I seem not to have?)
19:10:21 <richlowe> but that also means I'm not really in a position to fix it, if andy says noincorp is the way
19:16:42 <gitomat> [illumos-gate] 15037 cp_subr.s: Invalid instruction argument -- Toomas Soome <tsoome⊙mc>
19:18:28 <richlowe> I filed #15075 (fenix?), if people want bite-sized bugs
19:18:30 <fenix> BUG 15075: cmd/initpkg/dfstab.sh generates a completely static file (New)
19:18:30 <fenix> ↳ https://www.illumos.org/issues/15075
19:19:58 <gitomat> [illumos-gate] 15041 smbadm join reports clock skew as bad password -- Gordon Ross <gwr⊙rc>
19:24:12 <gitomat> [illumos-gate] 15058 vmm: comparison is always true -- Toomas Soome <tsoome⊙mc>
19:27:04 <gitomat> [illumos-gate] 15042 AD join fails if hostname longer than 14 characters -- Gordon Ross <gwr⊙rc>
19:30:01 <gitomat> [illumos-gate] 15051 SMB2 signature mismatch gets wrong error -- Gordon Ross <gwr⊙rc>
19:39:51 <gitomat> [illumos-gate] 14690 nvmeadm should refer to namespaces by their nsid -- Hans Rosenfeld <rosenfeld⊙gho>
19:41:23 <toasterson> For the people wondering about osnet on oi builds. osnet incorporation is frozen inside the build VM, so that pkg can update every build without new be and not killing the machine with a new kernel.
19:41:43 <toasterson> We probably need to update that server manually and then the next build should be ok again
19:44:13 <toasterson> Although that server has 564d52366b according to mtod
19:47:35 <richlowe> can people switch the facet, and then have it go away?
19:47:42 <richlowe> since that would be equivalent to us having noincorp'd it
19:47:52 <richlowe> which again, I can't really fix, since none of this happened to me :\
19:48:53 <toasterson> probably. But I now doubt it was the update issue
19:49:01 <toasterson> I'll have to see if I can reproduce
20:06:43 <KungFuJesus> today it allowed me to complete the update but it only updated like 700k worth of files.  I'm still stuck on a much older kernel
20:54:34 <gitomat> [illumos-gate] 15069 dladm_str2bw() does not reset errno which is checked after string conversion -- Dmitry Gromada <dmitry.gromada⊙nc>
22:02:47 <toasterson> yeah and I am locked out of all updated systems from a mac via SSH
22:02:54 <toasterson> I can only zlogin in
22:05:06 <toasterson> Did we do a change to pty recently. Between september 2022 and October 2022 I get a self closing ssh connection
22:05:08 <toasterson> debug1: Setting controlling tty using TIOCSCTTY.
22:05:08 <toasterson> debug1: Received SIGCHLD.
22:05:33 <toasterson> after that it gets a debug1: session_exit_message: session 0 channel 0 pid 20275
22:07:29 <toasterson> Jenkins remoting works
22:07:41 <toasterson> so it's bash that closes
22:07:49 <gitomat> [illumos-gate] 15065 vmm: variable set but not used -- Toomas Soome <tsoome⊙mc>
22:15:24 <toasterson> zsh works so it's bash.... but why
22:16:59 <toasterson> and the shell interaction is weird
22:17:22 <danmcd> For those who are sufficiently old:  I just installed `punchin` on a new VM.  :)
22:18:48 <tsoome> .oO 
22:21:22 <tsoome> .oO should we drop grub?:)
22:23:04 <danmcd> Is there a push coming up that'll break it?  I ask because literally just today I had a customer report something on their GRUB-booting USB key.  Knowing that's coming Will Be Important (TM).
22:24:27 <tsoome> no. but then again, *if* we are keeping it, there are something to be fixed 
22:24:56 <tsoome> well, unless we tell that we do not really care if its broken...
22:25:09 <danmcd> Yeah... TBH I'd LOVE to get rid of GRUB, but it'll increase our support load to make sure people are moving off of it in cases where they still have very old USB keys that boot grub.
22:25:47 <danmcd> (our == "SmartOS & Triton", mostly Triton)
22:26:00 <tsoome> anyhow, obviously this can not happen overnight.
22:28:07 <tsoome> but yes, reboot -f does not do fast reboot with grub because libgrubmgt is not there any more (and do not really get that one back;)
22:31:38 <tsoome> something simpler can be implemented but ...  anyhow, time to get some sleep:D
22:35:31 <danmcd> Get a good night's worth!
22:36:06 <toasterson> Any reason people have such old USB keys? Other than not wanting to upgrade at all
22:38:59 <richlowe> I think smartos stayed with grub longer than you might think
22:39:05 <richlowe> so I don't think they're _that_ old
22:39:23 <richlowe> if they use the smartos BE stuff, they're going to get an awful shock when the menu.lst thing happens though
22:39:28 <jclulow> I don't think we need to stop _working with multiboot_ to remove GRUB
22:39:38 <jclulow> Because we already also work with, e.g., iPXE and other multiboot loaders
22:40:15 <jclulow> And, yeah, SmartOS doesn't use the in-gate tooling to maintain the key contents
22:40:42 <jclulow> I think you could easily make the case for not producing any new GRUB-bearing USB key images while also not breaking existing SmartOS/GRUB users
22:40:45 <danmcd> What jclulow said ^^^  and @toasterson ==> Triton Head Nodes tend to be maintained entirely by Triton tooling which lets you Just Use The USB Key.
22:41:39 <danmcd> There are loader-booting USB keys now, but many are so long-lived they never bothered to move out from GRUB, never mind to disk-booting SmartOS (which is just like loader-booting USB keys except the "USB Key" lives on a zpool)
22:41:54 <danmcd> (specifcally a bootfs on a zpool.)
22:42:19 <jclulow> danmcd: I have some thoughts about transition from UFS to ZFS for SmartOS, btw; should I write them to you?
22:42:38 <jclulow> (we're using a ZFS-based ramdisk image at Oxide, and I think I can make it so that bits of it will work for you too)
22:43:00 <danmcd> Send me a mail, sure.  (And hopefully my find-bugs-during-the-sideport gifts will help both of us!)
22:43:03 <danmcd> Thank you!
22:43:35 <toasterson> What are benefits for a ZFS based Ramdisk? 
22:43:42 <danmcd> No Y2038 problem for one thing.
22:43:43 <jclulow> No 2038 problems
22:43:48 <jclulow> also in-place compression
22:43:54 <jclulow> and online expansion to a larger size
22:44:03 <toasterson> Ah, yeah thats a big one :)
22:44:06 <jclulow> e.g., you could ship a 200MB pool image that unfurls to 1GB at boot time
22:44:20 <jclulow> You can set quotas, create datasets, etc
22:44:26 <danmcd> Makes those iPXE times shorter for sure.
22:44:44 <jclulow> Also for SmartOS you could stop doing the lofi /usr hack
22:44:51 <jclulow> Just ship a flat image
22:45:23 <danmcd> By making /usr a read-only filesystem, right?
22:45:31 <jclulow> If you want!
22:45:44 <toasterson> whats the /usr lofi hack?
22:46:09 <jclulow> SmartOS does something a bit like an installer image, where there is a compressed lofi image shipped inside the ramdisk
22:46:17 <jclulow> which gets mounted over /usr
22:46:19 <jclulow> early in boot
22:46:25 <danmcd> gist coming (then I gotta run for dinner)...
22:46:43 <toasterson> oh, I thought it was one disk
22:46:44 <danmcd> https://gist.github.com/danmcd/6d35e69632e8f3c75b63a5913c8cb5b8
22:47:01 <jclulow> It is a disk within a disk
22:47:06 <danmcd> It is, the lofi image comes from /usr.lgz
22:47:13 <danmcd> NOW it's dinnertime.
22:47:57 <toasterson> ah because needs to be RO? I assume?
22:48:52 <jclulow> it's actually to make it smaller
22:50:41 <jclulow> It being read-only has actually been historically inconvenient at various moments haha
22:50:52 <jclulow> (it's always read-only inside a zone)
22:51:11 <jclulow> on the host though we have had to contort ourselves at various times to patch things or add stuff
22:56:30 <jbk> heh
22:59:24 <richlowe> I guess -Clzma is the best we can do?
22:59:30 <richlowe> (lofiadm -C lzma, that is)
23:00:46 <toasterson> hmmm I see ZFS starts to make a lot of sense
23:02:06 <richlowe> well, this is more the joys of small media
23:02:13 <richlowe> (and some smartos decisions)
23:02:22 <richlowe> it's basically the same awful tradeoff that caiman makes
23:02:59 <richlowe> I think alexander eremin actually did even more weird stuff for caiman
23:05:44 <jclulow> I suspect they should have just implemented squashfs or something like it for the install miniroot
23:05:47 <toasterson> Is there a guide how one can boot as ZFS Ramdisk image?
23:05:48 <toasterson> or create one?
23:05:58 <jclulow> Nope!  It's all work in progress stuff
23:07:37 <jclulow> What I have so far is: unix and a cpio boot_archive that just contains modules needed to get the ramdisk, and then a new module hook just prior to vfs_mountroot() where we'll let you make a ramdisk available and choose to mount it
23:07:58 <jclulow> for SmartOS I suspect they would then have the ZFS ramdisk image in another multiboot module
23:08:25 <jclulow> (ZFS needs a few new bootprop things to communicate that we want to import a specific ramdisk /devices path uncritically)
23:08:53 <jclulow> for the Oxide thing we are reading a ramdisk image (with a small header) from s0 on an NVMe device
23:09:05 <jclulow> which we unpack into a regular ramdiskadm-style ramdisk
23:09:37 <jclulow> of a particular size, which can be larger than the actual image size
23:09:53 <jclulow> and then we adjust the boot props to do use that device as the root pool
23:10:09 <richlowe> and zfs autoexpand into the ramdiskadm ramdisk?
23:10:23 <toasterson> I thought it wouldn't be as easy as lofiadm->zfs inside and laod :)
23:10:26 <jclulow> yup, that's where https://www.illumos.org/issues/14978 comes from
23:10:27 <fenix> → BUG 14978: ZFS autoexpand property should work for root pools (New) | https://code.illumos.org/c/illumos-gate/+/2355
23:10:44 <richlowe> that would make me so happy, for image-builder, too
23:10:44 <jclulow> Yeah this is all in-kernel change because it's prior to the first process
23:12:38 <jclulow> richlowe: I just have to do the last round of testing I think and then autoexpand will work
23:29:18 <tozhu> hello experts, does any tested zfs write (most 8k size) performance vs xfs?
23:29:43 <richlowe> probably not in a great many years
23:34:32 <tozhu> I have a client did the tests with postgres, they stress postgres by standard TPC-C, that tool generate about 50% writes for file system by database WAL log; on zfs, they insert/update a bunch of data have spent 300ms, but on xfs did the same, they just only spent 30ms; disks are NVMe SSD
23:35:01 <tozhu> so I’m not sure if there are write performance issue with zfs
23:38:24 <toasterson> Any people had this here happen before? https://youtu.be/NxoW_lNy70k
23:49:09 <jclulow> toasterson: Aren't you missing the ssh -t flag
23:51:03 <toasterson> that helps a bit but it broke on upgrade
23:51:34 <jclulow> Also, in that video you're using zsh not bash?
23:51:34 <toasterson> no only gets me a terminal on zsh but still crashes bash
23:52:12 <toasterson> yep :) bash gets a SIGCHILD and then quits
23:52:56 <toasterson> hmm motd does not get displayed when calling command
23:53:02 <toasterson> ok so thats ok
23:53:08 <toasterson> and -t does the thing
23:53:24 <toasterson> now I just need to find out why bash quits
23:54:44 <jclulow> When does it quit exactly?
23:55:30 <toasterson> when it starts and before it prints the prompt
23:55:46 <jclulow> What about: ssh -t $host bash --noprofile --norc
23:56:19 <toasterson> https://pastebin.com/25pPESNp
23:56:58 <toasterson> --noprofile --norc also quits the same
23:57:08 <jclulow> That's odd
23:57:58 <jclulow> What if you run bash under "ssh -t $host zsh"
23:58:02 <jclulow> can you truss it etc