Is there a good doc for helping Linux folks learn about how to get around on Illumos? Specifically trying to give good options for folks to inspect processes, what network, what files etc they are using?

Smithx10: there is this resource: wiki.smartos.org/the-linux-to-smartos-cheat-sheet

lol

so, there is that, but I'm not sure how much stuff like that really helps besides at a surface level

I think what you also need to do, is collect the specific questions these folks are trying to answer

and then pose them here

and there might be other stuff we can point you to, or maybe someone could even write up a gist

I think part of the problem in this area is that oftentimes you are just given a list of tools, but no real idea about how or when to use them, and to use the tools effectively you must have some knowledge of how the system works, because the tool is just answering questions about the system. In that regard, I think Gregg's DTrace book and the first edition of Systems Performance are both good in that regard.

I think Solaris Performance and Tools is still pretty solid too

Any trick to getting a 9p mount to work in a bhyve guest?

I think I have the bhyve process with the right provileges and it's picked up the virtfs setup correctly.

Shouldn't need a trick

The (Ubuntu 18) guest can see the virtfs device, but I get permission denied if I try and mount it.

It should appear in the guest as a PCI device, so see if it's there with lspci

ah, ok, I'll dig out the command I usually use

mount -t 9p -o trans=virtio,rw,uname=root,dirsync <share name> /a

Aha! So the trick is to add uname=root

Thanks!

Which OI/IPS package contains the kernel? onbld? osnet?

system/kernel?

system/kernel/platform has `unix`, the first mach-dependent bit of the kernel

hmm, well IPS then, while successfully completing update, is _not_ actually updating the kernel on OI right now

I seem to be stuck on this commit hash at the moment: illumos-e4f7ccbe1e

It also seems that most recent builds are only changing like 4 packages? hipster.openindiana.org/jenkins/job/illumos-gate/6862/console

I've seen various reports of update problems on OI at the moment. I think it's because the osnet-incorporation needs rebuilding after the obsoleting of system/library/processor since older versions of that have been purged.

damn, there's been a lot of useful fixes in smbd from Matt Barden that I'd like to test out. The bug reports sound eerily like some of the oddities we've seen

in particular I think we've hit the thread pool exhaustion one on multiple occassions for nfsd (though, that one isn't merged in yet)

aww, I can not test grub in this vm afterall... missing feature for read 'org.freebsd:zstd_compress'

andyf: Presumably it is not osnet-incorporation but rather one of their other metapackages

I didn't spend a lot of time looking but it appears that osnet-incorporation has a lock for system/library/processor⊙0

and I couldn't see that version of system/library/processor in the repository

but you know how IPS error messages are.. :)

I suppose my assumption is that they get that package from the gate build like everyone else

But maybe that is unfounded

It is, as far as I remember mostly auto-generated based on whether other manifests have the noincorp flag set

we often have no-incorped an obselete, for distro reasons

it is not in our package

so that seems like if not our fault, at least partially our fault

(I'm not sure how I have _not_ had this problem, to be honest, but I seem not to have?)

but that also means I'm not really in a position to fix it, if andy says noincorp is the way

[illumos-gate] 15037 cp_subr.s: Invalid instruction argument -- Toomas Soome <tsoome⊙mc>

I filed #15075 (fenix?), if people want bite-sized bugs

BUG 15075: cmd/initpkg/dfstab.sh generates a completely static file (New)

↳ illumos.org/issues/15075

[illumos-gate] 15041 smbadm join reports clock skew as bad password -- Gordon Ross <gwr⊙rc>

[illumos-gate] 15058 vmm: comparison is always true -- Toomas Soome <tsoome⊙mc>

[illumos-gate] 15042 AD join fails if hostname longer than 14 characters -- Gordon Ross <gwr⊙rc>

[illumos-gate] 15051 SMB2 signature mismatch gets wrong error -- Gordon Ross <gwr⊙rc>

[illumos-gate] 14690 nvmeadm should refer to namespaces by their nsid -- Hans Rosenfeld <rosenfeld⊙gho>

For the people wondering about osnet on oi builds. osnet incorporation is frozen inside the build VM, so that pkg can update every build without new be and not killing the machine with a new kernel.

We probably need to update that server manually and then the next build should be ok again

Although that server has 564d52366b according to mtod

can people switch the facet, and then have it go away?

since that would be equivalent to us having noincorp'd it

which again, I can't really fix, since none of this happened to me :\

probably. But I now doubt it was the update issue

I'll have to see if I can reproduce

today it allowed me to complete the update but it only updated like 700k worth of files. I'm still stuck on a much older kernel

[illumos-gate] 15069 dladm_str2bw() does not reset errno which is checked after string conversion -- Dmitry Gromada <dmitry.gromada⊙nc>

yeah and I am locked out of all updated systems from a mac via SSH

I can only zlogin in

Did we do a change to pty recently. Between september 2022 and October 2022 I get a self closing ssh connection

debug1: Setting controlling tty using TIOCSCTTY.

debug1: Received SIGCHLD.

after that it gets a debug1: session_exit_message: session 0 channel 0 pid 20275

Jenkins remoting works

so it's bash that closes

[illumos-gate] 15065 vmm: variable set but not used -- Toomas Soome <tsoome⊙mc>

zsh works so it's bash.... but why

and the shell interaction is weird

For those who are sufficiently old: I just installed `punchin` on a new VM. :)

.oO

.oO should we drop grub?:)

Is there a push coming up that'll break it? I ask because literally just today I had a customer report something on their GRUB-booting USB key. Knowing that's coming Will Be Important (TM).

no. but then again, *if* we are keeping it, there are something to be fixed

well, unless we tell that we do not really care if its broken...

Yeah... TBH I'd LOVE to get rid of GRUB, but it'll increase our support load to make sure people are moving off of it in cases where they still have very old USB keys that boot grub.

(our == "SmartOS & Triton", mostly Triton)

anyhow, obviously this can not happen overnight.

but yes, reboot -f does not do fast reboot with grub because libgrubmgt is not there any more (and do not really get that one back;)

something simpler can be implemented but ... anyhow, time to get some sleep:D

Get a good night's worth!

Any reason people have such old USB keys? Other than not wanting to upgrade at all

I think smartos stayed with grub longer than you might think

so I don't think they're _that_ old

if they use the smartos BE stuff, they're going to get an awful shock when the menu.lst thing happens though

I don't think we need to stop _working with multiboot_ to remove GRUB

Because we already also work with, e.g., iPXE and other multiboot loaders

And, yeah, SmartOS doesn't use the in-gate tooling to maintain the key contents

I think you could easily make the case for not producing any new GRUB-bearing USB key images while also not breaking existing SmartOS/GRUB users

What jclulow said ^^^ and @toasterson ==> Triton Head Nodes tend to be maintained entirely by Triton tooling which lets you Just Use The USB Key.

There are loader-booting USB keys now, but many are so long-lived they never bothered to move out from GRUB, never mind to disk-booting SmartOS (which is just like loader-booting USB keys except the "USB Key" lives on a zpool)

(specifcally a bootfs on a zpool.)

danmcd: I have some thoughts about transition from UFS to ZFS for SmartOS, btw; should I write them to you?

(we're using a ZFS-based ramdisk image at Oxide, and I think I can make it so that bits of it will work for you too)

Send me a mail, sure. (And hopefully my find-bugs-during-the-sideport gifts will help both of us!)

Thank you!

What are benefits for a ZFS based Ramdisk?

No Y2038 problem for one thing.

No 2038 problems

also in-place compression

and online expansion to a larger size

Ah, yeah thats a big one :)

e.g., you could ship a 200MB pool image that unfurls to 1GB at boot time

You can set quotas, create datasets, etc

Makes those iPXE times shorter for sure.

Also for SmartOS you could stop doing the lofi /usr hack

Just ship a flat image

By making /usr a read-only filesystem, right?

If you want!

whats the /usr lofi hack?

SmartOS does something a bit like an installer image, where there is a compressed lofi image shipped inside the ramdisk

which gets mounted over /usr

early in boot

gist coming (then I gotta run for dinner)...

oh, I thought it was one disk

It is a disk within a disk

It is, the lofi image comes from /usr.lgz

NOW it's dinnertime.

ah because needs to be RO? I assume?

it's actually to make it smaller

It being read-only has actually been historically inconvenient at various moments haha

(it's always read-only inside a zone)

on the host though we have had to contort ourselves at various times to patch things or add stuff

heh

I guess -Clzma is the best we can do?

(lofiadm -C lzma, that is)

hmmm I see ZFS starts to make a lot of sense

well, this is more the joys of small media

(and some smartos decisions)

it's basically the same awful tradeoff that caiman makes

I think alexander eremin actually did even more weird stuff for caiman

I suspect they should have just implemented squashfs or something like it for the install miniroot

Is there a guide how one can boot as ZFS Ramdisk image?

or create one?

Nope! It's all work in progress stuff

What I have so far is: unix and a cpio boot_archive that just contains modules needed to get the ramdisk, and then a new module hook just prior to vfs_mountroot() where we'll let you make a ramdisk available and choose to mount it

for SmartOS I suspect they would then have the ZFS ramdisk image in another multiboot module

(ZFS needs a few new bootprop things to communicate that we want to import a specific ramdisk /devices path uncritically)

for the Oxide thing we are reading a ramdisk image (with a small header) from s0 on an NVMe device

which we unpack into a regular ramdiskadm-style ramdisk

of a particular size, which can be larger than the actual image size

and then we adjust the boot props to do use that device as the root pool

and zfs autoexpand into the ramdiskadm ramdisk?

I thought it wouldn't be as easy as lofiadm->zfs inside and laod :)

yup, that's where illumos.org/issues/14978 comes from

→ BUG 14978: ZFS autoexpand property should work for root pools (New) | code.illumos.org/c/illumos-gate/+/2355

that would make me so happy, for image-builder, too

Yeah this is all in-kernel change because it's prior to the first process

richlowe: I just have to do the last round of testing I think and then autoexpand will work

hello experts, does any tested zfs write (most 8k size) performance vs xfs?

probably not in a great many years

I have a client did the tests with postgres, they stress postgres by standard TPC-C, that tool generate about 50% writes for file system by database WAL log; on zfs, they insert/update a bunch of data have spent 300ms, but on xfs did the same, they just only spent 30ms; disks are NVMe SSD

so I’m not sure if there are write performance issue with zfs

Any people had this here happen before? youtu.be/NxoW_lNy70k

toasterson: Aren't you missing the ssh -t flag

that helps a bit but it broke on upgrade

Also, in that video you're using zsh not bash?

no only gets me a terminal on zsh but still crashes bash

yep :) bash gets a SIGCHILD and then quits

hmm motd does not get displayed when calling command

ok so thats ok

and -t does the thing

now I just need to find out why bash quits

When does it quit exactly?

when it starts and before it prints the prompt

What about: ssh -t $host bash --noprofile --norc

--noprofile --norc also quits the same

That's odd

What if you run bash under "ssh -t $host zsh"

can you truss it etc