13:28:27 <megaTherion> Hi, Im looking for some idea/help regarding OmniOS. I tried to boot it on a machine which I like to use as VM host yesterday but it halted after CPU/USB initialization.
13:29:04 <megaTherion> I tried an older image I had somewhere, this one worked immediadly, the one which doesn't work was "omniosce-r151036.iso" - the older image is "omniosce-r151036.iso"
13:30:12 <megaTherion> I googled and found an older thread regarding a hanging boot on an Intel machine from 9 years ago. But the flags being proposed to add to boot command didnt really help.
13:30:51 <megaTherion> err, correction the image which worked to me was "omnios-r151052b.iso"
13:35:13 <m1ari> 151052b is the latest release so if that's the one that worked then that shouldnt be a problem
13:36:54 <megaTherion> I see, yes you're right indeed. Maybe I was confused with the 'ce' because the community edition was mentioned explicitly named in the docs. But nowadays its just "OmniOS" or are there two seperate releases?
13:37:42 <m1ari> if you're trying to boot on physical hardware I wonder if it's also worth trying to use the usb-dd image rather than the iso
13:38:22 <megaTherion> ah ok there are two seperate images as well? I usually use something called 'balenaEtcher' and simply use those images to flash (it works very well usually)
13:39:25 <megaTherion> and another thing I would be interested in is bhyve, how on par is it with the current FreeBSD implementation? Because from what I understood kvm is deprecated and will never be updated again right?
13:39:26 <m1ari> i think the ce was there historically to distinguish from the older omniti images
13:40:23 <m1ari> I've not made use of bhyve so don't know much about it and what it supports
13:41:27 <megaTherion> ls -l
13:41:29 <megaTherion> oops
13:41:44 <megaTherion> ok then I'll just have to test if bhyve on OmniOS works like on FreeBSD
16:55:36 <andyf> megaTherion - bhyve is pretty much on par with current FreeBSD. It's synced periodically.
16:55:47 <megaTherion> andyf: cool, thanks for the confirmation
16:56:10 <megaTherion> as I understood there was no way to get KVM more current due the ongoing changes which are very heavily relying on Linux (kernel) ecosystem
16:56:11 <andyf> Our kernel implementation is a bit different these days, but then we're a different OS so it's not entirely surprising. The userland pieces are fairly similar still.
16:56:46 <andyf> Right, there are no plans to update KVM now.
19:42:53 <megaTherion> ok now Im trying to install OmniOS and I've two NVMe's I want to use, creating a pool fails. I guess its  disturbed that there is still a gpt disklabel and all on those
19:45:21 <Woodstock> you can try to clear them with the "zpool labelclear" command
19:48:02 <megaTherion> no dice
19:48:29 <tsoome_> fails how?
19:48:54 <megaTherion> "failed to clear label for /dev/rdsk/....."
19:49:12 <megaTherion> but I get warnings that the label is corrupt, they had GPT and some btrfs on them I think
19:49:32 <tsoome_> and zpool create is failing how?
19:51:40 <megaTherion> its too fast gone... cant read it - what's OmniOS tool for GPT labeling disks?
19:52:14 <tsoome_> you can use format command for that
19:52:49 <tsoome_> format -e, then select the disk and then enter verify to see what is there
19:56:25 <megaTherion> ok, cleared the label with parted on Linux... now Im back on OmniOS shell, format -e / verify gives me "Cant read label"
19:56:31 <megaTherion> installer still fails with the same issue
20:00:05 <megaTherion> I tried labeling with format -e / label (select EFI label) this worked. Now I still need an ESP I guess
20:15:53 <tsoome_> if format agrees you have correct label, you can use zpool create -B to create whole disk pool and ESP for you.
20:16:51 <tsoome_> or use installer:)
20:17:44 <megaTherion> the installer fails, I'd really like to use it :D
20:18:07 <megaTherion> and since Im (yet) unfamiliar with OmniOS commands (my OpenSolaris knowledge is like 10+ years ago)
20:18:13 <tsoome_> does it fail even with fixed gpt label?
20:18:22 <megaTherion> yes
20:18:45 <megaTherion> I want to try from shell, but I need to figure out which device/partition it is 
20:18:53 <tsoome_> ou, ok, thats interesting.
20:19:47 <tsoome_> well, for whole disk setup you only need disk name without /dev/rdsk, and use zpool create rpool -B c...d0 -- thats d0 at the end of the name.
20:19:59 <megaTherion> so the disks get detected as c3t001B448B4AF180B4d0 and c6t6479A789800000CDd0
20:20:24 <megaTherion> format is displaying them and also the correct names (one is a WD 550 blue something and the other is a Crucial nvme)
20:20:36 <tsoome_> that should create GPT with 256MB ESP on s0 and then most of the space in s1 for rpool and 8MB slice at the end.
20:20:59 <megaTherion> ah ok so zpool will also create the ESP, didnt know that
20:21:26 <tsoome_> thats -B
20:21:57 <tsoome_> without -B, you will just get s0 and s8 (8MB)
20:22:59 <megaTherion> oh now I get an error, "Boot pool can not have more than one vdev"
20:23:28 <tsoome_> yes, if you want to present both disk to it, you probably want to use mirror
20:23:32 <megaTherion> (because I wanted to have those in a striped configuration)
20:23:40 <megaTherion> nah mirror I didnt want (I've backups elsewhere)
20:23:43 <tsoome_> so its zpool create -B rpool mirror disk0 disk1
20:24:34 <tsoome_> yea, boot pool is special there, it can not have multiple vdevs
20:24:47 <tsoome_> it can be single disk, mirror or raidz
20:26:00 <tsoome_> there was some work to fix this, but afaik it is still not completed.
20:28:29 <megaTherion> ok I cheated, maybe it works - I just added the other disk afterwards to the pool
20:29:19 <tsoome_> it will not. you can install it, you can boot from it, but kernel will panic.
20:30:34 <tsoome_> botloader can read stripe, but kernel rootfs mount will fail because it has no idea where that second disk is.
20:30:52 <megaTherion> thats sad :(
20:31:03 <megaTherion> but I've a bigger problem now, it hasn't detected my NIC :D
20:31:26 <tsoome_> eh...
20:36:12 <megaTherion> well RTL8125 2.5gbps
20:36:18 <megaTherion> indeed not an Intel chip, so it sucks
20:41:24 <megaTherion> ok I've some other Intel I225-V 2.5gbps card around, I'll just drop this one in it
20:47:40 <megaTherion> tsoome_: btw. booting the striped rpool works, would be suprised if not - if it works in a sane way then the kernel shouldnt bother where to find that damn vdevs
20:47:49 <megaTherion> works on FreeBSD too
20:58:09 <tsoome_> well, your blocks will be spread on both vdevs, yes loader can walk through the disk list (which also has limits),  but kernel needs to get boot pool "imported" and root dataset mounted, and for that it needs to find out and build in memory pool config.  from disk label it can collect vdev config, but to find next vdev, it needs to browse the disks (as loader also does).
21:00:17 <tsoome_> however.....  I just did realize we could pass device names from loader to kernel, so  the kernel can have a bit easier  work, but even that does not work for 100%, because if device names are changed, the kernel still needs to scan disks to find out the new reality.
21:16:08 <megaTherion>         rpool                    ONLINE       0     0     0
21:16:09 <megaTherion>           c1t001B448B4AF180B4d0  ONLINE       0     0     0
21:16:09 <megaTherion>           c4t6479A789800000CDd0  ONLINE       0     0     0
21:16:18 <megaTherion> well it seems to work, maybe I get crashes later who knows :D
21:16:57 <tsoome_> ou, 
21:17:03 <tsoome_> thats ok boot?
21:17:06 <megaTherion> yes it boots
21:17:48 <tsoome_> let me check if we have any notes around...
21:18:11 <megaTherion> thats why I couldn't progress with the installer, because I chose both disks and striped configuration (which was a valid choice) but I got the zpool error
21:23:06 <tsoome_> um, jclulow was some corner case left with disk device scan or did we just forgot to lift the multi vdev restriction on boot pool?
21:24:26 <jclulow> tsoome_: It would be more accurate to describe the thing that now works as a corner case
21:24:33 <jclulow> i.e., single vdev pools
21:24:45 <jclulow> see also the notes in https://www.illumos.org/issues/7119
21:24:46 <fenix> → BUG 7119: boot should handle change in physical path to ZFS root devices (Closed)
21:25:17 <tsoome_> ok, so we can drop this restriction.
21:25:34 <jclulow> Which restriction exactly
21:25:55 <tsoome_> multi-vdev boot pool is not allowed atm
21:26:58 <jclulow> oh, booting from a _stripe_?
21:27:13 <tsoome_> yes.
21:27:40 <megaTherion> I just did pfexec pkg update & reboot, still works
21:27:44 <jclulow> I don't think I would ever recommend doing it, but I can't immediately think of a reason it should be banned per se
21:38:37 <megaTherion> /dev/rdsk/c4t6479A789800000CDd0: NVMe devices are not supported in this version of smartmontools :(
21:39:47 <tsoome_> cd ..
21:39:51 <tsoome_> oops:D
21:45:41 <tsoome_> https://code.illumos.org/c/illumos-gate/+/3874 
21:45:42 <fenix> → CODE REVIEW 3874: 16983 zpool: we actually do support multi-vdev boot pool (NEW) | https://www.illumos.org/issues/16983
21:48:55 <megaTherion> btw. are there pkg mirrors in europe?
21:59:51 <jclulow> I think the primary pkg server is, itself, in Europe
21:59:57 <jclulow> It's why we need a us-west mirror haha
22:01:03 <megaTherion> ah I see
22:06:09 <megaTherion> well now I'll tinker with zones a bit and have to do a lot of reading
22:23:52 <megaTherion> I created a lipkg branded zone with zadm and added allowed-address / defrouter to the config... it installed but I've no route. So the defrouter config is somehow not effective