07:46:46 I tracked the source of the error down with some printf debugging in the boot script: Apparently set_bootorder() silently fails, because bootmap and rbootmap aren't being properly initialized, because the default for self.vars is empty, but i don't know what it exactly it's supposed to be. Disabling uefivars gets past the error, yet leads to bhyve silently silently exiting and halting the zone once again. What is the proper way to 07:46:47 debug/trace a branded zone? 09:47:20 Let's see if i got this right: The website instructions say to boot the zone, then connect using zlogin -C; the current boot script sets the option config.dump=1, meaning it only creates a configuration file and then exits, so zlogin connects to an empty console. I got bhyve to work for now by modifying the boot script, but i really don't see what's the intended way to launch the vm. 10:51:32 luxhalit`: is this a bhyve zone? 10:52:35 maybe the https://man.omnios.org/bhyve.7#bootrom can help, have you tried to use the debug version? 10:52:57 yes to both. system/zones/brand/bhyve 0.5.11 10:58:48 The system actually boots and i'm currently just having trouble setting up the exclusive ip zone. What's supposed to happen after the boot script makes bhyve dump its config, so i can access the vm without hacking it? Is there a service that needs to be activated, a different login program, a zone option i've overlooked? 11:21:33 luxhalit`: what kind of OS is this? 11:21:40 I think i've found the answer in dmesg: "init(8) for zone exited with status 4". Where do the logs for zone inits go to? 11:21:42 the guest, i mean 11:22:14 Alpine. Bhyve works, but the zone doesn't for some reason. 11:22:53 /zones/bhyve_zone_name/log/zone.log 11:23:38 No, there's nothing about init in that one. Only the boot script, which works fine. 11:25:13 It's root/tmp/init.log, which contains 'vm_open: No such file or directory'. 11:33:58 Apparently it can't access a file under /dev/vmm (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=265444), but the node itself is there. The vmm module is loaded, so it can't be that either. Next i'm going to try livepatching init to call the script with truss. 12:19:01 What is the guest? There is one guest (9front) that we know will exit with status 4 if the bootrom variable store is enabled. 4 is just what bhyve uses for "something went wrong" and it usually has a message along with it in root/tmp/init.log 12:19:11 Try setting the `uefivars` variable to false in the zone config 12:19:12 The bhyve zone doesn't support the 'boot -i' option, so i've had to substitute the brand init. The trace has been running for several minutes... 12:19:54 The message in init.log was 'vm_open: No such file or directory'. 12:20:35 Ah, ok, did you previously start the VM in the global zone? If so then the zone will not be able to access it until you destroy it. 12:20:56 andyf Yep, that resolved the 'cannot set VM boot order' isssue. 12:20:59 With `bhyvectl --vm --destroy` 12:21:19 I've uninstalled and reinstalled the vm several times by now. 12:21:44 The vmm instance could still exist, owned by the global zone, unless you did anything to clean up. 12:21:47 Is the boot script launched from the global zone? 12:23:02 Yes, it creates the VM configuration in the zone root/etc/bhyve.cfg, then the zone init starts the VM 12:23:45 Stepping afk for 10 minutes or so, but I'll be back after 12:23:57 I will wait for the trace to finish first. 12:38:36 This was it. Nice! 12:55:40 We should find a way to make that more obvious, I'll see what I can come up with. It's part of the zone security that a non-global zone cannot do anything with a vmm instance created in another zone. 12:56:01 and I'd like to understand the original "Could not set boot order" problem too. 13:08:05 As far as i understood, the default efivars configuration will cause all boot devices to be masked and this appears deliberate. Maybe add a hint to check efivars, if at the end of set_bootorder() order is empty. 13:13:25 Since failing to access an already created VM appears to be a hard to diagnose issue, i would add a check at the end of the boot script. Is there a simple way to retrieve the zone of a VM? 13:15:46 I don't know offhand, but I would expect there's an ioctl() that the GZ could call. That's a good idea. As a minimum I want to integrate something like https://code.illumos.org/c/illumos-gate/+/4591/1/usr/src/cmd/bhyve/common/bhyverun.c 13:15:47 → CODE REVIEW 4591: 1xxxx bhyve: more help for non-bootable VMs (NEW) | https://www.illumos.org/issues/1 13:46:29 Just managed to install OmniOS to a VPS. The provider was able to attach the iso on my request. 13:49:06 MY issue with the non-responsive keyboard/mouse in bhyve Windows VM still exists. The keyboard input works in the UEFI shell and in the Windows bootmenu, but I can't do anything after the gui starts. 13:49:25 type is already windows, so lpc is 31. xhci on/off makes no difference. 13:49:40 Mouse input works in other VMs via VNC. 21:23:17 I am fighting with zrepl right now. 21:24:27 I use ssh+stdinserver conntection type (eg trough ssh). While I can manually connect to the backup server using the freshly created ssh keys, zrepl says: 21:24:56 error listing receiver filesystems err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing ssh (exit status 255)\\nLoad key \\\"/root/.ssh/backup-id_ed25519.pub\\\": error in libcrypto\\r\\nroot⊙bec: Permission denied (publickey).\"" errType="*status.Error" 21:27:18 It was 600. If I set it to 644 I get: 21:27:20 WARNING: UNPROTECTED PRIVATE KEY FILE! @\\r\\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\\r\\nPermissions 0644 for '/root/.ssh/backup-id_ed25519.pub' are too open.\\r\\nIt is required that your private key files are NOT accessible by others.\\r\\nThis private key will be ignored.\\r\\nLoad key \\\"/root/.ssh/backup-id_ed25519.pub\\\": bad 21:27:25 permissions\\r\\nroot⊙bec: Permission denied (publickey).\"" 21:41:40 This "Permission denied (publickey)" feels like it is cused by sshd, or similar 22:36:11 the private key may be invalid 22:39:56 I'll recreate the keys tomorrow. 22:40:01 Thanks for the hint. 22:40:05 Good night