I have a VPS running debian right now. The provider doesn't let me to mount an own iso. Is there any way I can install OmniOS in this circumstances?

probably not easily,

my only crazy thought is whether you could get enough in a initramfs image to get the system up with network and a shell, then use that to stream an installed omnios image to the disk (but that feels like something that has lots of things to potentially go wrong)

if there would be netboot.xyz that possibly could load the install image...

szilard: a variant of what mlari suggests would be to repartition the virtual disk (reinstalling debian on a smaller slice first). But finding a different VPS provider that lets you supply your own iso might provide a superior experience in the long run.

Yep, i see that so too

(you'd then dd the illumos/omnios image to the unused disk slice and boot from that..)

I created a bhyve zone following the website instructions with the latest alpine iso and without vnc. Booting fails, giving the error "Could not set VM boot order:" and no other info except debug output. I've set the debug firmware and checked for every file referenced in the commandline, but it can't even boot into the EFI shell. What am i missing? litter.catbox.moe/znzl7jsvgl5a4ob7.log