Is there a clever way to install OmniOS onto a rpool that doesn't allocate the whole disk - so that another slice can hold a different pool?

I'm not sure, but you should be able to create pool manually and install on existing pool?

altho, it would be nice if installer would help you there;)

tsoome: thank you. I figured it out.

I had to let the installer eat up the whole disk to get the s0 EFI contents. s1 for the rest with rpool in it. Then I destroyed the rpool, used format to set slice s1 to old/same start but size only 150g. recreate rpool, enable compression, let the installert just use existing rpool. works + boots.