Moving forward, I have OmniOS w/bhyve running OI guest now. I changed lightdm's config so it runs headless since bhyve doesn't give a graphics device. I can use XDMCP to get the desktop via Xephyr, or even VNC, but when I try to login with known-good name/pass it just resets. Any ideas on debugging this?

I suspect Console kit. There's some Glib-GBoject errors in the logs but maybe that's a red herring.

VNC to the bhyve gives command line login fine. I updated to whatever was latest with pkg update yesterday.

This is what it looks like, left window just resets on login attempt. atr2.ath.cx/~jayjwa/XDMCP.png

Another interesting error from the greeter: solaris_xserver_cred: could not open /var/dt/sdtlogin/1 (looking in /var/log/lightdm)