-
Smithx10
HmM, booting up bhyve takes some times, should we use the bhyve memory reservoir ?
-
danmcd
Good idea. We'd have to change a few things, however. Enough for a PI change at least, and ideally we should plumb it all the way up to Triton.
-
Smithx10
I guess using the res would require a CN to be marked as a HVM ... not sure we can have zones + bhyve safely on that box....
-
Smithx10
or..... would you specify like 50% for VMs or something
-
danmcd
Smithx10: There's some design space to play with that. I can't remember off the top of my head how static-or-not bhyve's use of the reservoir is.
-
danmcd
e.g. if I ask for 16GB, but only 8's available, do I get 8, or do I get 0?
-
danmcd
I think it's time we plumbed it at least through SmartOS.
-
danmcd
(obvs not for this week's/tonight's-branch release. :) )
-
Smithx10
lol, nooooo.... i just bounced a machine and waited.... honestly it doesn't really make life too bad
-
Smithx10
more just annoying lol
-
danmcd
Still... I think being able to have a BHYVE zone request reservoir usage, esp. if the behavior is best-effort not all-or-nothing might be useful.
-
danmcd
One thing that COULD be done is to have reservoir be set on a CN to sizeof (BHYVE-VMs-on-CN) with the understanding that a new BHYVE VM might not succeed using the reservoir until next CN reboot.
-
danmcd
I RTI-reviewed it, but it was 1.5years ago.
-
Smithx10
ahhh nice
-
Smithx10
lol someone gave in and used packer on illumos :P
-
Smithx10
-
bahamat
Yeah, we're shifting our HVM image generation to use packer rather than a completely bespoke system that few people understand.
-
bahamat
It's basically going to be packer+ansible, two systems that are well documented outside of Triton. So that should be much more approachable by anybody wanting to either contribute or run their own builds.
-
jperkin
..if I can actually get it working ;) for some reason it's not opening plugins even building them manually, still debugging..
-
pmooney
danmcd: if you set an instance to allocate from the reservoir, it will fail (immediately) if there is not space available
-
pmooney
there is no fallback behavior today
-
pmooney
the assumption being that if you choose to use the reservoir, you're already making placement decisions which you believe should be successfully fulfilled
-
danmcd
Yep... post 1125amET inspection of bhyve, vmm, and even propolis have educated me. :)
-
pmooney
and any memory held by the reservoir is wholly unavailable to the rest of the system
-
pmooney
while there is a tentative guard rail for its upper size limit, it makes no guaratees about preventing one from wedging the system by requesting an excessively large reservoir size
-
pmooney
like if you have a bunch of pages tied up in i40e or whatever, it has no visibility into that fact
-
pmooney
one upside though, is that it's fine to resize the reservoir at runtime
-
danmcd
80% is the absolute top-off AIUI.
-
danmcd
Funny you mention i40e there. :) :) :)
-
jbk
heh
-
pmooney
yes, but 80% will absolutely wedge some (perhaps even many) systems
-
pmooney
it was hard to strike a balance between ensuring safety without being overly constraining on large-memory systems
-
danmcd
Yeah... understood.
-
Smithx10
danmcd: do you have the gist for converting the HN to a bootable pool?
-
Smithx10
Got 3 HNs I gotta get off USB lol
-
danmcd
Are your USB keys loader?
-
danmcd
If so it should be easy. If not, your first step is to convert the USB key to loader. This is not straight-from-GRUB-to-pool-boot that doesn't involve surgery.
-
Smithx10
I think I already did the conversion of the USB
-
Smithx10
Just required a more recent platform to boot in order to convert the pool
-
Smithx10
Finally got the HN on that platform
-
danmcd
Are you gonna boot off `zones` or boot off of a dedicated zpool?
-
Smithx10
zones
-
danmcd
And zones is not multi-vdev? (single raidz is a single vdev, single mirror is a single vdev...)
-
danmcd
(And note, I disappear in 30mins.)
-
danmcd
`zpool list -v ` is your friend here
-
Smithx10
-
Smithx10
actually looks like they are all bootable already.... might have already done this lol
-
Smithx10
think I just need to add the latest platform to them and remove the USBs
-
danmcd
`piadm bootable` is your best diagnostic here.
-
danmcd
`piadm list` is not necessarily reliable on HNs.
-
Smithx10
-
danmcd
Okay, as long as your CNs are EFI booting looks like you just need to:
-
danmcd
1.) Boot off the disk.
-
danmcd
2.) Update the pi using `sdcadm platform assign...`
-
danmcd
YOU DO NOT USE piadm(8) save for 'bootable -e` on HNs.
-
danmcd
3.) Reboot HN to newly-assigned PI.
-
danmcd
Missing 0: "remove usb key"
-
Smithx10
yea
-
Smithx10
I think tho if I cant get someone in there.... and I change the boot order
-
Smithx10
I could use cfgadm to disconnect the USB devices
-
Smithx10
so sdcadm platform doesnt see them*
-
danmcd
If you boot off the disk, `sdc-usbkey` will treat $BOOTPOOL/boot as the USB key.
-
Smithx10
even if there are USBs plugged in?
-
danmcd
-
Smithx10
nice
-
danmcd
You must boot off the disk.
-
danmcd
It sets an extra variable in bootparams that lets sdc-usbkey know WTF it's doing.
-
danmcd
Kebecloud's HN has a dedicated boot pool SSD, as you can see from the GIST.
-
danmcd
If you boot from disk AND the usb key is avaiable, you must invoke with `-u` to have sdc-usbkey try.
-
danmcd
e.g. `sdc-usbkey -u mount `will not mount $BOOTPOOL/boot, it'll actually seek out a USB key.
-
danmcd
If it can't find one (like on my Kebecloud HN) it'll spew:
-
danmcd
sdc-usbkey mount: error: no pcfs devices found
-
danmcd
`bootparams | grep triton_bootpool` ==> No output means you booted from the USB key OR SOMETHING IS REALLY FUCKING WRONG WITH YOUR BOOT POOL. :)
-
danmcd
Disk-booting output will look like:
-
danmcd
triton_bootpool=bootpool
-
danmcd
(or in your case: triton_bootpool=zones )