how does bhyve handle it if we oversubscribe cpu cores? let's say i have 16 cores and i give 8 cores to 3 vms. what happens?

dch: pretty goood

demido: probably run into a spinlock issue between instances against CPU cores

varsis ya that's what i was thinking. like a contention fight. what about if i have 16 cores, can i give 8 to 1 vm and 8 to another vm, leaving 0 for host system, and be ok?

na, you'll want to go n - 1

leave one core for the host which isn't contended with at all by guests

hm dang. ok i don't really know how to apportion cores varsis because when i divvy up cores, the system sits mostly idle except when a guest is doing lots of work. not sure how to word it better. so most of the time my vms sit idle so i could give them just a few cores. but then sometimes they get a surge of traffic and peg those few cores

so i either give vms few cores so there's no overlap but when they get work they peg their cores, or i give them more cores and there is overlap so they have more cores to use when they have work to chew through

is there anything wrong with giving odd number of cores to bhyve vms?

demido: yeah, that's the issue with hard allocation of resources. Depending on your wants and needs, using alternative solutions (i.e. Jail, using bastille optional resource limiting) would allow for a more dynamic allocation of resources (with limits).

demido: Nope. You may want to look at cores, sockets and threads, not just cores. I'm not a bhyve expert at all, though have used it a bit

so if i have a system with 64 cores and i want 8 equal vms, i can't give them each 8 cores even though they're rarely using even half of those 8 cores, because it would leave the host system with 0 dedicated cores?

Yes, there's a potential scenario of spinlock contention, you would need to leave one core for the host to avoid this scenario

ok, so if i wanted equal sized vms (doing the same workloads) i'd need to give 7 cores to each of the 8 vms, so 56 cores given to vms, and 8 cores left dedicated to host system?

There's probably some finer tweaks/combinations using the other options (sockets, threads AND cores) that may allow for more resource allocation closer to the 63 core limit you're looking at, though you would need to delve into the documentation around how those config options work. Otherwise, easy mode -> you leave resources to the host (as you outlined above)

Your guest VMs lose out on some resources though

ya i read the bhyve man page but it didn't elaborate too much on the package : socket : core : thead topic a whole lot

ok well i'll do what you suggest and assign <= n-1 to vms

ty

nps

feels weird to use odd number of cores :P

there is a bhyve channel on this irc network btw

ya it's always dead tho seems like

ah, bummer :|

so when ppl use bhyve to run a vps company, and i buy a vps with 2 cores, that's 2 dedicated cores and they don't/can't oversubscribe?

From my understanding (which may be completely without merit), yes :D

have you oversubscribed vms before and actually seen it cause trouble?

Nope, that's my understanding of vCPU allocation and hypervisors in general

"Don't do it"

You can hang the host system completely

if i back a bhyve vm with a sparse file and write 100GB into the vm, it'll show du -hs on the host of like 120GB for the 100 + OS data. then if i delete that 100GB, and write another 100GB into the vm, it shows more than 120GB in the host system being used. does freed space in a sparse file not get reused/reclaimed and it just grows forever or?

demido: I would assume it would be dependent on the FS being used (UFS vs ZFS vs x), there may be some sort of backup/snapshotting behaviour, cleanup await or similar. Not sure tbh

ufs

Sparse files are files that have been extended without having data written to the middle of the file. Therefore they don't consume block storage space for the empty sections in the middle.

When data is written to those empty regions then they are no longer empty and the file consumes space.

but i check their size with du -hs so isn't that real usage?

so empty spaces are filled in first?

First? No. Filled in as they are written.

Let's assume you have a file of 5 blocks just for an example to work through. I create it sparse allocating 5 blocks but not writing anything. It consumes zero blocks of actual storage.

ya but did you read what i wrote? i write 100GB into vm, then delete it, then write 100GB into it again, and seems like usage grows but maybe i'm wrong?

Let's number the blocks 1, 2, 3, 3, 4. I seek to block 3 and write one block. It now consumes one block of disk space.

Sorry: 0, 1, 2, 3, 4

Now that file consumes the block. If I write to block 2 then it consumes 2 blocks of storage.

If in the VM you delete the file from the file system that does NOT recreate the sparseness of the backing store file under the VM.

Deleting a file in a VM file system simply marks those blocks as free in the VM file system.

so in the vm i delete the 100GB file, then write the 100GB file again, and the host system might now have 200GB used right?

Now what I don't know is if TRIM can be used as it is in other VM systems to re-sparse out those files.

No. Why would it do that? If the VM file system is 100GB and you write 100GB then it will consume 100BG. Then free it. Then write it again and it would rewrite on top of the 100GB that is available.

well i give 1TB sparse file to vm so it thinks it has a big disk

The file system in the VM thinks it is writing to a disk block storage device. Because it is all a virtual world. It's on a holodeck.

If you allocate 1TB as a sparse file then the VM thinks it has 1TB to use for the file system. It might allocate blocks from any part of it at any time.

ya that's why i think it keeps growing (up to that 1TB)

so i should size the sparse file as close to what i want to limit the VM to without being too tight

For spinning disks the file system usually allocates an interleave across the entire platter so as to leave blocks in between available for a quick allocation later in other files. I don't think that is modified when the file system knows it is on an SSD since that just doesn't matter there.

Yes.

what % is safe to fill up to of allocated disk space? like if i give 1TB sparse file i shouldn't have the vm using more than how much at any given time? 900GB?

And remember that the main advantage of sparse files is that they are created immediately without waiting to actually write the blocks. Creating a 1TB sparse file is almost instantaneous. That's the advantage. It is not to save disk space. Because at any point in time the VM might balloon to the full size and the hosting system must be able to supply the space.

If you allocate space to a VM you must expect that the VM will use it at some point in time. Otherwise why allocate it to it?

If you only want a VM to consume 100GB then only allocate 100GB to it.

but isn't it bad to fill 100% of a disk?

the disk in this case is the sparse file backing the vm

in all cases running out of disk is bad

you just have to decide if its worse for a VM to consume all space on a server & render every VM and jail and daemon inoperable

or if you prefer to have a single VM full

with zfs you can do a reservation that prevents the user form doing this, and leave it to root to relax the reservation

I prefer to not over-commit VMs on disk, filesystems run better when kept under 75% in general

but I have run accidentally right up to the limit, then you risk downtime to clean up

this is a good summary, but the web page needs an adblocker linuxhaxor.net/code/setup-zfs-quotas-reservations.html

ok i'll make sure to make the sparse file size such that the vm can use the space i want it to have + 25% more ty dch. i'm not using zfs but i bookmarked that for when i do

i'm running all this on ufs on hardware raid rn

feel free to come back here when you're ready and ditch that h/w raid

ya i'll get just an hba card in it right?

there would be very few storage people I know, probably none, who would recommend it, if you have zfs or ufs+gmirror available on FreeBSD

demido: one thing at a time. but basically you can flip most h/w raid cards into JBOD (just a bunch of disk) mode.

then FreeBSD will see 2 disks, and you can then mirror your data across to them

i'll get a new box soon so i can choose if i want raid card or hba card

with hot swap sas drives

fair warning, I have no idea if this is possible without erasing/losing all data on the raid card when it switches

demido: how much data are you talking here roughly

not sure a few tb or so

maybe 5ish

usable

just wanna get a 2u for 2.5" sas drives

ok thats quite practical to copy/mirror/sync easily then

used tho

I've dealt with arrays that take over a month for a full mirror to complete

wtf like raid 6 or something? i run raid 10 rn

a little bigger .. you need a special datacentre floor and custom power.. HP StorageWorks EVA, XP24000 and similar.

in my day they were smaller, now they run to 300+TB per rack, up to 250PB for a maxed out install.

damn. now do enterprise scales like that also use zfs instead of hardware raid or?

they provide the vdevs to use zfs on top, but they have special ways to slice them to provide more robust performance guarantees

wow

EVAs are midrange. Also not very good anymore, unfortunately

The XPs are pretty good. Mainly because HP doesn't make them. :°)

They're rebadged Hitachis with a customized firmware and management sauce, basically.

Mmmm, old school Hitachi drives were noice

is it bad to give a vm odd number ram just like odd number cores? like give a bhyve guest 3 cpu cores, and 15gb ram

demido: bad in what sense? if you are trying to balance the number of bhyve vms with the number of physical cores, it would require some understanding of how much utilization is occurring in each of the vms for how long. As for the numbers, mentioned, i am a personal fan of powers of 2

just the numbers

ya i am too but some say odd numbers of cpu cores and mem is ok

no, it doesn't matter really

hi, I upgraded FreeBSD from 14.1 to 14.2, and now the machine won't boot anymore. I can connect to a serial console and I see a kernel panic: okturing.com/src/22685/body

1. is this a know issue/can I do some workaround?

2. how can I boot off the old boot environment? when I boot, I see the new kernel booting directly, no prompt. I can interrupt it and then I am in some sort of console that says "OK". Can I boot the old BE from here?

I made it boot.

it's this bug: bugs.freebsd.org/bugzilla/show_bug.cgi?id=281177

i see about 10W of power associated with "uncore" using powermon, regardless of activity. with the same processor (Xeon E-2388G) on a linux system, uncore is almost always 0W unless i'm using the integrated GPU, and even then is rarely over 3W. any guess as to what might be "on" and consuming power?

so i'm prepping to upgrade a 13.2 server to 14.2, which has thick jails. any tips, warnings, or a good task sheet?

I would make a stop at 13.4 first.

I wonder, is there anyone who works on Podman on FreeBSD here in this channel?

Not sure which people are involved.

$ cd /usr/ports/sysutils/podman && make maintainer

dfr⊙Fo

make -C /usr/ports/sysutils/podman maintainer

:p

CrtxReavr: not sure, can't do package upgrades anymore... i could try to stop all the jails from starting and do 13.4 first

Right-o :P

Though there are a bunch of other components involved, it's not just the maintainer

The maintainer could probably answer your questions though.

I take it dfr isn't in here?

many aren't

#freebsd-ports is a thing.

I'm well-aware, just checking ^^

As is #bsdports on EFnet

CrtxReavr: was about to join the former. Forgot about the latter, TBH.

CrtxReavr: so upgrade/migration issues are more likely to be address in 13.4, than shipping with 14.2?

Demosthenex, as a general rule, yes. . . though I've been advised there's some specific issues with the 13.x to 14.x jump.

it's not really good idea to jump so far

I've not had issues with big version jumps in the past.

I think I've done src upgrades from 6.x to 10.x

i mean, for the base os, barring any hardware compatibility changes, i'd expect it to be pretty smooth

well you could in theory do 2 majors

the thick jails... i'm going to have to look at

or 3

Honestly, it really just boils down to what parts of the OS are being changed in the version delta.

or in your case hmm more

yes

generally 2 is ok

Though. . . RELENG works very hard to make upgrades feasible.

does it stand to reason that the thick jails at 13.2 will operate when i upgrade the native to 14?

i figured 13 is getting old, may as well goto 14

well 14 runs 13 binaries so

but who knows what happens

Though as always, read /usr/src/UPDATING

the binary compat goes quite far back

and you can extend it iirc

Raise your hand if you mostly just do the upgrade, and only look at UPDATING if there's a problem. . .

_o/

hah

it depends

What? I would never do such a thing! What psycho maniac would not read UPDATING, am I right?

*eyes around suspiciously*

it's also same in ports

can't be arsed to look changelogs

but (sometimes) you should

ideally if there were a problem, i can just roll back to my bectl snapshot...

I do for some things like DBMS'

(e.g. postgresql-server)

any ocaml coders think they can make a relatively simple package compile on FreeBSD? savonet/ocaml-mem_usage #8

we can get liquidsoap building on freebsd again