one question for CPU topology. the physical machine is 2 socket, and 20core/40 threads per socket ; if create a bhyve VM with 16 vcpus, then which topology is better: 1.) just set vcpus=16, 2.) set sockets=2,cores=4,threads=2 3.) set sockets=1,cores=8,threads=2 which set is better for performance, anyone could point me? best wishes

tozhu: are you constraining the guest to particular physical cpus? which one is better depends on communications and memory access patterns within the workload. spreading across sockets may have more total memory bandwidth; keeping all threads on one socket may mean lower latency for interprocess synchronization.

sommerfeld, do you mean the option 2, will allocate 4cores for each phy sockets? and option 3 will allocate all cores on same sockes ?

thank you very much

I'm actually not sure what bhyve will do with that configuration information (whether it lets you reserve specific physical cores/threads for a guest or not). I was under the impression that the default config an N-cpu guest gets N threads on the host that get scheduled by the host kernel.

copec here's a tip: suggest the change during an emergency. "OH NO THE DHCP IS DOWN? I THINK DEBIAN'S APT IS HAVING DEPENDENCY ISSUES!" and then install OmniOS and setup DHCP... never have issues... and step by step (or rather, emergency by emergency) half of your infra will be running <insert favorite technology here>

some of that config is no doubt what you tell the guest VM about the machine topology.

sommerfeld that is absolutely true

I use sockets=,cores=,threads and CPU pinning every day at $work

and during our benchmarking we've noticed differences with/without s=,c=,t= and with/without pinning

specially if it's a large VM

antranigv: only way to know for sure what's best is to benchmark each config (and look at performance counters).

I have a post about that: antranigv.am/posts/2023/10/bhyve-cpu-allocation-256

as a matter of fact, if you're allocating a large number of CPUs (240 in our case), then the only way to make it work is to use s=,c=,t=

sommerfeld what's even more interesting, is that the booting process gets almost 2x faster (Linux guest) if you do CPU pinning.

and that happens only with Linux. without CPU pinning it's just slower at boot

I tried FreeBSD as well, the speed is always the same

another fun fact: if you boot Linux on bhyve with more than 1TB of memory then the Linux guest will ignore the first 1TB of memory and choose the rest, aka allocate 1.2TB, Linux sees only 0.2TB

documented here: bugs.freebsd.org/bugzilla/show_bug.cgi?id=276463

I’m reading news.ycombinator.com/item?id=39178521 Its amazing… I didn’t realize I was that old

copec huh? :D

Generations have grown up with “the cloud” and fundamentally don’t understand what the infrastructure is

thanks sommerfeld antranigv

copec oh yeah. I'm very young (late 20s) but I feel very sad about my generation. Most of them have never setup a "server" at home. And they look weird at me that I run a home server. :/

copec I showed my OmniOS and FreeBSD infra to a friend, which is built with Zones and Jails, and he told me "why not just kubernetes on AWS?"

sommerfeld, tozhu: It's just lies that it tells to the guest. Unless you do something specific in the host it won't make a diference.

rmustacc: thank you very much

antranigv: i you tell him because you value you sanity (and cash) ? :)

One of the upside listening to Oxide and friends is that i now read some messages in the persons voice 😅

sjorge: the thing I find fascinating about their (great, great) podcast is that it does absolutely everything but actually sell their product

jclulow can you share the "helios powered office clock" code? I understand that you're using /dev/kmem, but for the love of god I do not understand how one would do the implementation :)) cheers!

antranigv: okay, what? (don't really have N hours to listen to the podcasts right now..).

sommerfeld huh?

the "helios powered alarm clock" you mentioned about 5 hours ago.

(really looking for a "podcast date, time offset" pointer)

sommerfeld ah thaaaaat. so jclulow mentioned in the Oxide and Friends chat that he made a clock using helios and /dev/kmem. I'll send the picture link here

sommerfeld here's the image: media.discordapp.net/attachments/12…&=&format=webp&width=934&height=700

sommerfeld is at here, I marked the minutes for you: youtu.be/9ejokdrByUs?t=3336

Thanks!

BTW I spent some time yesterday looking at whether it would be feasible to use sqlite3 to store pkg metadata (in place of the adhoc pile of files in /var/pkg)

short answer: looks feasible, and it's likely to save a considerable amount of space and probably also be faster.

sommerfeld omg thank you very much. I like the idea of IPS but not its implementation. I also hate that it doesn't keep a local copy of the things it downloads. I wonder if that can be configured.

second thought on the ginormous office clock screen at oxide: I've seen a display mounted like that used as a test status dashboard.

antranigv - `pkg set-property flush-content-cache-on-success False`

and then, on OmniOS at least, there is `pkg clean` to clear the cache if you ever want to

sommerfeld and that would be perfect to integrate with Jenkins/BuildBot :D

andyf thank you! I needed that

andyf I'm working on the r48 on r48 build. I'll report back in an hour or so.

antranigv: Ask, and ye shall receive: github.com/oxidecomputer/clock

antranigv: If you don't want it to flush the downloaded cache you should look at the "flush-content-cache-on-success" property in pkg(1)

Most people do, because if you don't flush it, then... it obviously gets big haha

if you want to retain a local copy in a meaningful way, mirror the parts of the repo you care about with pkgrecv.

Yese

*Yes

It'll then be useful for more than one system, etc

that's how I created pkg.omnios.illumos.am! by pkgrecv everything! :P :D

I have to say, IPS's documentation is pretty awesome.

sommerfeld: I suspect a SQLite store would also allow you to access subsets of a large catalogue without having to load the whole thing into memory as well. And probably would allow the "fast search index" to be maintained by SQLite as regular table indexes, rather than as a thing that occurs explicitly in the python code

Indeed.

the hack I have so far fits (almost) all actions of (almost) all installed packages in a 168MB pkg.db file

(well, 168MB in /tmp; du -sh of a copy of it on zfs says 78.5M with lz4 compression.

I believe that it has most of the content that's in /var/pkg/cache, of which du -sh says it's 1.22GB on this system.

and there are some additional semantic compression hacks I could pull off that would likely save more space

cracking apart FMRIs for one.

if the table fits in a modest amount of memory, you may not need complex indexing.

the "almost" is due to a few weird corner cases that defeated some of the simplifying assumptions I've made so far as I was going for an order of magnitude estimate for the backend database size to compare with the filesystem footprint.

the database went from 200-something to 168MB with a hack that stored attributes ending in a lowercase hex string to encode that trailing part as a binary BLOB.

Makes sense. I have been very impressed with SQLite haha

It sits under buildomat, our CI thing, at Oxide

yeah.. the only instances i've really seen where it's been a problem is more people not understanding sql and just doing things poorly

*cough* select * from table; *cough*

jbk: and then filtering the output to look for the one row they wanted?