00:02:21 one question for CPU topology. the physical machine is 2 socket, and 20core/40 threads per socket ; if create a bhyve VM with 16 vcpus, then which topology is better: 1.) just set vcpus=16, 2.) set sockets=2,cores=4,threads=2 3.) set sockets=1,cores=8,threads=2 which set is better for performance, anyone could point me? best wishes 00:19:31 tozhu: are you constraining the guest to particular physical cpus? which one is better depends on communications and memory access patterns within the workload. spreading across sockets may have more total memory bandwidth; keeping all threads on one socket may mean lower latency for interprocess synchronization. 00:21:39 sommerfeld, do you mean the option 2, will allocate 4cores for each phy sockets? and option 3 will allocate all cores on same sockes ? 00:23:10 thank you very much 00:30:18 I'm actually not sure what bhyve will do with that configuration information (whether it lets you reserve specific physical cores/threads for a guest or not). I was under the impression that the default config an N-cpu guest gets N threads on the host that get scheduled by the host kernel. 00:31:15 copec here's a tip: suggest the change during an emergency. "OH NO THE DHCP IS DOWN? I THINK DEBIAN'S APT IS HAVING DEPENDENCY ISSUES!" and then install OmniOS and setup DHCP... never have issues... and step by step (or rather, emergency by emergency) half of your infra will be running 00:31:30 some of that config is no doubt what you tell the guest VM about the machine topology. 00:34:47 sommerfeld that is absolutely true 00:35:08 I use sockets=,cores=,threads and CPU pinning every day at $work 00:35:46 and during our benchmarking we've noticed differences with/without s=,c=,t= and with/without pinning 00:35:50 specially if it's a large VM 00:36:00 antranigv: only way to know for sure what's best is to benchmark each config (and look at performance counters). 00:36:29 I have a post about that: https://antranigv.am/posts/2023/10/bhyve-cpu-allocation-256/ 00:36:58 as a matter of fact, if you're allocating a large number of CPUs (240 in our case), then the only way to make it work is to use s=,c=,t= 00:37:51 sommerfeld what's even more interesting, is that the booting process gets almost 2x faster (Linux guest) if you do CPU pinning. 00:38:07 and that happens only with Linux. without CPU pinning it's just slower at boot 00:38:19 I tried FreeBSD as well, the speed is always the same 00:40:01 another fun fact: if you boot Linux on bhyve with more than 1TB of memory then the Linux guest will ignore the first 1TB of memory and choose the rest, aka allocate 1.2TB, Linux sees only 0.2TB 00:40:24 documented here: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=276463 01:04:01 I’m reading https://news.ycombinator.com/item?id=39178521 Its amazing… I didn’t realize I was that old 01:10:41 copec huh? :D 01:13:27 Generations have grown up with “the cloud” and fundamentally don’t understand what the infrastructure is 01:22:10 thanks sommerfeld antranigv 01:25:30 copec oh yeah. I'm very young (late 20s) but I feel very sad about my generation. Most of them have never setup a "server" at home. And they look weird at me that I run a home server. :/ 01:26:42 copec I showed my OmniOS and FreeBSD infra to a friend, which is built with Zones and Jails, and he told me "why not just kubernetes on AWS?" 01:48:52 sommerfeld, tozhu: It's just lies that it tells to the guest. Unless you do something specific in the host it won't make a diference. 01:51:45 rmustacc: thank you very much 03:56:02 antranigv: i you tell him because you value you sanity (and cash) ? :) 11:13:06 One of the upside listening to Oxide and friends is that i now read some messages in the persons voice 😅 11:16:42 sjorge: the thing I find fascinating about their (great, great) podcast is that it does absolutely everything but actually sell their product 12:09:35 jclulow can you share the "helios powered office clock" code? I understand that you're using /dev/kmem, but for the love of god I do not understand how one would do the implementation :)) cheers! 16:53:00 antranigv: okay, what? (don't really have N hours to listen to the podcasts right now..). 17:09:05 sommerfeld huh? 17:13:24 the "helios powered alarm clock" you mentioned about 5 hours ago. 17:13:59 (really looking for a "podcast date, time offset" pointer) 17:14:34 sommerfeld ah thaaaaat. so jclulow mentioned in the Oxide and Friends chat that he made a clock using helios and /dev/kmem. I'll send the picture link here 17:15:04 sommerfeld here's the image: https://media.discordapp.net/attachments/1201689415307759616/1201725824785453126/PXL_20240130_030550596.jpg?ex=65e68ce3&is=65d417e3&hm=2d49f3ee479b5748e25d567ef9adbbed517f4a40c376f30792a8324d7072d09c&=&format=webp&width=934&height=700 17:24:21 sommerfeld is at here, I marked the minutes for you: https://youtu.be/9ejokdrByUs?t=3336 17:27:30 Thanks! 17:30:03 BTW I spent some time yesterday looking at whether it would be feasible to use sqlite3 to store pkg metadata (in place of the adhoc pile of files in /var/pkg) 17:35:11 short answer: looks feasible, and it's likely to save a considerable amount of space and probably also be faster. 18:40:40 sommerfeld omg thank you very much. I like the idea of IPS but not its implementation. I also hate that it doesn't keep a local copy of the things it downloads. I wonder if that can be configured. 18:40:52 second thought on the ginormous office clock screen at oxide: I've seen a display mounted like that used as a test status dashboard. 18:41:32 antranigv - `pkg set-property flush-content-cache-on-success False` 18:42:22 and then, on OmniOS at least, there is `pkg clean` to clear the cache if you ever want to 19:33:57 sommerfeld and that would be perfect to integrate with Jenkins/BuildBot :D 19:34:01 andyf thank you! I needed that 19:34:21 andyf I'm working on the r48 on r48 build. I'll report back in an hour or so. 20:53:36 antranigv: Ask, and ye shall receive: https://github.com/oxidecomputer/clock 21:11:24 antranigv: If you don't want it to flush the downloaded cache you should look at the "flush-content-cache-on-success" property in pkg(1) 21:11:38 Most people do, because if you don't flush it, then... it obviously gets big haha 21:12:29 if you want to retain a local copy in a meaningful way, mirror the parts of the repo you care about with pkgrecv. 21:12:36 Yese 21:12:37 *Yes 21:13:22 It'll then be useful for more than one system, etc 21:14:43 that's how I created pkg.omnios.illumos.am! by pkgrecv everything! :P :D 21:15:02 I have to say, IPS's documentation is pretty awesome. 21:16:01 sommerfeld: I suspect a SQLite store would also allow you to access subsets of a large catalogue without having to load the whole thing into memory as well. And probably would allow the "fast search index" to be maintained by SQLite as regular table indexes, rather than as a thing that occurs explicitly in the python code 21:18:07 Indeed. 21:22:38 the hack I have so far fits (almost) all actions of (almost) all installed packages in a 168MB pkg.db file 21:23:10 (well, 168MB in /tmp; du -sh of a copy of it on zfs says 78.5M with lz4 compression. 21:23:36 I believe that it has most of the content that's in /var/pkg/cache, of which du -sh says it's 1.22GB on this system. 21:24:34 and there are some additional semantic compression hacks I could pull off that would likely save more space 21:25:08 cracking apart FMRIs for one. 21:28:10 if the table fits in a modest amount of memory, you may not need complex indexing. 21:30:01 the "almost" is due to a few weird corner cases that defeated some of the simplifying assumptions I've made so far as I was going for an order of magnitude estimate for the backend database size to compare with the filesystem footprint. 21:32:37 the database went from 200-something to 168MB with a hack that stored attributes ending in a lowercase hex string to encode that trailing part as a binary BLOB. 22:47:00 Makes sense. I have been very impressed with SQLite haha 22:47:34 It sits under buildomat, our CI thing, at Oxide 22:48:42 yeah.. the only instances i've really seen where it's been a problem is more people not understanding sql and just doing things poorly 22:48:50 *cough* select * from table; *cough* 23:01:00 jbk: and then filtering the output to look for the one row they wanted?