07:01:11 [illumos-gate] 13095 ZFS ARC minimum size is too large -- Joshua M. Clulow 15:04:09 @jclulow --> While merging your arc changes, I noticed in illumos-joyent that: 15:05:26 1.) zfs_arc_overflow_shift is lower in illumos-joyent (3 instead of 8, thanks to seemingly workaround OS-6385). 15:05:59 2.) A similar low-end clamp on large-memory machines was 1GB instead of 2GB. 15:06:38 So as part of merging, eliminating #2 is easy (since yours/upstream's says 2GB instead), and I'm thinking resetting zfs_arc_overflow_shift back to 8 to match upstream seems reasonable 15:07:09 because the change to 3 was a one-line workaround for what became fenix illumos#9284 15:07:10 BUG 9284: arc_reclaim_thread has 2 jobs (Closed) 15:07:10 ↳ https://www.illumos.org/issues/9284 15:08:06 Dammit, lemme rephrase. Setting it to 3 was a placeholder UNTIL 9284 (aka OS-6363) and also fenix illumos#9018 15:08:07 OS-6363: system went to dark side of moon for ~467 seconds (Resolved) 15:08:08 ↳ https://smartos.org/bugview/OS-6363 | https://github.com/joyent/illumos-joyent/commit/daa3911 15:08:23 (aka OS-6404) 15:10:25 I'm mentioning this in case the ghosts of SmartOS past are suggesting "Oh shit, maybe either 1GB min-ceiling or zfs_arc_overflow_shift set to 3 should actually be upstreamed!!!" I think those ghosts have been exorcised (and I'll be making merge choice there appropriately), but hearing they have from someone else would be nice. 15:56:45 speaking of systems going to the dark side of the moon... once it gets some more soak time, i can throw up the i40e patches that should largely prevent the 'stuck creating vnic for minutes' issue 16:05:08 the only catch is to avoid having to do an extensive rewrite of the tx logic (which the hw makes complicated enough), it does have to bump the minimum DMA threshold up to 512 bytes 16:30:02 at least with our stuff, we've found setting it to 512 yields better performance anyway, so at least to us wasn't a big deal 16:52:20 You mean for the whole system DMA threshold? 16:52:54 for the nic 16:53:11 when it decides to DMA bind vs. copy into pre-allocated buf 17:22:13 Oh yeah. 17:39:28 the usual high fixed cost for DMA setup vs variable cost for copy tradeoff that's been with us forever. 17:39:35 Oh, this is too funny.. https://github.com/illumos/illumos-gate/pull/89 17:40:12 it needs a polite but firm rejection. 17:40:56 looking at the account, done that to a few other repos 17:41:54 danmcd: Yeah I suspect small delta in thresholds should go away, and substantive improvements to mechanisms should come upstream 17:42:13 The more we're running the same bits everywhere the better I reckon 17:42:48 I am planning to resurrect https://code.illumos.org/c/illumos-gate/+/1272 in some form too. 17:42:49 → CODE REVIEW 1272: vm_pageout improvements from SmartOS/OmniOS - WIP (NEW) 17:42:56 SO thank you @jclulow --> this reduces our arc.c diffs to the zone ZFS I/O priority, which I'm totally cool about. 17:43:00 FWIW, I would also totally support arc_c_min cap coming down to 1GB 17:43:08 Oh? 17:43:29 I can keep it 1GB (by modifying your new upstream code) in SmartOS for the time being. 17:43:37 I pulled 2GB from my hat, really. It could be smaller. Mostly making it not smaller was because I am chicken 17:43:55 But SmartOS had it at 1GB for a long time. Tell you what, I'm keeping it 1G and filing a bug. 17:44:01 Yeah I think that's better 17:45:02 As with most of these values that scaled linearly with some physical property of the machine (RAM size, CPU thread count, etc) I think we were only ever looking at a very narrow region of the chart of appropriate values 17:45:21 And it would be appropriate to put relatively low absolute caps on many of them that grow to be quite large 17:45:58 The ARC definitely won't shrink unless you push it to shrink, and if you're pushing it to shrink it really probably needed to happen based on what you're putting on the machine 17:50:35 looks like a bunch of ignored entries under https://github.com/illumos/illumos-gate/pulls beyond the AGPL troll 17:52:58 * nomad first read that as "a bunch of ignorant entries" 18:25:19 not entirely wrong, if viewed as ignorance of the actual process to integrate changes to illumos-gate 18:31:29 I read the latest username as "eejit", which seemed appropriate 19:04:26 jclulow I remember running zfs in al old pentium 3 (nevada b35) with 512 mb so maybe arc on that time could be smaller? 19:05:59 andyf just added a new column to the ::vmm dcmd, just the zone_t to now which vm is running in which zone, just to save typing https://code.illumos.org/c/illumos-gate/+/3095 and the indestructible flag 19:06:00 → CODE REVIEW 3095: 14525 Would like kmdb module for vmm (NEW) | https://www.illumos.org/issues/14525 19:32:36 @jclulow fenix illumos#15995 19:32:37 BUG 15995: 13095's 2GB arc_c_min ceiling should be 1GB (New) 19:32:37 ↳ https://www.illumos.org/issues/15995 21:49:44 [illumos-gate] 15931 Add ::tdelta dcmd to mdb -- Jason King 22:20:22 alanc: Do you recall if KM_PUSHPAGE and the pageout_reserve threshold were added specifically for ZFS, or if there were also issues with swap to UFS files and NFS that precipitated it? I don't see KM_PUSHPAGE in a Solaris 9 manual PDF I found online. 22:24:11 PSARC 2002/761 KM_PUSHPAGE makes no mention of ZFS (which was still in early development then) nor any other filesystems 22:26:30 looks like it was first integrated to early S10 though 22:31:26 the PSARC makes it sound like pageout_reserve dates back to 2.6 though 22:33:00 I think this whole web of stuff is a mess, honestly 22:33:05 but I'm not sure what we could do about it 22:34:48 I think `pageout_reserve` might be in the Solaris Internals books 22:34:56 if that helps at all 22:35:43 yeah, pageout_reserve is much older 22:35:54 bonwick, 96/07/28: 22:35:54 We introduce a new variable, pageout_reserve, which by default 22:35:54 is 1/2 of throttlefree. In page_create_throttle(), we allow all 22:35:54 allocations by pageout or sched to succeed (as we do today). 22:35:54 However, for non-blocking allocations, we fail the allocation 22:35:55 for any other threads if freemem < pageout_reserve. 22:36:14 from Solaris 2.6 22:37:06 that one mentions "third party drivers or if you page over NFS" 22:37:54 https://illumos.org/opensolaris/bugdb/bug.html#!1261108 22:37:55 → OpenSolaris issue 1261108: page_create_throttle() should keep a few pages solely for pageout (Fix Delivered) 22:37:56 Yeah thank you that's very helpful 22:39:34 and that references a bug from an entirely different universe: https://illumos.org/opensolaris/bugdb/bug.html#!1168966 in which paging parameters are tuned to support "about 300 telnet users" 22:39:35 → OpenSolaris issue 1168966: paging thresholds are too low on very big systems causing kmem alloc failures (Fix Delivered) 22:41:08 alanc: this is delightful