hrm.. this is odd... this system (annoyingly I can't access to it to dig more) is getting SMF_ERROR_NO_RESOURCES when a bunch of disks are connected to it (and failing to boot).. with the disks removed, comes up fine...

i was thinking maybe /etc/svc/volatile filling up, but at least with the disks removed, df -h is showing GBs of free space...

SCF_ERROR_NO_RESOURCES you meant? altho there is also SCF_ERROR_NO_MEMORY; apparently we do get SCF_ERROR_NO_RESOURCES translated from REP_PROTOCOL_FAIL_NO_RESOURCES but that too seems to be related to allocation errors.

still there are constructs like res = backend_tx_run(tx, q, fill_property_callback, &ci); if (res == REP_PROTOCOL_DONE) res = REP_PROTOCOL_FAIL_NO_RESOURCES;

basically, there are functions to return either OK or NO RESOURCES:)

[illumos-gate] 17314 Obsolete reference to mibiisa in the netstat(8) manual -- Peter Tribble <peter.tribble⊙gc>

[illumos-gate] 17315 The ddi_ufm_op_readimg(9E) manual is missing -- Peter Tribble <peter.tribble⊙gc>

tsoome_: yeah..

though out of memory seems odd.. the system has something like 300+gb of ram

... though svc.configd and svc.startd are 32-bit

... i wonder how many problems would crop up if I built an image w/ 64-bit versions of those...

tsoome_: i don't suppose you've tried that yet w/ your work of building things 64-bit? :)

I have done very little to build things 64-bit, just some zfs commands and few ones which used isaexec anyhow...

jbk: if you want "64bit versions of things", check the arm64-gate, they're all that way necessarily

for startd/configd in specific, you want _all_ the arm fixes to cmd/svc and lib/libscf, at least

also the "repcache" stuff is kinda scary, and where one of those bugs lives

heh..

so if tsoome is right pointing you into that, you probably want to make assumptions that what it seems to be telling you is true

it's a real hoot

the hard part is that I can't actually get my hands on the system

i get at best a description of what comes up on the console

and not even copy/paste or a screenshot

jbk: Is it an install to disk system, or a UFS ramdisk system?

ufs ramdisk (mostly, /etc is re-mounted later as a zfs dataset)

Really dumb question... should a swap volume have `primarycache=none` set? I mean, it's for swap, i.e. memory, right?

danmcd: I've used primarycache=metadata on swap volumes on the theory that you want to let indirect blocks and the like stay in cache.

but I haven't measured it

I'm worried about high-memory pressure situations where the reclaim takes so long IO times out. :( Corner-case, but something that prompted me to ask.

danmcd: in memory pressure like that so many things go wrong it's hard to point fingers

(you run out)

Or the ostensibly free memory just can't be reclaimed fast enough. :O

one thing i'm hoping we could upstream soon is we noticed a couple of zfs kmem caches that seemed to have been skipped when doing reaping

since we've noticed the ones that were missing could potentially hold GBs of data in their kmem caches

Whoa!

Which ones?

one related area of work I've thought about but haven't really done anything towards is having our swap system free blocks in the underlying storage when they are no longer needed.

sommerfeld: like UNMAP/TRIM operations?

I have a kernel dump from a too-large-BHYVE/transient-reservoir allocator that Just Froze Everything. Would love to inspect the state of the kmem caches you're targetting on this dump, jbk

sommerfeld: for fragmentation?

danmcd: dnode_cache && dbuf_kmem_cache

yeah. the swap allocator does a clock hand thing which means that an initially-sparse zvol winds up fully populated if you only have a little bit of swap usage.

the latter looks like it's named 'dmu_buf_impl_t' if you're looking in mdb

Not even a Gig...

> ::kmem_cache !egrep "dnode|dmu_buf"

fffffdf266f92008 dnode_t 1200 000000 688 636735

fffffdf2871ae008 dmu_buf_impl_t 1000 000000 216 1897380

> !echo "(688 * 636735 + 216 * 1897380)" | bc

847907760

>

not looking for actual overcommit, but it means the underlying pools have more actual free blocks to play with and don't have to preserve the value of swap pages that nobody will ever read again.

@sommerfeld I get it now.

kinda related to that, though perhaps of less interest given semi-recent events.. i do have a change to sd.c that'll issue WRITE SAME w/ a zero-filled block for VMware virtual disks instead of issuing an UNMAP CDB

since the latter does nothing w/ vmware

but the former will actually shrink thin provisioned vmdk files

WOW!

at least with ESX

but i'm guessing other variants are probably the same

(it actually has three different 'TRIM' implementations -- UNMAP, write same w/ unmap bit siet, write same w/o unmap bit set)

and it looks at the various vpd pages (and supported ops) to pick the method to use (or override via sd.conf)

(including disabling issuing unmap for the given disk type completely)

jbk: If it's a UFS ramdisk, and something fails when you attach more disks, your UFS ramdisk image is probably out of inodes?

All of the things created in the mounted /dev actually create a shadow object _under_ that in the root file system, to store permissions and such

Which, while useless for persistence in a ramdisk situation, means that you need to have a fair amount of inodes in your UFS ramdisk thing

I believe it's the -i option to newfs(8)

oh hmm...

"df -F ufs -o i /" should tell you how much you're using now

ahh firmware... don't ever stop being terrible (unrelated to that issue)

now identified two firmware bugs today and I wasn't even meaning to

so a regular day