[illumos-gate] 17877 libnvpair(3LIB) lists non-existent nvpair_empty function -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 17865 SCF_PROPERTY_CREATE(3SCF) has wrong signature for scf_pg_get_property -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 14591 list_create(9F): provide example of safely deleting specific entries in the loop -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 13255 Missing man pages for userland mutex_enter()/mutex_exit() -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 18068 libc: initializer-string truncates NUL terminator -- Toomas Soome <tsoome⊙mc>

Anyone seen "NOTICE: ixgbe1: ixgbe_rx_copy: allocate buffer failed" in their logs? We had 1,794 instances of that this morning on an NFS server.

i'd need to dig into the source, but a possible cause would be if there's too many received packets stuck unprocessed upstack

yeah.. it's calling allocb() and that's failing..

could also suggest a memory shortage (though you'd probably see other things start to have problems first)

should I open a ticket at github.com/omniosorg/illumos-omnios/issues ?

Evidently this has happened on this host 5 times since January, 2025.

This time we noticed because a bunch of VMs lost their VDIs when it happened.

This host is a dedicated ZFS-based NFS file server with 96941 Megabytes of RAM according to prtconf.

do you have any disks that are maybe misbehaving?

or doing a scrub?

There is a high probability of a scrub in process. We do that at the beginning of the month. Let me look.

nope, the scrub finished last Friday.

anything running that might be sucking up ram?

If this doesn't look like a software problem then I'm going to just say we're done looking. This host is being retired in the next few months and is the only one that's reported the problem.

looking at when the reports were logged, the first 4 times were all around 03:30ish. The most recent one was at midnight. I'll have a look at crontabs.

mdb -ke '::memstat' _might_ tell if it's zfs being a pig or not

the immediate issue is that there's not enough memory available to satisify the allocation request without waiting

ZFS File Data 18511065 72308 75%

the question is 'why?'

I don't see anything in cron that corresponds with those times.

pastebin.com/KvUBK6Tk is the full output of that mdb.

zfs is unfortunately missing some backpressure mechanisms that'd be a bit of an extensive change to get in there where enough sustained demand (faster than the disks can answer) can cause zfs to exhaust memory

(running in a VM makes this a lot easier to hit, though even some backup software running over NFS or SMB on physical hardware can induce enough demand to trigger it)

yeah, we'd need to dig into that 21% and see how that breaks down

(could still be ZFS, could be something else -- since as the description notes, the ZFS bit there is basically memory for cached blocks, all of the kernel structs that track things still get lopped in with the rest of the kernel

pastebin.com/R68sSzrV is the output of zpool iostat. Interesting read numbers.

backups finished running hours before the outage so that wasn't it, though it sure could have been.

I sounds to me like this was a "this host" problem and not something that needs to be noted for OmniOS or Illumos itself. Am I correct in that reading?

you should probably if you file bugs etc include the values of `lotsfree`, `desfree` and all those

if it's described as tunable in vm_pageout.c, have a peek at it.

Unless you think I *should* file a bug report about this I'm not planning to do so. I don't want to waste anyone's time chasing a specific-host-only problem.

especially when that specific host is over 5 years old and scheduled for demotion Any Day Now.

I missed anything except you and jbk were suspicious of consumption :)

no problem. $NFS-fileserver lost network briefly with "NOTICE: ixgbe1: ixgbe_rx_copy: allocate buffer failed"

turns out we'd had that happen 5 times since January, 2025.

(all on the same host. No other host has done this.)

We introduced in SmartOS recently the zfscache(8) command for out of control ARCs. You can at boot time set an arc profile that caps ARC kernel memory consumption.

feels like that should be a zpool subcommand?

Those are globals in zfs, therefore they aren't per-pool, but you make a good point.

See Lines 92-102 of our `fs-joyent` SMF method run at boot-time.

hadfl, andyf: are y'all going to upgrade git in bloody soon?

(I walked into a need for `git split` which doesn't exist yet)

richlowe - yep. Now that r58 is out bloody gets unfrozen and we move it to r59

so it will get a few package updates.

great stuff