jclulow: Here is with shared disks between two nodes - (case when only one connection to each node)

jclulow: gist.github.com/gusev-vitaliy/69f70…25f9cdf6a2fad06b#file-gistfile1-txt

is there sort of "obvious" cause for vmem_hash_delete() panic with "bad free" message?

like there are 'freeing into free cache' and such....

the address you're trying to free isn't in a vmem segment

that is, we hashed `addr` and walked the bucket, and it wasn't there

yep. that much was easy to see:)

Hey, I'm not the one who asked the question

nono, I meant no harm:) but yes, I guess it really has no obvious answers anyhow

seeing it from reap, so the kmem_cache_free() and friends have not spotted anything wrong

by default, kmem_cache_free() doesn't do a lot of checking on the fast path.

if it's a reproduceable crash, try the workload with kmem debugging enabled.

yep, thats the plan. there has been 2 cases, but not sure if it is reliably reproducible.

you could turn the lighter-weight kmem debugging on for longer periods of time, even if the heavier pieces would be a problem, too

(I've mostly found the checking wasn't a major concern, but the logging was problematic)

unfortunately, the logging is also often invaluable to fix what the checking has found :)