I'm still collecting data but I'm starting to suspect some changes were made that impact write speed in encrypted filesystems.

NFS write to encrypted filesystem:

: || lvd@blivit tmp [1012] ; scp bigfile waiter:/media

bigfile 100% 3708MB 35.2MB/s 01:45

write to the same test filesystem without encryption:

: || lvd@blivit tmp [1013] ; scp bigfile waiter:/media

bigfile 100% 3708MB 205.4MB/s 00:18

that's scp of a file from host a to host B, writing over NFS to file server fs1.

This is a very early test and it is quite possible memory buffering is involved, though I've copied that file multiple times with similar results each time.

pastebin.com/f9BbUTH7 has some interesting results. It *looks* like it is a combination of NFS + encryption that causes write speed problems.

nomad note, writes to local files are async, nfs writes are sync. You want to check zilstat to confirm.

what am I looking for, exactly?

zilstat does show stats of sync mode writes. indirect way to check this hypothesis is to zfs set sync=disabled for the share dataset and see if it will change the result. just remember to set it back to standard after;)

setting sync=disabled on the test filesystem resulted in no change to the dd write over NFS.

with encryption? so thats not the reason:)

oh, no, I think I might have misspoken. It was majorly improved. 21s instead of 121s.

er, 31s

so yea, that was the problem but why did it change in the past few months? This filesystem has existed (and been encrypted) for years.

it will take few sec to get the buffers written out and code paths switched, yep.

I'm trying again with sync=standard and seeing slow writes again.

I expect some slowness from this but certainly not this much.

and why would encryption impact this?

I guess its about block sizes and IO aggregation

with async, system has chance to aggregate neighboring blocks (dd is essentially huge sequential write machine;) with sync mode, every block is forced down to stable storage as it appears

is there a separate log device for the pool?

Was there a "recent" (~ 2-4 months ago) code change that would touch this? The user who filed the ticket says he started noticing it getting slower compared to previous writes "compared to a month or two ago".

nfs 4.1?

jbk: it has separate cache and mirrored log SSDs.

are those working ok?

tsoome: mount on the AlmaLinux 9 box says vers=4.0

IIRC, zfs will write + flush to the log device(s) for sync writes (so it's on persistent storage)

jbk: there are no problems reported. Anything I should check to verify?

(in a pool w/o a log device, the same still happens, but the log is just another on-structure disk in the pool)

probably worth checking ssd health, but it also, may it be the IO paths are more loaded than usual?

(also why you want your logs to be faster than the main disks in your pool)

otherwise they can become the bottleneck

That's why logs are on SSD, the pool itself is spinning rust.

if they can take a risk and experiment, remove ssd from mirrored log and add as second log device, would it change anything?

sadly, I don't dare do that on this production fileserver.

of course, indirect way to check is to observer iostat -xn 1 for some time, see high svc_t column values, high busy% or just high throughput or IOps numbers

observe*

smartctl reports "PASSED" for both SSDs.

log ssd's are hammered with writes, if the bandwith or iops numbers are going high, then single disk may not be enough there

Our log is a mirrored pair.

(further on smartctl, they're both saying "no errors logged")

mirror pair is essentially single disk for write - just providing redundancy.

gack, need to run. I'll be back in a bit to resume.

i don't _think_ it'd be needed, but one thing you could try (with some care) would be to detach one log device, trim the whole disk, re-attach, then repeat on the other disk to see if that makes any difference (though I'm trying to think of how to do that outside of zfs -- I don't think there's anything bundled on any distro that'd issue the ioctl)

although (if the data is available) a comparison of the write performance on the disks before/after would likely give you an idea if it'd be worth trying

since the zil tends to be 'append only' (more or less) in behavior, I'm not sure it should care too much

but that's also reliant on firmware behavior, so.. can be a bit of a crapshoot

and obviously there is a bit of a risk while one is removed (and probably would want to wait a little bit after added before doing the other)

small, but not zero, so need to weigh that

[illumos-gate] 18216 mdb: 'cp' may be used uninitialized -- Toomas Soome <tsoome⊙mc>

[illumos-gate] 18152 fold /usr/lib/pci/pcieb into pcieadm -- Andy Fiddaman <illumos⊙fn>

Given what's stored on pool0 (which is to day, everything but VDIs), I'm reluctant to take any risks with it.

is to say

jbk: just as an FYI: the host that's exhibiting this problem is the one we're hoping to replace with the new hardware as soon as we have a working ICE driver. No pressure or anything. :)

(though I expect management to tell me to stop waiting and move the data if things like this keep coming up.)

hopefully once our lab move is finished, I'll be able to pick back up on it, though it's definitely feels like an endless grind

(probably because of all of that 'really 3-4 drivers in a trenchcoat' bit)

yeah, doesn't sound like much fun.

If it's half the shit i40e was (and it sounds like MORE) it won't be.

My real suspicion is this old fileserver just doesn't have sufficient RAM for the job we're asking it to do. 96GB was a lot 6 years ago. Now, notsomuch.

well, it *is* a lot, if you're talking cost.

danmcd: much more

it's a NIC, a network SWITCH, a CPU, a floor wax, _and_ a RDMA engine

i40e at least lacked the latter

given how complex it is, it would be nice if tehre was a way eventually to take advantage of all the other bits to at least some degree

just to like make the effort feel more worthwhile :)

(i'm guessing it'd vaguely look like some sort of vnic offloading)

but highly speculative (i.e. don't read too much into it)

I have no opnion here, but "this is complicated, let's use it all" has really bitten my ass in other contexts

i mean, it's a NIC that's really been designed with the intention of servicing VMs

everything it does beyond the basic checksum offload/LSO (such as it is) is geared towards that purpose

i suspect it's also probably easier to hit higher line rates if you can let the NIC do more of the work (e.g. LSO even though Intels implementation is somewhat terrible)