15:28:33 I'm still collecting data but I'm starting to suspect some changes were made that impact write speed in encrypted filesystems. 15:28:35 NFS write to encrypted filesystem: 15:28:35 : || lvd@blivit tmp [1012] ; scp bigfile waiter:/media 15:28:35 bigfile 100% 3708MB 35.2MB/s 01:45 15:28:35 write to the same test filesystem without encryption: 15:28:35 : || lvd@blivit tmp [1013] ; scp bigfile waiter:/media 15:28:37 bigfile 100% 3708MB 205.4MB/s 00:18 15:29:01 that's scp of a file from host a to host B, writing over NFS to file server fs1. 15:33:58 This is a very early test and it is quite possible memory buffering is involved, though I've copied that file multiple times with similar results each time. 16:02:34 https://pastebin.com/f9BbUTH7 has some interesting results. It *looks* like it is a combination of NFS + encryption that causes write speed problems. 16:34:47 nomad note, writes to local files are async, nfs writes are sync. You want to check zilstat to confirm. 16:43:57 what am I looking for, exactly? 16:44:48 * nomad has no real dtrace experience 16:48:22 zilstat does show stats of sync mode writes. indirect way to check this hypothesis is to zfs set sync=disabled for the share dataset and see if it will change the result. just remember to set it back to standard after;) 16:51:46 setting sync=disabled on the test filesystem resulted in no change to the dd write over NFS. 16:52:17 with encryption? so thats not the reason:) 16:52:59 oh, no, I think I might have misspoken. It was majorly improved. 21s instead of 121s. 16:53:04 er, 31s 16:53:40 so yea, that was the problem but why did it change in the past few months? This filesystem has existed (and been encrypted) for years. 16:53:55 it will take few sec to get the buffers written out and code paths switched, yep. 16:54:44 I'm trying again with sync=standard and seeing slow writes again. 16:55:47 I expect some slowness from this but certainly not this much. 16:56:23 and why would encryption impact this? 16:57:20 I guess its about block sizes and IO aggregation 16:59:21 with async, system has chance to aggregate neighboring blocks (dd is essentially huge sequential write machine;) with sync mode, every block is forced down to stable storage as it appears 16:59:45 is there a separate log device for the pool? 16:59:50 Was there a "recent" (~ 2-4 months ago) code change that would touch this? The user who filed the ticket says he started noticing it getting slower compared to previous writes "compared to a month or two ago". 17:00:26 nfs 4.1? 17:00:26 jbk: it has separate cache and mirrored log SSDs. 17:00:44 are those working ok? 17:01:06 tsoome: mount on the AlmaLinux 9 box says vers=4.0 17:01:20 IIRC, zfs will write + flush to the log device(s) for sync writes (so it's on persistent storage) 17:01:23 jbk: there are no problems reported. Anything I should check to verify? 17:01:54 (in a pool w/o a log device, the same still happens, but the log is just another on-structure disk in the pool) 17:02:00 probably worth checking ssd health, but it also, may it be the IO paths are more loaded than usual? 17:02:22 (also why you want your logs to be faster than the main disks in your pool) 17:02:30 otherwise they can become the bottleneck 17:02:40 That's why logs are on SSD, the pool itself is spinning rust. 17:04:27 if they can take a risk and experiment, remove ssd from mirrored log and add as second log device, would it change anything? 17:06:24 sadly, I don't dare do that on this production fileserver. 17:08:05 of course, indirect way to check is to observer iostat -xn 1 for some time, see high svc_t column values, high busy% or just high throughput or IOps numbers 17:09:05 observe* 17:09:41 smartctl reports "PASSED" for both SSDs. 17:11:18 log ssd's are hammered with writes, if the bandwith or iops numbers are going high, then single disk may not be enough there 17:11:46 Our log is a mirrored pair. 17:12:16 (further on smartctl, they're both saying "no errors logged") 17:12:32 mirror pair is essentially single disk for write - just providing redundancy. 17:21:28 gack, need to run. I'll be back in a bit to resume. 17:26:40 i don't _think_ it'd be needed, but one thing you could try (with some care) would be to detach one log device, trim the whole disk, re-attach, then repeat on the other disk to see if that makes any difference (though I'm trying to think of how to do that outside of zfs -- I don't think there's anything bundled on any distro that'd issue the ioctl) 17:27:27 although (if the data is available) a comparison of the write performance on the disks before/after would likely give you an idea if it'd be worth trying 17:28:08 since the zil tends to be 'append only' (more or less) in behavior, I'm not sure it should care too much 17:28:41 but that's also reliant on firmware behavior, so.. can be a bit of a crapshoot 17:29:47 and obviously there is a bit of a risk while one is removed (and probably would want to wait a little bit after added before doing the other) 17:30:03 small, but not zero, so need to weigh that 17:53:28 [illumos-gate] 18216 mdb: 'cp' may be used uninitialized -- Toomas Soome 19:00:23 [illumos-gate] 18152 fold /usr/lib/pci/pcieb into pcieadm -- Andy Fiddaman 19:32:57 Given what's stored on pool0 (which is to day, everything but VDIs), I'm reluctant to take any risks with it. 19:33:04 is to say 19:59:54 jbk: just as an FYI: the host that's exhibiting this problem is the one we're hoping to replace with the new hardware as soon as we have a working ICE driver. No pressure or anything. :) 20:00:20 (though I expect management to tell me to stop waiting and move the data if things like this keep coming up.) 20:01:26 hopefully once our lab move is finished, I'll be able to pick back up on it, though it's definitely feels like an endless grind 20:01:49 (probably because of all of that 'really 3-4 drivers in a trenchcoat' bit) 20:11:51 yeah, doesn't sound like much fun. 20:13:12 If it's half the shit i40e was (and it sounds like MORE) it won't be. 20:13:12 My real suspicion is this old fileserver just doesn't have sufficient RAM for the job we're asking it to do. 96GB was a lot 6 years ago. Now, notsomuch. 20:13:23 well, it *is* a lot, if you're talking cost. 20:14:52 danmcd: much more 20:15:11 it's a NIC, a network SWITCH, a CPU, a floor wax, _and_ a RDMA engine 20:16:05 i40e at least lacked the latter 20:17:15 given how complex it is, it would be nice if tehre was a way eventually to take advantage of all the other bits to at least some degree 20:17:37 just to like make the effort feel more worthwhile :) 20:18:11 (i'm guessing it'd vaguely look like some sort of vnic offloading) 20:18:24 but highly speculative (i.e. don't read too much into it) 20:19:13 I have no opnion here, but "this is complicated, let's use it all" has really bitten my ass in other contexts 20:24:59 i mean, it's a NIC that's really been designed with the intention of servicing VMs 20:26:15 everything it does beyond the basic checksum offload/LSO (such as it is) is geared towards that purpose 20:31:49 i suspect it's also probably easier to hit higher line rates if you can let the NIC do more of the work (e.g. LSO even though Intels implementation is somewhat terrible)