Good news: At least on the big NUC my keyboard suddenly works.

Bad news: Before that happened I mistyped this: `cfgadm l` and I seem to have induced an ndi_devi_enter() wedge of some sort. Yes I do have the coredump.

More bad news: the other NUC appears to be still failing.

[illumos-gate] 17959 nlm_copy_netbuf(): NULL pointer dereference -- Toomas Soome <tsoome⊙mc>

[illumos-gate] 17961 klm: memory leak in nlm_host_destroy() -- Toomas Soome <tsoome⊙mc>

I'm missing something, probably something quite obvious, but $newfileserver is reporting zpools of absurd size and I'm confused. A raidz2 with 5 20TB drives should not be reporting a size of 90.9T, right? It should be closer to 60T.

but: : || lvd@fs2 ~ [519] ; zpool get size pool0

NAME PROPERTY VALUE SOURCE

pool0 size 90.9T -

the ssdp0 in the same host is also reporting an impossible size.

raidz sizing reports the raw size -- unlike mirrors

so that's the obvious thing I was missing.

raidz overhead depends in part on the block sizes written which is impossible to predict

I'm trying to chase down why our new fileserver is slower than our 5 year old one. Back to the drawing board.

so it punts and reports raw capacity while inflating usage

(I was hoping I'd just made the pools wrong.)

any ssds involved? One I just found was that m.2 NVME sticks without power loss prevention will be slower log devices than a SATA ssd with PLP.

I/O is faster but the cache flush op is slower.

the pool we were testing was entirely SSD based.

(ssdp0 in that paste)

c3t58CE38EE22F3C9C2d0 Soft Errors: 0 Hard Errors: 0 Transport Errors: 0

Vendor: KIOXIA Product: KPM7XVUG6T40 Revision: 0103 Serial No: 1E60A01G0993

Size: 6401.25GB <6401252745216 bytes>

that's the SSD in question.

IIRC, they're connected via SAS.

claims to have power loss protection so probably not the thing I ran into (with lower end hardware..)

to check something, oldpool (also entirely SSD, but SATA-connected), has 6 devices in the VDEV while newpool has 4. I presume that isn't going to impact speed to any noticeable amount.

might help to have a crisper definition of "speed". (The slowness I was chasing turned out to be fsync() latency; adding a log device that didn't suck made a big difference)

sommerfeld, I'm trying to work on that definition. Problem is, oldfileserver is under load. The new one isn't.

what we're seeing isn't really useful to quantify things, it's too many layers deep

(VM-based database server with all disks coming as VDI from an NFS mount from the fileserver. VM's VDIs hosted on old fileserver gives X database speed. same VDIs migrated to new fileserver, database is slightly slower.)

I want to do a speed test on the fileserver itself to see if it's the pool or NFS or something else.

but noticed the size report thing first so thought I'd check it before digging deeper.

IIRC, NFS does a fair amount of synchronous writes, or I guess on zfs will exercise the log

this pool doesn't have a dedicated log (or cache) device since it's entirely SSD already.

(same for the existing pool.)

the log always exists

just a question of it it's on a dedicated disk or not

though did you say the new disks are SAS vs the old being SATA?

that's my recollection, yes.

one thing I've wanted to investigate, but never had the time is queue depth.. SATA has a fixed depth (I'd need to re-read the specs, but I think it's basically either 1 or 16)

while SCSI doesn't really have a way to discover what it is for a disk

other than at some point, it might return an error that means 'queue full'

i've wondered (and the answer might be 'no') if sd is perhaps not always letting enough work get to SSDs

would prtconf -v confirm it's SAS?

I think the iostat -e shows SATA drives as 'ATA' for the vendor

since we pretend they're scsi drives and then translate the SCSI commands into ATA commands

iotstat -En returns a vendor name.

then it's probably SAS (or NVMe)

(NVMe would have a device path that end with 'blkdev⊙.' while SCSI would end with 'sd⊙.')

the diskinfo command should show all of that..

we don't have any NVMe in this. Everything is old-school hot-plug drive bays.

(as I just remembered :P)

the rpool drives are SATA and show as "ATA" in the vendor field. The SAS SSDs (and the spinning rust in pool0) show the proper vendor name.

so I guess I remembered correctly.

I can pastebin diskinfo output if you'd like.

jbk, is there an easy way to verify your queue depth question since I'm here with these drives?

(with a server that isn't in production yet.)

hmm. interesting. diskinfo says these disks's aren't removable when they very much are.

I guess that's doesn't really reflect hotswap, though. Never mind.

it was more of an 'in general' type question.. since scsi doesn't have a way to tell you..

diskinfo "removable" means something more like "cd, floppy, or usb stick"

k

unless I'm badly mistaken..

and there's probably a lot of things in sd.c that probably aren't useful much anymore

sommerfeld, the manpage agrees with you. I just spoke before reading it :)

(there's that whole 'prototype'

xbuf stuff

amongst other things

So I'm wondering if there may be benefits to a slog even in an all ssd pool. But that's in part based on experience with a pool which started with mirrored m.2 nvme's as "special" vdevs (metadata) plus mirrored spinning rust for main storage. adding a bad slog took fsync latency from ~1s to 2-5ms; adding a better one took fsync latency to more like 0.5ms

sadly, we don't have budget to add anything to this host. It's already way overbudget thanks to the [expetives deleted] AI bullshit.

the person who did the timing tests on the database notes writes were worse than reads. I presume he means in terms of the delta between current and new.

He's smart enough to know writes will be slower than reads.

gist.github.com/Bill-Sommerfeld/b6cf6713ed0edd72d04e4350dd47dc7f <- looks at fsync latency with dtrace

thanks sommerfeld. I haven't used dtrace in years so I'll need to review how to do it. I'll let you know what I find out.

in case you're curious. I did some fio tests. pastebin.com/uBkKGVfq has the raw data. I haven't even looked at it yet.

looking at the run status of the two runs doesn't make me happy.

ok, time to look into dtrace.

there was no load on fs2 (the new server) so I'm running fio over NFS to it.

sommerfeld, can you help me understand these results, please? pastebin.com/S6bfkpKM

hmm. ignore that paste. I re-ran with fio running against each server and the charts look a lot more similar.

just to ask, have you verified the zfs settings for the dataset are the same on both systems?

pastebin.com/0V46NHNC updated result.

jbk: yes

ok, zpool settings are the same but the directory I'm writing fio stuff isn't. compression is off for the old fileserver but on for the new one. hmmm. hmmmmmmmmm.

need to test that, obviously.

jbk, it is quite possible you've not only hit the nail on the head but driven it through the board and out the other side.

Run status group 0 (all jobs):

READ: bw=309MiB/s (324MB/s), 6241KiB/s-11.7MiB/s (6391kB/s-12.3MB/s), io=18.4GiB (19.8GB), run=60009-61036msec

WRITE: bw=312MiB/s (327MB/s), 6660KiB/s-12.1MiB/s (6820kB/s-12.7MB/s), io=18.6GiB (20.0GB), run=60009-61036msec

I was absolutely certain I'd set that right. Thank you for encouraging me to recheck my work.

no problem -- i've had enough issues myself that I always try to do an explicit sanity check to catch the 'easy' fixes

I thought I had checked it. Clearly I hadn't. Always good to ask the "It's obvious but..." questions.

yeah.. been there many times :)

I'm very glad you folks are here. I don't have anyone here to sanity check me on these things. The joy of being the only 'nix admin in my group.

Glad to help!

(Though I did nothing of the sort for this time.)

danmcd, you are a font of help.

( s/help/$SOMETHING_ELSE/g I think. :D )

a bold thing to say

sorry, had stepped out for a bit.

old server: most fsyncs take 0.5-1ms. new server: most fsyncs take 16-32ms.

directionally similar to what I was seeing on my new server and which I fixed by adding a pair of old sata ssds with power loss protection.

Your fsync test is on-local-disk? Or over NFS @sommefeld ?

on-local-disk.

or, well, it measures wall-clock-time taken by zfs_fsync()

Is it a simple write-a-test-program-and-time-it or are you running dtrace on the syscall over a load.

sommerfeld, I never updated the chart. The re-run on oldserver looks a lot more similar to the new server with fio running.

(prob the latter as you beat me to the typing punch).

I'll probably re-do with compression disabled on the new server shortly but right now we're setting up to see if the database server notices the difference.

timing fsyncs as used by whatever the workload is.

link I pasted earlier with dtrace script: gist.github.com/Bill-Sommerfeld/b6cf6713ed0edd72d04e4350dd47dc7f

[illumos-gate] 17960 want format_arg attribute -- Toomas Soome <tsoome⊙mc>

TY

without a slog I saw fsync latency in the 100-200ms range which was higher than I was expecting.

here's the fio-based dtrace against both servers. pastebin.com/0V46NHNC (I meant to paste this earlier but I think I got distracted and forgot.)

that was from before I found the compression discrepancy.

FTR: our database speed test results are much better now so yeah, me being stupid was the problem.

one thing I recall someone at oxide mentioning is that for NVMe at least (not sure if it'd more broadly apply to SATA or SCSI SSDs or not) that (IIRC) the reordering of I/Os by the zio scheduler wasn't beneficial and I think actually somewhat detrimental to performance... which seems reasonsable since SSDs don't have to worry about rotational latency and such

jbk: yeah. I'd imagine there are probably cases where there would be a benefit coalescing adjacent small-block i/o's but they'll be relatively small compared to the benefit of an elevator sort on a real disk..

is scheduling I/O wrongly actually real pain, or just something that shows up when you measure it?

at least some SSDs appear to use some combination of super capacitors, ram, and SLC flash to 'buffer' I/Os to the 'main' flash.. so it wouldn't surprise me if at least some of them can handle small I/Os to adjacent (or adjacent enough) blocks internally

danmcd: I would be surprised if there was only that AF from LX that you probably want to reserve

For now, it's the one we have. Fixing this can probably set a precedent for bizarro-brand stuff like this.

Also, I wonder about some of the lesser-used AF values that are probably around only for historical purposes. (Looks askew at AF_X25...)

all removing them can do is break weirdo software, right?

danmcd: AF_DATAKIT!

wow, and chaosnet

and imp!

the anachronism of AF_INET < AF_IMPLINK

I've *written code* that established an X.25 call and sent data, but it was a completely different API, something more akin to DLPI or SOCK_RAW, and it was (checks calendar) 36 years ago this summer.

I have written code that did something over chaosnet, but only as an historical adventure

This was paid-summer-intern duty. I was supposed to *test* the aforementioned API.

Found more bugs in the C compiler than I did in the API, FWIW.

as recent as the late 90s, you had to use X.25 to communicate with phone switches... which was fun

well some switches..

It's why the OS/400 "Open FM API" supported Ethernet (both normal and 802.whatever), Token Ring, and X.25.

I've found ONE reference to it in a google search, in this PDF: public.dhe.ibm.com/systems/power/docs/systemi/v5r2/it_IT/apiexmp.pdf

i'm surprised it supported ethernet...

It HAD to... $MOTIVATING_CUSTOMER needed it.

ahh

I see their naming is almost as bad as mainframes

'can you spare a vowel'? :)

LLC... that was what I was trying to remember. I *can't* recall if the Ethernet support REQUIRED LLC or not. It has been 36 years.

heh

i remember trying to use dlpi with the llc2 driver on solaris 7 and having it immediately panic after opening the device :)

'ok then..guess I won't be messing with that'

It was not that many years ago when I had to maintain solaris cluster setup which was fetching call records for accounting over X.25 and was feeding them to database:D

i think i mentioned this before, but one of my coworkers had a system like that, except it was using sun's SNA (cue screams) stack to talk to an IBM mainframe since it had copies of all the call records (prior to this, there was a group that had to manually load 8-track 6250 DPI tapes sent from the various phone companies)

it was amusing in that after the mainframe did some upgrades, it couldn't communicate at the full 1.54Mbps (T1)

but the mainframe is never at fault :)

(the fix was to clock the circuit down to 768k)

oops:)

but at one point, my coworker had whoever the direct # for the engineer at Sun that wrote (or at least maintained at that point) the SNA stack..

because it took quite a while to figure that out