andyf: is your media build in the... kayak? repo?

I suppose some day I should learn how to build omnios media too

[illumos-gate] 17526 cxgbe could use more queues -- Ryan Zezeski <ryan⊙zc>

[illumos-gate] 17974 64-bit programs deserve fmtopo too -- Robert Mustacchi <rm⊙fo>

Hey wacki: you around? Someone pointed me to the eariler chat about NFS problems after the recent work. Does it seem like it's the NFS client that's having problems?

I gave the NFS code some additional review and came up with these two items:

It would help a lot if we can come up with some methods to reproduce the problems you see.

Hi gwr, I have some time later (in about an hour). My first impression was that the nfs server is the problem because booting an older BE fixed the problems. But the discussions here let me now think that our nfs client has problems with the new server.

In my network I have a server (running OI) that mainly provides nfs shares to the clients (mostly OI).

The OI clients are being used as desktop systems and development systems for OI.

When using them as dev systems they all share a common source archive where all downloaded source packages land (mostly compressed tar archives but also a bunch of other formats, too).

Beginning of Arpil 16th I got some problems unpacking source archives. These problems are reproducable but not every try fails. Sometimes the problem vanishes with a 2nd or 3rd try. Sometimes it takes more tries until an archive can be successfully unpacked.

I did some additional tests with a GhostBSD (FreeBSD 14.3 based) client. This one was able to unpack all archives I tried to unpack. That was roughly 20 different archives of different sizes and ages.

That made let me change my mind and now our nfs client seems to be more likely to be responsible.

At the moment I am using a BE from April 15th (that is one from before the latest nfs related illumos-gate changes) on my server (and the latest bits on the clients) and that has fixed the problems.

gwr: check the IRC logs from Saturday for more discussion. Problem seems to occur when a client is storing a source archive to an NFS directory and then immediately unpacking it; the archiver or decompression utility reports malformed input.

so some mishandling/miscaching of file attributes by the client when talking to the new server but not when talking to the old one.

so for a reproducer I'd try something like: for n in $(seq 100); cp foo.tar /nfs/mounted/directory/foo-$n.tar; tar xvf /nfs/mounted/directory/foo-$n.tar; done

(or perhaps miscaching of file content; it's not clear which)

ok, I have 2 packet trace files from snoop, bad and good file read.

apparently our nfs client does not like READ Status: NFS4ERR_DELEG_REVOKED.

based on its also unknown for snoop;)

so our client not coping with something new in 4.1 ?

client is 4.0 (noone has touched it to support 4.1). but it may be something we missed - gwr did create diffs, or it could be bug in 4.0, fixed at server side and now biting back at client...

DELEG_REVOKED appears to be a new error code in 4.1

"A stateid designates recallable locking state of any type (delegation or layout) that has been revoked due to the failure of the client to return the lock when it was recalled."

don't see that in RFC7530 but I didn't look that hard.

when we get error, dtrace does list number of calls to nfs4_lockrelease(). but I used a bit too big cannon there to trace with...

so can the server tell if the client in 4.0 or 4.1?

err, is 4.0 or 4.1 ?

wonder what would happen if the server returned NFS4ERR_EXPIRED instead of NFS4ERR_DELEG_REVOKED to a 4.0 client...

yes, the server knows the protocol level of each client.

(from rfs4_get_deleg_state)

more generally, don't return error codes new in 4.1 to a 4.0 client...

Is there evidence the NFS server returns NFS4ERR_DELEG_REVOKED to a 4.0 client? That seems wrong (it's a 4.1 error)

gwr: see above: <tsoome> apparently our nfs client does not like READ Status: NFS4ERR_DELEG_REVOKED.

thats what wireshark told, yes

OK. Will need to look at those. If there's a way to repro this, I can dtrace how we end up with that error return.

I mailed the steps I used

Ah, thanks.!

what are the actual rules for the format of the 'external bug' field in redmine?

I have an issue that ended up requiring 3 different internal tickets, and it's not liking trying to list all three related tickets

jbk: I suspect it only takes one value

jbk: I'm not sure who to talk to except jclulow about extending that

since it's custom

i'll just stick them in a comment for now

and once it's figured out, I can update the field

I guess the only people it helps are you, so go for it?