01:38:00 [illumos-gate] backout: 15405 Port NFSv41 base (needs work; see 15869) -- Patrick Mooney 05:49:55 really? 07:31:21 my non plex vm also had a weird issue when I woke up (like 10 min ago), not sure it's related as nothing in any logs but it's again something that uses sqlite dbs in it's /var/lib (which is on NFS for me) 07:31:45 Will poke it more later tonight when I have time 07:33:47 Internet seems to say, just don't use sqlite on NFS. Fair I guess, but It's been working fine for years for me. (Note it's single host -> ds, so there is like no multiple host accessing the same files) 07:34:03 sjorge ok, sorry to hear 07:34:47 Current plan is: use plex with vers=4.0 for a bit more, to make sure that's a valid workaround 07:34:50 sjorge: It's trendy to take a dim view of NFS, but if it was previously working and now it's not working it's a regression which is why we backed it out -- to give us time to debug it and fix it so we can integrate it again with the fix 07:35:20 That's fair 07:35:48 (that is: I'm sure people say "don't use SQLite on NFS, but I don't think there's any reason it shouldn't work, even if it doesn't perform super well, provided you're only using it from one system at a time) 07:35:50 I'll also try and replicate it with vers=4.1 and get a pcap dump 07:35:58 sjorge we definitely need samples of network stream from the time of problem. 07:35:59 sjorge: That would be very helpful thank you 07:36:44 Just want to make sure vers=4.0 is fine and vers=4.1 isn't running the new bits, that would hopefully indicate there is a new feature that is tripping up the client 07:37:18 Will report back later this weekend 07:37:19 tsoome: For what it's worth we were discussing it with you and Vitality on the bug tracker 07:37:31 We'll make sure it gets integrated once the issues are worked out 07:37:55 File system issues are always critical issues, and we need to take them seriously 07:39:34 *Vitaliy 07:39:40 Phone autocorrect is the worst 07:40:34 qq, does sharectl's nfs property of server_versmax take minors? e.g. can I set that to server_versmax=4.0 or server_versmax=4.1 ? 07:41:04 I suspect it does not 07:41:10 no, versmax is only for major version 07:41:27 OK, I know the min/max_protocol for smb takes minors. 07:41:37 because till 4.1 there was no minors 07:41:41 Right 07:41:51 Would have been a nice easy server side workaround if it's indeed all good on 4.0 07:42:06 OK, off to do saturday offline obligations 07:42:29 Do weekend things! Enjoy 07:43:22 jclulow in my book, discussion means also responses are involved. That is, you hear out what other people have to say. 07:44:16 it does not mean I object the backout as such, but I do object how this was done. 07:45:12 tsoome: I understand, and I sympathise. We had to make a call to unbreak master while people were around to make it happen, and to make time to debug without demanding people do that on their weekend 07:45:31 It's tough to do all of that asynchronously with people in different timezones 07:46:14 I promise that once this gets sorted out it can go back in 07:46:58 We just can't have known regressions lingering for days, it makes it hard for people to ship master to users 07:57:01 this is another problem, we can not expect single developer, company or single developer, can get everything "properly" tested. hell, we still find bugs more than *decades* old. So, if you want stable version, then stable branch must be built, but we can not block development just because we fail to handle versions. 07:58:35 single person* 07:59:58 tsoome: I agree that no testing is ever going to cover everything. In this case, we made a judgement call to accept the test plan you presented and integrate the patch. But then a regression was found, and when we find those we have to back things out 08:00:05 It's all a balance 08:00:49 If a regression was found months from now, then it would be different -- but it only just landed 08:02:07 You talk about how hard it is to find resources, but I have no idea where the resources would come from to run a stable branch. Who's going to be responsible for taking things from unstable to stable? That's also a full time job. 08:02:22 well, sure, we could have been arguing if backout is necessary given there is simple mount -o vers= option, but now there is no point. 08:03:05 As I said, we'll get the packet traces and we'll get it debugged, and in the meantime we're not asking anyone to work around a known defect with mount options on their clients. 08:03:51 There's nothing to stop sjorge from running a system with the patch to get those in the next couple of days and then we'll have it squared away. 08:06:44 it is full time work, and there is two option for this - either we do this at gate level, or we assume distributions do this, and currently we *assume* distributions do this *and* we are requesting gate to receive very stable patches, but that means we have practically no updates. just compare number of commits done on FreeBSD and on illumos - the numbers are there on github. 08:09:39 We definitely don't assume distributions do this. OI ships master to people straight up. Joyent and Oxide both stay close to master, merging often, and with often a fairly short runway between those merges and users running the software. 08:11:54 At any rate it's after 0100 here. I'm sorry we weren't able to have a full interactive discussion before we backed out the patch. I hope we can all come together to get it integrated with a fix this week. 08:12:33 (I guess s/Joyent/MNX/ now -- old habits!) 08:32:46 It's a damned if you do, damned if you don't situations. I run Omnios Bloody and upgrade every 2 weeks to 1 months. Because I like shiny things, and generally it's pretty darn stable for me with the occasional minor issue. e.g. the sudo RequireTTY default change that bit me. But because I run bloody Andy was quickly able to spot the difference and find the issue. If bloody was very unstable, I would probably just run Omnios Stable. Meaning we 08:32:46 probably would not have spotted the sudo issue anytime soon. 08:34:49 Pending more data, if vers=4.0 is a decent workaround for my workload I don't mind running like that for a bit. So a backout would not be needed, on the other hand, I'm not a company that depends on it for business reasons. Dan having just cut a SmartOS release with those bits, even if 1/100 of the customer workloads is impacted, it will be a big deal for them. 08:35:28 I'll probably file a feature request later for having server_versmin/server_versmax support minors too 08:36:38 That would also have allowed for an smoother integration, the default could have been server_vermax=4.0 until the 4.1/4.2 stuff had baked a bit more. Hightside 20/20 and all that though. 08:44:49 The backout won't be in omnios bloody until this time next week, most likely, so you have time to test the mount option workaround and grab packet traces or whatever will help. 12:07:55 andyf was there a way to install the debug variant into a new BE ? 12:08:05 While leaving the current be at the normal variant? 12:55:45 iirc something like `pkg change-variant --be-name debug.illumos=true` 13:21:35 that sounds vagually familier 13:21:45 thanks, will give that a go if needed 16:51:24 3h of playback and a full library scan on vers=4.0 without issues, I'll try to replicate it with vers=4.1 now and get a capture 16:51:48 cool 16:55:02 Good news is 16:55:07 I can trigger it with a simple library scan 16:55:28 with vers=4.0 it completes with maybe one or two warnings it took 300ms to respond to the db 16:55:35 But with vers=4.1 I get 16:55:49 https://gist.github.com/sjorge/d16eab4faf774c961200d3629eb4f75a 16:56:08 And I only switched the plex dataset to vers=4.1 I kept the ones with the scanned media on vers=4.0 16:56:16 Let me erm hard reset the vm and try to capture the packets 17:00:43 tsoome: so you want an a/b capture? ones on vers=4.0 were there are no issues an one on vers=4.1 ? 17:01:23 rmustacc: the nvme 2.0 rfc will that include nvme over tcp client ? 17:01:29 if you can make both, that would be great 17:07:09 ofcourse I can't replicate it with a tcpdump running, where would be the fun in that 17:07:32 It still gets the super long query t imes warnings though but not the full lockup 17:07:43 Stop the dump, restart the scan and instantly trigger it 17:09:48 Got it on the 3rd try! 17:17:46 Smithx10: Not at all. 17:18:08 I'm working on what I need. The library is providing the ability to admit if it's that. 17:18:11 But I'm not making a new target. 17:18:40 Always happy to collaborate with folks who do. 17:19:27 tsoome got both, uploading now... very intresting on vers=4.0 it never shows queries higher than 500ms and it finishes the scan in like ~1min dump is smallish 17:19:42 With vers=4.1 it never finishes and the dump is huge 18:09:08 For those reading along, tsoome now has a 4.0 and 4.1 dump where I trigger a library scan, where the alter just hangs plex and the former comples in ~1 minute or so 18:57:03 Neat 19:03:54 from what (little) I read, it does look like NVMe fabric and NVMe over TCP seem to be going in the direction that feels like 'scsi, without all the legacy stuff' 19:04:02 for lack of a better description 19:04:40 granted, i suppose what's in a standard vs. what actually exists is probably also a different matter 19:05:18 e.g. you can use IKEv2 over SCSI to do key exchange to encrpt SCSI frames over whatever transport you're using, but I'm not sure i've found anything that actually implements it 19:07:04 so in theory that could be a iSCSI replacement? 19:07:27 i have a love/hate (mostly the later) relationship with iSCSI 19:17:50 NVMe over TCP does (at least from a very superficial point of view) looks like the NVMe equivalent of iSCSI 19:18:08 and yes, i hate iSCSI 20:51:49 sjorge: Hi, Where did you put pcaps ? 20:56:22 tsoome has then, i removed them once he downloaded them 21:05:34 sjorge: I suppose tsoome is not online now. How big were those files? 21:06:13 90M for the vers=4.0 where I triggered a library scan, AKA the one were everything works and responds fine 21:06:25 450M_ for the vers=4.1 one 21:06:46 That one dit not finish and resulted in plex haging and requiring a pkill -z -9 bhyve 21:07:05 Nothing in dmesg, even with the echo thing from the ticket 21:07:24 It's getting late, but tsoome should still be online. I think he's -1 or +1 from me 21:07:33 So 22:00-00:00 range 21:07:47 00:07 here:P 21:07:52 sjorge: Did you try "echo w > /proc/sysrq_trigger" on Ubuntu VM? 21:09:00 Yes, thats what I ment with the echo from the ticket 21:09:13 sjorge: Good. Thanks! 21:11:27 I did not wait super long after it got stuck though, I know the first time round plex had the Z status once I got to it, like an hour after it hung. I could probably snapshot, force the issue and wait for a bit tomorrow to see if one pops up way later 21:12:41 The good news is that at least vers=4.0 is a workaround for now. 22:01:57 sjorge: Could you check vers=4.0 mount, but before set this on the server side: sharectl set -p server_delegation=off nfs 22:03:11 sjorge: As assumption that missed delegation can be reason why you have had a problems. 22:04:23 So you want me to try 4.0 with delegation off ? 22:04:27 Or 4.1 with delegation off ? 22:04:44 sjorge: 4.0 with delegation off 22:05:20 And you expect it to also fail ? 22:05:38 sjorge: yes. 22:06:28 sjorge: nfsv4.1 bits does not support delegation for now and it is planned following up work. 22:06:54 should I restart nfs/server after setting server_delegation ? 22:07:43 sjorge: I guess it should be restarted in the sharectl command. 22:08:03 online 22:05:47 svc:/network/nfs/server:default 22:08:06 svcs seems to agree 22:09:36 You might be on to something 22:09:47 ERROR - [MusicAnalysis] Waited over 10 seconds for a busy database; giving up. 22:09:53 Same error and plex hangs 22:10:41 Let me set that back to on and try again 22:11:37 Would lack of delegation support... cause more lockX calls? 22:11:52 tsoome saw none in the old 4.0 pcap but lots in the 4.1 pcap 22:14:00 Hmm restoring that to on seems to leave it in a broken state 22:14:25 sjorge: yes. It can. Delegation means that client "owns" that file and if someone tries to open, owner will be notified (recall delegation . 22:14:50 Interesting, so I set sharectl set -p server_delegation=on nfs again, restarted the vm 22:14:52 still broken 22:14:59 even with vers=4.0 22:15:19 but tcpdump shows LOCK calls too so that's new 22:15:28 brb, gonna reboot the entire box so my bouncer will be gone for a bit 22:24:50 sjorge: interesting. I would expect that GRACE PERIOD can effect for some fails, but LOCK calls shouldn't be seen. 22:25:18 Not sure those were cause by plex 22:25:26 as I was poking around my self a bit 22:25:42 But server_delegation=off and vers=4.0 on the client has plex hung too 22:25:50 Even after restoring server_delegation=on 22:26:06 VMs still booting so hopefully a full physical host reboot solves it 22:26:51 All good again after a host reboot 22:26:59 So you're theory on delegations is probably correct 22:28:52 sjorge: That's good. Thanks! I am going offline for now. 22:29:10 I should head to bed 22:29:21 thanks for looking into it vetal_ tsoome 22:29:47 sjorge: And you!