13:50:31 <jperkin> richlowe: I'm seeing stacks similar to illumos#14722 on a platform which should include that fix, do you want me to comment on the original bug or open a new one?
16:42:43 <rmustacc> jperkin: I'd probably suggest opening a new bug.
16:45:42 <jperkin> ok
16:46:27 <rmustacc> Mostly because if there is a problem, we'll want a new bug to describe it anyways.
16:56:18 <KungFuJesus> rmustacc: 16 512-bit ZMM registers 
16:56:21 <KungFuJesus> heh, actually, 32
16:57:12 <KungFuJesus> I am a bit worried there are still some bugs in the kernel FPU path, though.  The parity calculations in ZFS have been utilizing this for a while, haven't they?
17:12:17 <rmustacc> KungFuJesus: are you referring to my comment on 15254?
17:12:44 <rmustacc> And when you say the kernel fpu path, what do you actually mean?
17:27:15 <KungFuJesus> yes, and I mean the kernel corrupting userspace's register state for ymm*
17:27:20 <KungFuJesus> and zmm* I guess
17:32:23 <rmustacc> KungFuJesus: Because the kernel is using the FPU or something else?
17:32:28 <rmustacc> Do you have anything more for me to go on?
17:32:59 <rmustacc> The signal handling issue is a serious one that I'm trying to get turned around and solved.
18:32:33 <richlowe> jperkin: do you have the objects etc?
18:39:12 <richlowe> jperkin: bugger-ld or whatever would help
18:40:05 <richlowe> jperkin: you could also check `strings libld.so.4 | grep 'Link Editor'` to be sure you got both bits at the right version. 
18:49:38 <richlowe> jperkin: Can you also try running elftest and tell me if the tests I added fail?
19:41:43 <jperkin> ah yeh, lemme get a bugger-ld run
19:43:23 <KungFuJesus> rmustacc: sorry, went and grabbed lunch.  I'm only surmising what that bug report already claims
19:43:49 <richlowe> jperkin: super interested in the test run too
19:44:06 <KungFuJesus> your analysis does indicate that it's an issue relegated to signal handling, so perhaps that's not the biggest bug surface
19:44:08 <richlowe> the bits from that bug I have here work.
19:44:24 <richlowe> KungFuJesus: you're making a limited amount of sense, I think you missed what is affected and how.
19:44:53 <richlowe> the problem is that when a user process takes a signal, its avx state isn't saved so if the signal handler uses avx that state is corrupt when we return from the signal.
19:45:11 <richlowe> nothing in the kernel, except that that's where signals are implemented.
19:45:18 <KungFuJesus> Perhaps?  My understanding is that the kernel is corrupting the state of the FPU when the kernel context switches out of a signal interrupt
19:45:57 <KungFuJesus> meaning, the FPU state is not properly restored after the kernel clobbers those vector registers
19:47:09 <KungFuJesus> Now from what I understand, this only happens in the scenario that the kernel makes use of those registers, which is limited to the very limited use case where the kernel does hardware floating point math (almost never), or needs to use the vector registers (e.g. SIMD vectorized parity calculations, AES-NI for crypto, etc)
19:48:30 <richlowe> no, the kernel isn't clobbering them because it's using them
19:48:38 <richlowe> it's clobbering them because the _user signal handler_ is using them.
19:48:48 <richlowe> they're not being actively corrupted, they're being passively not saved when they should be
19:48:53 <KungFuJesus> ooof, I see
19:49:35 <KungFuJesus> this is similar to say, addressing an xmm register is a ymm register, where the upper half is going to be undefined
19:49:39 <KungFuJesus> I've been bit by that one, before
19:49:45 <KungFuJesus> as a*
19:54:49 <rmustacc> Yes, none of this is due to kernel use. It requires a rather exacting combination of things.
19:55:18 <KungFuJesus> well that's at least a good limited surface area for the bug, anyway
19:55:37 <rmustacc> After a fashion, I guess.
19:56:12 <rmustacc> It's true it can always be worse, but I don't feel good about what appears to a user program as arbitrary data corruption and that's why I'm trying to write up how it gets fixed, the nuances, and the challenges.
19:58:05 <KungFuJesus> but yeah, so the hazard is something like user program context switches, kernel runs proper xsave to dump register state to stack, user program sends signal, signal handling logic calls older fxsave or equivalent behavior, leaving half of the vectore register state on the stack as undefined
19:58:22 <rmustacc> No.
19:58:35 <rmustacc> It is not that.
19:58:41 <jperkin> richlowe: bugger-ld tarball posted to the issue, stupid elftests question, can I just run them in an illumos-gate checkout on a non-build machine?
19:59:32 <richlowe> you don't need a build machine, you don't need an illumos checkout, you need the package and the test runner.  It defaults to testing /bin/ld
19:59:51 <richlowe> oh, and a compiler/assembler.
20:00:01 <rmustacc> FWIW, all passing for me right now.
20:00:08 <richlowe> me too
20:00:12 <jperkin> where do I find them then?
20:00:29 <rmustacc> In smartos land they're in the proto area, but not in the ramdisk.
20:00:35 <richlowe> I don't actually know smartos-ishly, danmcd do you make this convenient?
20:00:58 <richlowe> jperkin: /opt/elf-tests and /opt/test-runner, conventionally
20:01:00 <jperkin> I already have an illumos-gate checkout on the host, just wondering if I can run them from there
20:01:01 <richlowe> and in the proto
20:01:19 <rmustacc> jperkin: Maybe if you lofs mount it into /opt
20:01:28 <richlowe> I have been known to link the proto area copy into /opt
20:01:37 <richlowe> I don't know if it'll run outside of there, because of test-runner stuff
20:02:04 <richlowe> I'll pull your tarball down when this build/test cycle for another thing is finished
20:02:13 <richlowe> I'm sorry I seem to have screwed it up somehow.
20:03:28 <jperkin> don't take this as any indication that things were fine before and now they're breaking, it's just that I only now actually have time to start filing bugs for stuff that previously I'd have just rm /cores/* ;)
20:06:26 <richlowe> no, but it means the fix was incomplete or bad.
20:09:26 <rmustacc> KungFuJesus: The kernel is only saving an fxsave state and retreiving it across the signal handler. It uses that to update the xsave state. So it the upper register state is well defined to be whatever was in the FPU. So if the signal handler doesn't use it, it'll not be touched.
20:11:34 <richlowe> jperkin: this might be a different bug that crashes in a similar place
20:11:44 <richlowe> and/or a bug I introduced fixing the other
20:28:54 <richlowe> since this is rust, has anyone else seen this?
20:28:58 <richlowe> oxide people?
20:30:28 <richlowe> jperkin: so I have suspicions, but I don't know why an entire company isn't shouting at me about this if I'm right :\
20:32:18 <richlowe> jclulow: Do you know how to influence the flags rustc passes to ld?
20:32:19 <jperkin> so looking at the other cores the others are from rust programs too (devel/git-cliff, www/zola), still looking to see if there are others..
20:32:37 <richlowe> jperkin: right, I don't see why this isn't making oxide _real_ angry 
20:32:41 <richlowe> which is interesting in its own right
20:33:01 <richlowe> jperkin: I want you to try something for me, but I don't know what that something is
20:33:14 <jperkin> but there are also hundreds of other rust programs that build fine..
20:33:51 <richlowe> jperkin: see dm
20:33:56 <jclulow> richlowe: A lot of it is pretty hard-coded I believe in the target type
20:34:10 <richlowe> jclulow: well I tihnk I know what's happening here, but I don't know why it's not happening to you
20:36:23 <richlowe> jclulow: can I ask you questions about how y'all build stuff, or are you busy?
20:36:39 <jclulow> I'm always pretty busy haha but fire away
20:48:12 <danmcd> (richlowe: SOrry I've been away... if you need something built on SmartOS I'm here-ish now, modulo me upgrading Kebecloud to the Triton update we cut last night.)
20:49:33 <danmcd> I believe, rich, you need to be in a zone that can build smartos-live to do what you wish.
20:49:37 <danmcd> (/bin/ld is there, e.g.)
21:11:04 <richlowe> it feels like the only real path to a fix in finite time is a bit shit
21:37:27 <jperkin> at least I have a fix for pkgsrc, with the added benefit of ensuring rust stuff has CTF now
21:38:33 <rmustacc> What do you mean there?
21:39:03 <rmustacc> Are you generating a subste of rust types as CTF or something where stuff with things partially built from rust has problems.
21:39:47 <jperkin> removing -Wl,--strip-all avoids the ld bug and ensures I can ctfconvert stuff before pkgsrc does its own strip handling
21:40:11 <rmustacc> Ah, gotcha.
21:42:31 <richlowe> jperkin: I can give you a fix to try in a little bit too
21:42:56 <jperkin> sure, though will definitely be tomorrow now, nearly zzz
21:43:04 <richlowe> same, I have to stop typing soon
21:43:15 <richlowe> _a_ fix is easy, convincing someone to approve it might not be 
21:43:58 <richlowe> rmustacc: if you're playing along at home this is #15260 (fenix?)
21:43:59 <fenix> BUG 15260: ld segfault similar to 14722 (New)
21:44:00 <fenix> ↳ https://www.illumos.org/issues/15260
21:46:47 <rmustacc> Oh. That's going to be a thing.
21:46:59 <rmustacc> I will try to carve out some thoughts after I get through this fpu write up.
21:48:11 <richlowe> luckily, it appears to only happen if you explicit configure a rust profile with strip=true, or otherwise use ld -s 
21:54:59 <richlowe> tsoome: replied again about cw, you might want to try to attract some attention that isn't me, too :)
21:55:11 <richlowe> I'm much more open to "in a followup" than other people might be
21:58:49 <rmustacc> richlowe, tsoome: Is there something you'd like me to look at?
21:59:24 <tsoome> rmustacc absolutely. https://code.illumos.org/c/illumos-gate/+/2541
21:59:25 <fenix> → CODE REVIEW 2541: 15230 cw: add support for clang (NEW) | https://www.illumos.org/issues/15230
21:59:58 <rmustacc> OK. Can you add me to it? It might take me a bit to get there as this fpu issues is... gnarly.
22:02:13 <tsoome> I'll add some background to issue too -- this feature would just be there to make it possible for people to start poking clang -- and I still need to push some tools patches too to make it even remotely possible:D
22:02:40 <tsoome> richlowe that followup sounds really reasonable, however. 
22:03:00 <tsoome> be it real followup, or just update to this patch:)
22:06:51 <richlowe> jperkin: sent you the branch and binaries to try out
22:07:30 <richlowe> if anyone doesn't like the awful conservative approach, I look forward to them trying to untangle what is less conservative but also provably safe
22:07:32 <richlowe> because I sure can't.
23:40:01 <alanc> commenting on mastodon (subtooting?) is an acceptable way to let you know your proc(1) man page is missing some proc tools, right?  https://fosstodon.org/@alanc/109520277885645860
23:50:02 <gitomat> [illumos-gate] 15207 libfmd_agent: memory leak in fmd_agent_nvl_ioctl -- Yuri Pankov <ypankov⊙tc>