00:00:39 kill(893, SIGSEGV) = 0 00:00:39 Received signal #11, SIGSEGV [default] 00:00:39 siginfo: SIGSEGV pid=893 uid=0 00:00:39 yes then it segfaults 00:01:05 can a not existant file cause a segfault? 00:01:32 https://paste.ec/paste/h97cqYIq#6EZgX6-QVwO4Rb8Pxn+pGWRl0bBkGmTd52JtxC+rAmO 00:03:37 it seems to be sending itself a signal?! 00:03:43 why does it do that 00:06:28 ah, I found the problem 00:06:39 my mac sends env variables that cause this 00:06:49 locale things? 00:06:57 limiting AcceptEnv limits things 00:07:00 yes locale 00:07:05 that still shouldn't segv? 00:07:11 One would hope not 00:07:46 Can you undo your AcceptEnv thing and "ssh -t $host env" maybe? 00:07:56 sure 00:08:59 https://paste.ec/paste/UxLQcB3-#H4bhkCGb2rh8MEzLTtCyDwxVuLakWlrcRmSqpxkGHV9 00:09:51 LC_CTYPE=UTF-8 might be the problem 00:13:53 no value in any LC variable should cause a segfault 00:14:04 true 00:15:38 if you "LC_CTYPE=en_US.UTF-8 ssh -t $host bash" does that work 00:21:20 yes 00:45:52 why would it do that 00:49:02 well even GNU has to work sometime. It just has no warranty when it will do which 00:59:19 i mean if there's no "UTF-8" (or the variables are empty), just use "C", why segfault 01:08:00 rzezeski: as my brain is mush at this point, if you happen to recall this stuff: https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/io/mac/mac_client.c#L3668-L3676 01:08:14 I'm kinda certain that on 3676 that should be |= 01:08:24 but have lingering doubts 01:09:12 (i have a panic where https://github.com/illumos/illumos-gate/blob/master/usr/src/uts/common/io/mac/mac_util.c#L977 is getting tripped) 01:09:24 and in this case the adapter isn't advertising lso or any hw checksum 01:24:12 jbk: so, I think you might be right about the |=, but I would want to double check that tomorrow with fresh eyes 01:24:20 however, I think that's separate from your panic 01:24:43 as that assert is checking that the IP stack requested the IP header checksum offload along with LSO 01:25:07 in your case you are saying the part advertisers neither offloads, which makes it very sus that you are in mac_sw_lso 01:25:23 if you can put the panic up somewhere I could take a look at it 01:26:38 let me see if I can figure something out 01:26:46 is this a customer thing? 01:26:57 no, just a place to upload it 01:27:02 ah okay 01:27:21 it's just a debug build of illumos-gate running under ESX w/ an emulated e1000g nic 01:27:42 okay, that's good to know, but I'm not going to have the cycles to stand that up 01:27:56 i might be able to do something tomorrow 03:10:57 umm.. 03:12:35 ok.. i'm completely confused.. 03:15:51 oh.. that's confusing.. dblk_ts have a number of fields (cksum flags) uninitialized 03:15:54 while others are 04:37:25 is this a vanilla kernel? because that sounds like it could be UAF 04:38:04 anyways, I'm going to bed now, but if you can share a panic tomorrow I'll look 07:51:42 Good morning. 08:32:37 morning 14:26:00 i believe kmem uses a different pattern for freed memory.. 14:28:39 i looked at what allocb() does (because it was confusing the hell out of me) 14:28:59 it's using some kmem caches, but it doesn't initialize all of the fields 14:29:28 so some of them are set, some of them have the uninitialize pattern 14:31:20 deadbeef is the free pattern, baddcafe is the uninitialized pattern, and feedface is the redzone pattern 16:32:59 Has anyone here tried VRML over Infiniband? 16:45:29 jbk - right, I usually have to look at umem_debug(3MALLOC) to remind myself, since it's the same as kmem 17:22:25 i don't think many people have used infiniband on illumos.. we have support for it, but I don't think there are drivers for any recent cards 17:23:30 at least for the infiniband functionality 17:24:00 (some cards e.g. mellanox ones work fine as regular network cards) 17:28:50 Did some spelunking on your panic jbk. There interesting bits thus far: 1) the mblk does indeed have HW_LSO, but nothing else (which is invalid), 2) the mblk is larger than e1000g0's MTU, 3) the ill this mblk is going over DOES NOT have ILL_CAPAB_LSO set (it does have ZC set, not sure if that's normal or not), 4) the e1000g0 mac DOES NOT have HW LSO or CHECKSUM enabled, 5) vmxnet3 ALWAYS provides HW LSO and CHECKSUM capab 17:29:46 it's almost like this packet was created to go over vmxnet, but instead ended up on e1000g0 17:30:24 they are on the same subnet--as i mentioned i threw that on the VM so I didn't have to use the console while testing out vmxnet3 improvements 17:30:42 so it's a bit unusual, but something that should be valid 17:31:14 granted, the chances of having real hw these days that doesn't do at least checksum offload seems slim to none 17:31:25 so it's probably a very niche corner case 17:46:31 Sure, but what I'm saying is for some reason a packet that seems like it should have gone over the ip interface on vmxnet3 is instead going over e1000g0 17:47:00 although given that vmxnet3 exposes partial checksum unconditionally it's very odd there is no hardware checksum flag turned on 17:47:07 do you have a modified vmxnet3? 17:48:16 oh wait, I just noticed vmxnet3 doesn't unconditionally expose LSO, but I don't see any way to easily check the state from a dump (this is a really weird way to code this) 17:49:18 their driver.conf file is also weird :) 17:49:41 if you changing this code you should change it so that it does the `vmxnet3_getprop` somewhere else and sets a flag in the software state, that way we can examine this state from a dump 17:50:08 but I did verify that your e1000g0 has no hardware offload enabled 17:50:29 yeah.. since it's virtual, not too surprising 17:52:20 like, I don't know if there's some confusion somewhere in the stack when you have multiple interfaces on same subnet with differing hardware offloads 17:55:34 jbk: were you running this ssh session for a while before the panic? Or was it the moment you try to ssh into the guest? 17:56:12 I guess it could work for a period before an LSO packet is generated 17:57:11 yeah it's a while 17:57:27 it was annoying because the assert would usually trip right as I was trying to look at something :) 17:58:28 oh wait, duh, this is hitting mac-loopback, wait wtf is going on 17:58:55 well the assert won't trip until you cause an LSO segment to be generated 17:59:04 so the session will work for a while and then boom 18:00:22 i think (just while doing this work) i'll take out the asserts... at first I had thought it was maybe the bit I pointed out last night (which sounds like it is a legitimate, but separate issue) 18:00:48 which is why I pinged you (since it looked like you were the most recent visitor to that part of the code) 18:10:42 richlowe: andyf I could confirm the update problem KungFuJesus was reporting. They are indeed caused by system/library/processor still being incorporated and not having a new version. uninstalling the package solves this. It should however either be obsoleted or not incorporated anymore 18:10:44 https://paste.ec/paste/ARPd+Mav#8mYSRpJasRzrl1x1z1kq4lo9auJGNIs8d4Rj8FaVgqX 18:11:02 I wouldn't take out the assert, it's a correct assert. You might run into even weird behavior. I don't understand why this is invoking the mac_hw_emul routine, that might be a much larger bug, as I thought it was only being invoked when about to traverse mac-loopback, but in this case we are at code which comes before the mac-loopback check 18:15:41 I also don't understand why the packet has HW_LSO set but not HCK_PARTIALCKSUM (assuming this packet was created to go over vmxnet3), mac_sw_cksum() does erase that flag after it does that work, but the assert is made before that runs 18:17:32 also it looks like MAC_LSO_EMUL also implies checksum fix ups, which makes sense, because it *has to* do that work since its generating a completely new chain of segments, so it has to generate checksums for all those, that's why it's `=` vs `|=` 18:17:45 but it wouldn't hurt for me to add a comment above that explaining what's going on 18:18:43 so for me, the big questions for me are: a) why is a packet come down the e1000g0 ill with HW_LSO enabled and b) why is mac_hw_emul() getting invoked 18:20:49 It seems like (b) could be a fuck up on my part, how we handled local packets and offlaods changed, and I decided that it was too much of a liability to not emulate offloads when going over loopback, as the IP forwarding mechanism and the TAP devices brought in all kinds of edge cases 18:22:22 my comment about the mac_hw_emul() call in mac_tx() says this is only getting invoked if IP-forwarding over mac-loopback is happening, but this is a generic portion of mac tx, and I'm not really sure if past Ryan was correct or if this was some slip up on my part when I undid all the checksum trickery around mac-loopback 18:22:58 is pkg uninstall graphics/nvidia considered generally safe? 18:23:15 it seems to be blocking some updates for me and I have no nvidia hardware installed 18:38:37 it should be 19:04:34 [illumos-gate] 15043 smbadm - need ability to read credentials from stdin -- Gordon Ross 19:42:02 [illumos-gate] 15026 libsec mistakenly assumes a SID is a group SID -- Gordon Ross 20:03:52 [illumos-gate] 15039 kernel initializes early GDT -- Dan Cross 20:07:59 I'll do the RTI build for 15077 once this build is done... 20:12:40 btw, is there way to serialize gmake build? (other than -j 1) 20:14:15 this build VM has 11G ram + 22G swap and oomkiller is having party there:D 20:16:53 gmake or dmake? 20:17:19 both should serialize with -j1 or no -j 20:17:25 our make also has -mserial 20:17:25 gmake 20:17:37 dmake is nice;) 20:17:51 but I'm really confused about both having an oomkiller, and what you need to build for an RTI with gmake :) 20:18:03 unfortunately that build system is rewritten to use gmake... 20:18:22 and no, thats not RTI build;) 20:41:50 jbk I was making a stupid joke :-P although I did play around with old IB on OI probably more than a decade ago 20:44:47 you say that, but I have a coworker that wants to do iSCSI over infiniband 20:50:12 jbk: okay, so pretty sure _part_ of this is my bug, introduced in my work for SmartOS OS-8027, where i reverted a bunch of the mac-loopback hardware-offload avoidance schemes (with the goal of better perf), but forgot to undo part of that work in mac_tx(). 20:51:32 However, I'm also thinking that logic in mac_tx() was never appropriate, it did correctly address the problems discussed in OS-7520, but it also was applying to situations outside of IP forwarding when it shouldn't 20:51:48 Or at least that's my current theory, and I'm going to test that out later today 20:51:52 jbk iirc I had iser and srp working with the comstar on ib, but those are ancient adapters now 20:52:19 However, I think there is potentially a second issue at play, where having multiple NICs on the same subnet with different hardware offloads, might somehow be broken 20:52:31 that's something I'll also try to dig into if I can 20:54:16 The good news is that if my theory is correct we should get some nice efficiency wins and maybe some perf bump 20:58:36 Even if I fix what I think is my bug, from what I can see in your crash, you are still going to have a problem because you are going to be sending a packet with HW_LSO down to e1000g0, and it's going to drop that on that floor because e1000g_retrieve_context() will return false since there is an LSO flag but no checksum flag. 20:58:50 But since it's TCP you probably won't notice at all 20:59:44 anyways, I gotta take a break, but should return to this later 21:19:28 KungFuJesus: it's safe 21:23:40 on the plus side, i've got vmxnet3s working w/ multiple rings now... so hopefully anyone using illumos in vmware should see some benefits soon 21:24:16 though it does feel a bit 'wrong' using camel case in the kernel :( (matching existing style here) 22:08:01 camel case was only for definitions coming from VMware, no? 22:17:28 yuripv: yeah 22:18:42 it looks like freebsd has made more changes here (while I didn't port any code from freebsd, I did use it as a guide to know where you reported the additional rings)