kill(893, SIGSEGV) = 0

Received signal #11, SIGSEGV [default]

siginfo: SIGSEGV pid=893 uid=0

yes then it segfaults

can a not existant file cause a segfault?

it seems to be sending itself a signal?!

why does it do that

ah, I found the problem

my mac sends env variables that cause this

locale things?

limiting AcceptEnv limits things

yes locale

that still shouldn't segv?

One would hope not

Can you undo your AcceptEnv thing and "ssh -t $host env" maybe?

sure

LC_CTYPE=UTF-8 might be the problem

no value in any LC variable should cause a segfault

true

if you "LC_CTYPE=en_US.UTF-8 ssh -t $host bash" does that work

yes

why would it do that

well even GNU has to work sometime. It just has no warranty when it will do which

i mean if there's no "UTF-8" (or the variables are empty), just use "C", why segfault

rzezeski: as my brain is mush at this point, if you happen to recall this stuff: github.com/illumos/illumos-gate/blo…mon/io/mac/mac_client.c#L3668-L3676

I'm kinda certain that on 3676 that should be |=

but have lingering doubts

(i have a panic where github.com/illumos/illumos-gate/blo…c/uts/common/io/mac/mac_util.c#L977 is getting tripped)

and in this case the adapter isn't advertising lso or any hw checksum

jbk: so, I think you might be right about the |=, but I would want to double check that tomorrow with fresh eyes

however, I think that's separate from your panic

as that assert is checking that the IP stack requested the IP header checksum offload along with LSO

in your case you are saying the part advertisers neither offloads, which makes it very sus that you are in mac_sw_lso

if you can put the panic up somewhere I could take a look at it

let me see if I can figure something out

is this a customer thing?

no, just a place to upload it

ah okay

it's just a debug build of illumos-gate running under ESX w/ an emulated e1000g nic

okay, that's good to know, but I'm not going to have the cycles to stand that up

i might be able to do something tomorrow

umm..

ok.. i'm completely confused..

oh.. that's confusing.. dblk_ts have a number of fields (cksum flags) uninitialized

while others are

is this a vanilla kernel? because that sounds like it could be UAF

anyways, I'm going to bed now, but if you can share a panic tomorrow I'll look

Good morning.

morning

i believe kmem uses a different pattern for freed memory..

i looked at what allocb() does (because it was confusing the hell out of me)

it's using some kmem caches, but it doesn't initialize all of the fields

so some of them are set, some of them have the uninitialize pattern

deadbeef is the free pattern, baddcafe is the uninitialized pattern, and feedface is the redzone pattern

Has anyone here tried VRML over Infiniband?

jbk - right, I usually have to look at umem_debug(3MALLOC) to remind myself, since it's the same as kmem

i don't think many people have used infiniband on illumos.. we have support for it, but I don't think there are drivers for any recent cards

at least for the infiniband functionality

(some cards e.g. mellanox ones work fine as regular network cards)

Did some spelunking on your panic jbk. There interesting bits thus far: 1) the mblk does indeed have HW_LSO, but nothing else (which is invalid), 2) the mblk is larger than e1000g0's MTU, 3) the ill this mblk is going over DOES NOT have ILL_CAPAB_LSO set (it does have ZC set, not sure if that's normal or not), 4) the e1000g0 mac DOES NOT have HW LSO or CHECKSUM enabled, 5) vmxnet3 ALWAYS provides HW LSO and CHECKSUM capab

it's almost like this packet was created to go over vmxnet, but instead ended up on e1000g0

they are on the same subnet--as i mentioned i threw that on the VM so I didn't have to use the console while testing out vmxnet3 improvements

so it's a bit unusual, but something that should be valid

granted, the chances of having real hw these days that doesn't do at least checksum offload seems slim to none

so it's probably a very niche corner case

Sure, but what I'm saying is for some reason a packet that seems like it should have gone over the ip interface on vmxnet3 is instead going over e1000g0

although given that vmxnet3 exposes partial checksum unconditionally it's very odd there is no hardware checksum flag turned on

do you have a modified vmxnet3?

oh wait, I just noticed vmxnet3 doesn't unconditionally expose LSO, but I don't see any way to easily check the state from a dump (this is a really weird way to code this)

their driver.conf file is also weird :)

if you changing this code you should change it so that it does the `vmxnet3_getprop` somewhere else and sets a flag in the software state, that way we can examine this state from a dump

but I did verify that your e1000g0 has no hardware offload enabled

yeah.. since it's virtual, not too surprising

like, I don't know if there's some confusion somewhere in the stack when you have multiple interfaces on same subnet with differing hardware offloads

jbk: were you running this ssh session for a while before the panic? Or was it the moment you try to ssh into the guest?

I guess it could work for a period before an LSO packet is generated

yeah it's a while

it was annoying because the assert would usually trip right as I was trying to look at something :)

oh wait, duh, this is hitting mac-loopback, wait wtf is going on

well the assert won't trip until you cause an LSO segment to be generated

so the session will work for a while and then boom

i think (just while doing this work) i'll take out the asserts... at first I had thought it was maybe the bit I pointed out last night (which sounds like it is a legitimate, but separate issue)

which is why I pinged you (since it looked like you were the most recent visitor to that part of the code)

richlowe: andyf I could confirm the update problem KungFuJesus was reporting. They are indeed caused by system/library/processor still being incorporated and not having a new version. uninstalling the package solves this. It should however either be obsoleted or not incorporated anymore

I wouldn't take out the assert, it's a correct assert. You might run into even weird behavior. I don't understand why this is invoking the mac_hw_emul routine, that might be a much larger bug, as I thought it was only being invoked when about to traverse mac-loopback, but in this case we are at code which comes before the mac-loopback check

I also don't understand why the packet has HW_LSO set but not HCK_PARTIALCKSUM (assuming this packet was created to go over vmxnet3), mac_sw_cksum() does erase that flag after it does that work, but the assert is made before that runs

also it looks like MAC_LSO_EMUL also implies checksum fix ups, which makes sense, because it *has to* do that work since its generating a completely new chain of segments, so it has to generate checksums for all those, that's why it's `=` vs `|=`

but it wouldn't hurt for me to add a comment above that explaining what's going on

so for me, the big questions for me are: a) why is a packet come down the e1000g0 ill with HW_LSO enabled and b) why is mac_hw_emul() getting invoked

It seems like (b) could be a fuck up on my part, how we handled local packets and offlaods changed, and I decided that it was too much of a liability to not emulate offloads when going over loopback, as the IP forwarding mechanism and the TAP devices brought in all kinds of edge cases

my comment about the mac_hw_emul() call in mac_tx() says this is only getting invoked if IP-forwarding over mac-loopback is happening, but this is a generic portion of mac tx, and I'm not really sure if past Ryan was correct or if this was some slip up on my part when I undid all the checksum trickery around mac-loopback

is pkg uninstall graphics/nvidia considered generally safe?

it seems to be blocking some updates for me and I have no nvidia hardware installed

it should be

[illumos-gate] 15043 smbadm - need ability to read credentials from stdin -- Gordon Ross <gwr⊙rc>

[illumos-gate] 15026 libsec mistakenly assumes a SID is a group SID -- Gordon Ross <gwr⊙rc>

[illumos-gate] 15039 kernel initializes early GDT -- Dan Cross <cross⊙oc>

I'll do the RTI build for 15077 once this build is done...

btw, is there way to serialize gmake build? (other than -j 1)

this build VM has 11G ram + 22G swap and oomkiller is having party there:D

gmake or dmake?

both should serialize with -j1 or no -j

our make also has -mserial

gmake

dmake is nice;)

but I'm really confused about both having an oomkiller, and what you need to build for an RTI with gmake :)

unfortunately that build system is rewritten to use gmake...

and no, thats not RTI build;)

jbk I was making a stupid joke :-P although I did play around with old IB on OI probably more than a decade ago

you say that, but I have a coworker that wants to do iSCSI over infiniband

jbk: okay, so pretty sure _part_ of this is my bug, introduced in my work for SmartOS OS-8027, where i reverted a bunch of the mac-loopback hardware-offload avoidance schemes (with the goal of better perf), but forgot to undo part of that work in mac_tx().

However, I'm also thinking that logic in mac_tx() was never appropriate, it did correctly address the problems discussed in OS-7520, but it also was applying to situations outside of IP forwarding when it shouldn't

Or at least that's my current theory, and I'm going to test that out later today

jbk iirc I had iser and srp working with the comstar on ib, but those are ancient adapters now

However, I think there is potentially a second issue at play, where having multiple NICs on the same subnet with different hardware offloads, might somehow be broken

that's something I'll also try to dig into if I can

The good news is that if my theory is correct we should get some nice efficiency wins and maybe some perf bump

Even if I fix what I think is my bug, from what I can see in your crash, you are still going to have a problem because you are going to be sending a packet with HW_LSO down to e1000g0, and it's going to drop that on that floor because e1000g_retrieve_context() will return false since there is an LSO flag but no checksum flag.

But since it's TCP you probably won't notice at all

anyways, I gotta take a break, but should return to this later

KungFuJesus: it's safe

on the plus side, i've got vmxnet3s working w/ multiple rings now... so hopefully anyone using illumos in vmware should see some benefits soon

though it does feel a bit 'wrong' using camel case in the kernel :( (matching existing style here)

camel case was only for definitions coming from VMware, no?

yuripv: yeah

it looks like freebsd has made more changes here (while I didn't port any code from freebsd, I did use it as a guide to know where you reported the additional rings)