hi

Can someone show me the output of their `uname`?

; uname

SunOS

that's on OmniOS 151046 (LTS)

The complete `uname -a` is "SunOS web0 5.11 omnios-r151046-c6bc13f710 i86pc i386 i86pc"

and `uname -o` outputs "illumos"

[illumos-gate] 16206 lmrc can race against itself, causing a panic in lmrc_tran_start() -- Hans Rosenfeld <rosenfeld⊙gho>

danmcd: are you around?

Woodstock: yes

(pardon latency)

danmcd: it seems 16118 doesn't build without 16115

Ouch?

fenix illumos#16118

BUG 16118: Fix problems with libnwam communication between 32-bit and 64-bit applications (Closed)

↳ illumos.org/issues/16118 | code.illumos.org/c/illumos-gate/+/3188

fenix illumos#16115

BUG 16115: Want 64-bit libnwam (Pending RTI)

↳ illumos.org/issues/16115 | code.illumos.org/c/illumos-gate/+/3186

I built locally from scratch with 16118; did so before I actually pushed it.

Do you perhaps need to `make clobber` ? Is this a bit of a flag day?

i got a build failure in 'dmake check' as amd64/libnwam.so was missing

hm maybe

I'm about to merge with your just-pushed into illumos-joyent fix. I'll see if SmartOS build ALSO builds or not.

So I'm rebuilding -gate on HDC and just-merged illumos-joyent right now.

Oh damn.

I missed it.

==== cstyle/hdrchk errors ====

dmake: Warning: Command failed for target `lib'

dmake: Warning: Command failed for target `libnwam-nodepend'

dmake: Warning: Don't know how to make target `amd64/libnwam.so.1'

dmake: Warning: Target `check' not remade because of errors

The TYPECHECK target won't work if we're not building both librarierds.

*libraries

@Woodstock

Yeah.

So back it out?

If you delete $(TYPECHECK) from check then that should work and a general question then to follow up on the mail_msg we had with the clean build.

danmcd: it might be easier to just push 16115

It's up for RTI, isn't it?

yes

i wrote it originally, but marcel has taken care of review and RTI

Hang on.

so i can't just approve it :)

In OI for 2mos, huh?

You're pushing it though.

Thanks @Woodstock, and sorry for not seeing this when I built. (I checked mail_msg directly on the build box and was sloppy).

@rmustacc worth an e-mail?

[illumos-gate] 16115 Want 64-bit libnwam -- Hans Rosenfeld <rosenfeld⊙gho>

danmcd: Yes, it's worth two e-mails. One as a heads up. The second as a question as to how the build didn't fail for the submitter.

And to be clear, stuff like this happens, so just want to make sure folks are aware for the next time.

Can someone show me the output of their `uname`?

What options do you want?

(Also several people did yesterday and its in the logs after your ask, fyi)

What are you trying to figure out SamuelMarks?

danmcd - Trying to see if it still reads "SunOS" somewhere

Ahhh,.

Yes, `uname -a` on two different distros starts with "SunOS".

rmustacc - darn I went to bed; log link?

danmcd - SmartOS and OpenIndiana?

SmartOS and OmniOS

ANd log link: log.omnios.org/illumos

oh 2.5 hours later!

Thanks that's helpful

nomad andyf - Thanks

Okay, with 16115 pulled in no more `make check `breakage. Thanks @Woodstock, and sorry @carba for not catching this myself pre-push.

If there's something else you need, don't hesitate to ask SamuelMarks.

thx

I remember seeing a weird runner years ago that spun up a vm inside macos to run illumos for testing under github actions - what's the recommended CI tool for illumos testing?

rmustacc: do you know a method to reserve 32bit dma memory at boot, or alternatively force driver to attach as early as possible?

OpenIndiana

Woodstock: Not off hand no. I think I'd figure out if other devices grabbing from that 32-bit DMA region need to be or if we biased towards the wrong arena or not.

You can force attach early with some modifications I guess.

Otherwise, I think the longer term answer is IOMMU.

Woodstock ddi-forceattach(9p)?

The driver.conf file won't be read ahead of the driver normally being processed.

It also doesn't prevent detach.

It just forces it to reattach, IIRC.

hm

Other things that you can maybe do will depend on the BIOS/platform.

For example, we move config space up to be above 4 GiB on our platforms to save 32-bit MMIO space.

i see that mlxcx allocates gigabytes worth of dma memory, and lmrc attach is starving trying to get 37mb for SGL chains :)

So mlxcx supports full 64-bit addresses.

yes

So I think my question is why isn't it getting 64-bit DMA?'

is there a way to find out?

that is, whether it's 32bit or 64bit dma?

the driver looks like it wants 64bit dma

Probably by looking at the PAs and the the kmem cache or vmem segments?

Dunno.

There's a way. How painful I dunno.

lmrc could do 64bit dma, too, at least in theory. but i haven't seen a controller which switches that config bit on.

hm yes

Probably can walk all the DMA handles it allocated.

I don't know of anything off hand.

Woodstock: and then some :) we had a server where 64GB of ram wasn't enough for mlxcx

That seems problematic. Where's arekinath to ship us some DRAM?

Now I'm wondering about 64-bit DMA being enabled in the BIOS or not on my own test boxes...

thankfully, the integrator seems to have forgotten a few DIMMs, so once it had 256gb of ram, it was fine

(it had multiple mlxcx ports aggregated together + vnics, which wasn't helping with memory usage)

jbk: too many oversized rx queues?

well i think each instance is allocating <# CPUs> rings * a few thousand DMA buffers per ring

plus vnics on top of that I think with mlxcx will also allocate additional rings

yep, too many oversized rx queues at least for most workloads..

not sure offhand if the mlxcx driver lets the hw segment inbound rx packets or not

if not, jumbo frames would be even worse

always fun when dladm hangs for 10 minutes because it's blocked while the kernel (essentially) defrags (if only it had a graphic :P)

oh, more users here!

can anyone explain this? I have no idea where to start googling :) antranigv.am/misc/omnios_installation_err.jpg

do we support intel chips either E and P cores?

You mean with them mixed?

We should boot on it. The scheduler isn't doing anything special with it.

Though I have not had any such system that i've tried to boot on.,

antranigv: what hardware? the IDE errors suggest old (but it may be a SATA put into IDE emulation mode which you should probably turn off in the BIOS)

sommerfeld you are right! it was in IDE mode. we changed it to AHCI and now works all fine

ah, good that you figured it out...

cmdk needs some love...

damn this rge0 interface

I can't even see the ARP packets going out

antranigv I have a rge card it works, what problem do you have?

neirac nothing is coming out. I can't even see the ARP packets at the other end. lemme check the switch

antranigv if you see the nic with dladm show-phys, means the driver attached and should work, I'm using one right now in omnios

neirac rge0 ethernet unknown 0 unknown rge0; this is the output of dladm

Have you created an ip interface on top of it?

rmustacc yes! I can see it in `ipadm`

OK, then I'm a bit surprised that dladm says the link is in an unkonwn state (versus explicitly up/down), but given that it's not up, I guess it's not really a surprise then that you're not seeing packets.

rmustacc bad driver?

Don't know.

Not enough information to make even a speculative guess.

rmustacc how can I debug such issue? sorry, completely #NewHere

So what'd probably help someone is to gist/pastebin/whatever (please don't dump into the chat), the full output of ipadm and dladm.

I'll get to it. it's hard to do that when there's no inet :D

Sure, I understand. Or an image or something.

Just trying to make sure we understand the state of the system.

The other thing I'd see is if there is anything in /var/adm/messages from the rge driver.

okay, so I'm not in front of the display, I asked my intern to run some commands, make an image and send it. I will upload them all in a single page

I've never seen anything like this in the last 5 days that I've been playing around with illumos xD

It's not something I know right away either. So it's basically going to first be about confirming the basic set up, links plugged in, the driver internally called the MAC mc_start(9E) entry point, etc.

antranigv: You can always `grep rge /var/adm/messages` and see if rge(4D) whined at all? Long shot, but sometimes one gets lucky...

okay, so.

ipadm

dladm

dladm show-ether

dladm show-phys

grepping the messages

should I get any more info?

ping

test

Can you also grab /usr/lib/pci/pcieadm show-devs output antranigv?

That'll just let us confirm what exact rge device we're working with.

sure!

ararat can you run these commands in the server? send the images to me, I will upload them to a server

ararat to be specific, the file we're grepping (you were not here) is the following: grep rge /var/adm/messages

antranigv this is the output of prtconf -vp and this is the nic I have termbin.com/yodo check for Ethernet controller and name, mine is name: 'pci1462,7721'

If you're trying to get the name neirac /usr/lib/pci/pcieadm show-devs rge is a friendlier way there.

The name can be a little deceptive as that's defaulting to a device subsystem ID.

So there's a lot more variance in it.

rmustacc let me try that

No real need as all the info is in the output you had.

The device-name string is more useful there and the primary device id as opposed to the subsystem tells us what particular chipset it is.

rmustacc I don't find pcieadm in omnios is that in a package?

Yeah, pkg:/diagnostic/pci.

awesome 1/0/0 PCIe Gen 1x1 rge0 RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller

looks a lot nicer

ararat thank you!

here's the images

and as much output as I could

rmustacc turns out pcieadm was in packages, so we were not able to install atm

neirac looks like we both have the same card

but the model is the same? like the device-id ?

neirac yup!

neirac the revision is different

but the device-id is the same: 00008168

test

neirac we'll try rebooting, sometime that works :)))

ararat can you reboot the machine please?

what I see related to the error message regarding MSI is src.illumos.org/source/xref/illumos…n/io/rge/rge_main.c?r=0dc2366f#1731 , I don't see how to disable MSI in rge

sure

I don't see an error there.

neirac in BSD land, that would be using sysctls.

Just a note that we opted to use MSI interrupts, whichi is generally fine.

There is nothing in that to suggest there as an interrupt delivery problem.

rmustacc oh ok, just greped the first thing I saw in the picturs

ping still doesn't work

for .1

ararat I wonder if there's something in BIOS settings (AGAIN) that can help us, but probably not

ararat is there another network card on the desktop? or is that the only NIC?

rmustacc is there a way to check with dladm or other command if a cable is connected?

yhy

neirac the cable is connected for sure in our case, as rebooting into Linux, we can get an IP via DHCP. but I think the LINK is the answer to your question, no?

`cannot plumb rge0: already exists`

neirac: Ethernet PHYs don't usually distinguish between a cable being present and a carrier there.

Unlike an SFF tranceiver, there isn't a dedicated presence pin that I know of.

rmustacc thanks

I don't know what else to check based on the pictures, maybe rem_drv rge and add_drv?

antranigv run rem_drv rge and then add_drv rge, and check dmesg

That's not going to work while there's a valid ip interface.

I think we can delete the IP interface

and delete the if

and then do that

rem_drv requires reboot

rebooting

69.55/24

ararat_ still can't ping it

ararat_ what's the new state now, after you rebooted?

disabled?

yed

yes

can you try neirac 's suggestion now and run `add_drv rge` ?

it says can't driver set in system but cant attach

driver set but cant attach*

Unfortunately, using rem_drv manually removes all the PCI alias.

So a standalone add_drv won't get it to attach.

I'm not sure what this was going to accomplish, but I can try to go figure out the right syntax for you in a few minutes.

rmustacc danke <3

rmustacc wanna get this done soon, as ararat_ is still in the data center trying to have illumos up and running :DDD ararat_ is it cold in there? xD

a lil bit, it's fine

update_drv -i '"pcie10ec,8168"' rge

rmustacc I thought rem_drv rge only removed that driver

It does, but it removes all the aliases that were installed.

rmustacc then add_drive -i add that alias to /etc/driver_aliases and tries to attach ?

You would need to put that in the add_drv line again.

But in general, I also don't really remove/add the driver for this unless I have a theory for what's going on because you end up fighting a bit against packaging.

rmustacc is there any other way to debug when a driver attaches ?

I mean force to attach/reattach

Sorry, I'm not in a place where I can lead a debugging session. But generally with stuff like this I use mdb to understand the driver's state and then DTrace.

ararat_ after running `update_drv -i '"pcie10ec,8168"' rge` let us know what the output is. if there' no output, then send me a picture of `dmesg`

In terms of loading and unloading, modunload and modload will probalby show up. But if we don't understand anything, I'm not sure that an add_drv/rem_drv will do anything that different from a reboot.

at least one of m/i/P/p must be specified with -a amd -d

Ah, sorry, I forgot the -a.

So toss that in before teh -i.

failed to attach

ararat_ update_drv -a -i '"pcie10ec,8168"' rge

damn

Ah, sorry, one other typo there it looks like. Sorry, typing this out without a system to try this on.

update_drv -a -i '"pciex10ec,8168"' rge

Basically, the alais was pciex, not pcie. My mistake.

ahah

no error

ararat_ can you send me the output of `dladm show-phys` ?

rge0 Ethernet unknown 0 unknown rge0

link media state speed duplex device

damn it

today's not our lucky day