I have a trashtop that I would like to run a flavor of unix on. I see there are multiple forms of illumos, which should I use? It will be for mainly genral use stuff, not a daily driver, and seeing if I can get my code to run on it. (also possibily some packaging if it is avaible)

Dotz0cat: illumos has "distros" similar to how Linux does. If you're running a desktop system you might find OpenIndiana suitable. If you're running a server, probably OmniOS, and if you're doing mainly virtualization, then SmartOS.

I'm getting quite worried about the dead silence on illumos.org/issues/15302 - Our server vendor says the older cards are getting hard to get. They're talking boneyard at this point.

→ BUG 15302: broadcom 3108 kernel driver update (New)

Is there anything that can be done to push this along?

i'm currently working on a lmrc driver for the latest generations of megaraid controllers, but that probably won't help you with 3108

the part I don't understand about illumos #15302 is that it says the system is unstable, but there's no description of how that instability manifests

given that mr_sas complains about unsupported features in the HBA, I'd expect it to not attach at all

It boots, it runs, but it accumulates errors in the HBA's internal log.

eventually, when the host is rebooted (e.g. for patching) it refuses to boot until those errors are cleared.

I'll add that to the ticket.

interesting

I think I have some screen shots from when I was working with SuperMicro to try to fix the problem. If I find them I'll add them to the ticket.

let me know if there's anything else that's unclear, please.

(I'm trying not to clutter the ticket with all the side discussion so may have trimmed too much.)

e.g. I have screen shots of the entire process of clearing the error if you want to see them. I somehow doubt that is the case, however.

[illumos-gate] 15265 loader: add mechanism to set os console based on last input device -- Toomas Soome <tsoome⊙mc>

[illumos-gate] 15313 document pcitool register peek/poke mode -- Hans Rosenfeld <rosenfeld⊙gho>

[illumos-gate] 14956 nvmeadm list improvements for namespace management -- Hans Rosenfeld <rosenfeld⊙gho>

nomad: about issue with hba 3108 - did you try different firware IT for it ?

igork, if you mean "the latest from the vendor" then yes.

Most 3108s don't support IT firmware.

nomad: nope, i mean try to older different version and test it. also ticket missed firmware version

igork, I'll add the firmware version to the ticket.

We didn't try older firmware as that's not a good long term solution to the problem.

yeah, that doesn't like a solution at all really

nomad and list of others versions what you tested. if will be interest if your issue can be fixed by older firmware

The card is available to the kernel developer who wants to work on this problem. They're welcome to test that but, again, that isn't a good long term solution.

Without docs, it's pretty hard to help.

what's doing the SCSI<->SAS translation? as i noted in the ticket, the error seems to be related to that

err SATA

(monday :P)

is there an expander in there that might have firmware that needs to be updated?

ticket has been updated with firmware information.

jbk, sorry, didn't see your update until just now. If there's a way to ask prtconf for that information I'm happy to do it. The only thing we replaced was the HBA itself so anything else involved is still in there.

I'm pretty sure prtconf doesn't know, something else might.

everything in that particular host is SAS. It's a SSD-only fileserver.

there's sas/ses/etc knowledge in the FM topo, but I'm not sure if it includes things like this I'm afraid.

I'm willing to fish if someone hands me the rod and line.

It has been a very, very long time since I've been proficient with Solaris so I'll need direction from those who remember how it's done.

/usr/lib/fm/fmd/fmtopo will list the FM topology, and -V will make it verbose, but I'm not sure it'll tell you anything useful.

it's just more likely to know than prtconf.

that's a lot of output.

Who maks the system?

*makes

SuperMicro

What model?

(This is to try and answer your expander question)

should I paste the output of fmtopo -V to a pastebin?

Let me go find the quote to see what they call it.

SSG-6049P-E1CR36H 8600694390 / SERVER

and that matches the top of the tmtopo output.

hc://:product-id=SSG-6049P-E1CR36H:server-id=hvfs1:chassis-id=S287680X2712897/chassis=0

group: protocol version: 1 stability: Private/Private

resource fmri hc://:product-id=SSG-6049P-E1CR36H:server-id=hvfs1:chassis-id=S287680X2712897/chassis=0

group: authority version: 1 stability: Private/Private

product-id string SSG-6049P-E1CR36H

chassis-id string S287680X2712897

server-id string hvfs1

group: system version: 1 stability: Private/Private

isa string i386

machine string i86pc

smbios -t3 will also report that as well in the future.

But yes, I can tell you that server uses SAS expanders.

I used to use it a lot in a past life.

Simliarly, you should see some things in topo there, IIRC. Though the sestopo command would confirm it as well.

Especially if you only have a single 3108 in the box.

is this the expander by any chance? It's listed between the 3008 (the current HBA) and what appears to be the first SSD.

hc://:product-id=SSG-6049P-E1CR36H:server-id=hvfs1:chassis-id=S287680X2712897/motherboard=0/hostbridge=4/pciexrc=4/pciexbus=59/pciexdev=0/pciexfn=0/iport=0

group: protocol version: 1 stability: Private/Private

resource fmri hc://:product-id=SSG-6049P-E1CR36H:server-id=hvfs1:chassis-id=S287680X2712897/motherboard=0/hostbridge=4/pciexrc=4/pciexbus=59/pciexdev=0/pciexfn=0/iport=0

group: authority version: 1 stability: Private/Private

product-id string SSG-6049P-E1CR36H

chassis-id string S287680X2712897

server-id string hvfs1

group: storage version: 1 stability: Private/Private

initiator-port string w5003048023d621f1

initiator-port-pm string ff

right now it has a 3008 since we needed it to work and it was failing with the 3108.

The iports aren't expanders.

Those are the phys on the 3108.

this host doesn't have a 3108 in it right now.

Oh, well, then it'll be 3008s PHys.

That box with a 3008 was what we used for a while at Joyent, which is why it has some additional topo bits there like drive bay location info, etc.

would sas3ircu give the expander info?

I think if you just want expander info can you gist/pastebin fwflash -l?

Should I just put that in the ticket?

Wherever.

Those first two devices are expanders, IIRC.

thanks

I'll add that to the ticket.

Most expanders support SES (scsi enclosure services) which is how we enumerate and can update them.

Are the expanders build into the drive backplane, perchance?

Yes.

those error messages in the ticket are from the HBA log, correct? The CDB in the message is an ATA-PASSTHROUGH(16) CDB... so not sure why it'd be trying that if it's not a SATA disk

jbk, they came from the HBA, yes.

that host has 2 INTEL SSDSC2KB24 and 7 Micron_5300_MTFD and that's it.

IIRC, the Intel SSDs are directly connected to the motherboard. They're for rpool, everything else is in ssdp0 pool.

and the sense key appears to be basically a pass-through result as well..

nothing in the OS will send that CDB

do you have smartmontools installed?

evidently not.

that and some of the sg3 utilities are the only things i'm aware of that'd issue that sort of CDB

oh wait, those smartmontools. Yes, I do have that installed.

442 ? S 0:00 /opt/ooce/sbin/smartd --pidfile=/var/opt/ooce/smartmonto

I asked this yesterday but I did not get any reply that I saw. I have a trashtop that I would like to run a flavor of unix on. I see there are multiple forms of illumos, which should I use? It will be for mainly genral use stuff, not a daily driver, and seeing if I can get my code to run on it. (also possibily some packaging if it is avaible)

could be worth trying to turn that off to see if the errors go away

jbk, the card is no longer in that host.

oh

though in a month or so I'll be retiring an older file server and can, in theory, install the card there.

I'd prefer to hand the card over to someone who can update the kernel driver to match the new firmware features, though.

i wonder if it's maybe misidentifying the type of drive and issuing those CDBs

and the HBA can't handle the response

jbk, we needed the host to be functional so bought a 3008 and swapped the 3108 out.

SuperMicro said it was this warning in the card's log: 35992: 22-12-19,18:46:59 WARNING:Host driver needs to be upgraded to enable extended LD support

(on illumos, both SCSI and SATA drives 'appear' like scsi disks, and the SATA framework translates SCSI commands to SATA)

so smartd has to look at other bits to determine if it needs to send an ATA PASSTHROUGH command

(though our support for ATA-PASSTHROUGH(16) is also currently busted for most things)

Dotz0cat: If you want a basline system then probably illumos. The other Solaris options are generally based on that. (I speak in very broad swaths here). As for "which Unix", that's a much bigger question.

I'm not even up on the complete list of Illumos-based distros, let along all the Unix options.

I run FreeBSD and I'm happy with it. Take from that what you will.

I guess, what is a trashtop and do you care about a graphical console?

Most of the time when I hear people talking about a trashtop it's just old'garbage' laying around. E.g. 10+year old laptop they had in storage.

Generally I would say that making use of older hardware is more Tribblix territory than the other distros

Is there something that you've done differently on x86?

10 years ago at this point is inbetween Ivy Bridge and Haswell.

Yes a trashtop is a old garbage laptop. It does endup being x86_64

Although I would regard laptops as being generally not an ideal target for illumos

or desktops in general;)

It's not specifically age, but Tribblix tends to handle resource-constrained (especially small memory) systems fairly well

Although looking around my lab, the fact that my current SPARC box is one of the most modern devices might be a bit worrying

Ill look into tribblix. Part of the fun is getting stuff that shouldn't work to work

[illumos-gate] 15476 csh should be mediated in the packaging -- Dominik Hassler <hadfl⊙oo>