Did it work @neirac?

danmcd do it now, I lost power last night, so resuming building now

danmcd there is a problem with pkgsrc, I just installed the lasted smartos iso to a vm, then after installing python311 to create zones with ansible, pkgin is broken : ld.so.1: pkgin: fatal: libssl.so.1.1: open failed: No such file or directory

neirac, just reinstall the bootstrap

CmdLnKid thanks!

welcome

iirc atm there is a pkgsetup script that comes with smartos that may help you too

don't have access to the machine at the moment

found it by pkg[tab][tab]

hello all, I have simple question about replace the ‘BAD’ disk? I have two spares disk in my pool, currently show the status is incorrect, had a spare disk in ‘INUSE’ status, how can I replace the ‘BAD’ disk safely? thank you for the advice, here is status on the link pastebin.com/Q4AJpYQ8, big thanks in advance

danmcd build worked perfectly today, root cause should havee been that my smartos-live repo was too old

tozhu maybe this helps docs.tritondatacenter.com/private-c…ud/troubleshooting/disk-replacement

neirac: Thank you very, I’ll take a look

by the way, is there any docs introduce how to build triton image from source? I have got a new machine, would like to try build the triton from source

neirac: Phew. :)

I've had enough explode on me this week.

one thing I've been wanting to do is provide the option where (auto) replacement promotes the spare to a regular disk and the swapped disk is added as a spare (assuming the disk being swapped has had a spare kicked in)

instead of reverting the used spare back to a spare

so you don't have to do yet another resilver

jbk: a big thank, according to instraction, all of the disk are ONLINE, and no failed disk, why the spare disk became a part of the mirror-4?

did you zpool clear? that'd lose any checksum errors

but fmadm faulty might show

or fmdump -e

(fmdump -e shows all the events, fmadm faulty should show if the system determine if events suggest there's a problem)

let me check the status

I have not run ‘zpool clear’

when run ‘fmdump -e’, it show ‘Nov 17 14:53:22.1049 ereport.io.scsi.cmd.disk.dev.uderr’, where to check the detailed error for the device?

either add -v or -V (can't remember which offhand)

to fmdump

it should show the sense key and asc/ascq values

have got the detailed value bye fmdump -e -V at here pastebin.com/2MdRJ35n

could repaire the status by reboot system?

you can just remove the spare if the existing disk is online

it looks like that disk (is it connected via usb?) is having issues with reporting its write cache status

thanks for the advice, all of the disk are sas disk, not USB (the boot disk is a usb flash disk)

those errors are probably then complaining about checking the write cache status on the boot disk

you could take the cXtYdZ name of the boot disk and look at what the symlink in /dev/dsk points to..

if you want to be sure

but the device path for those suggests usb (due to 'hub@1' being in the path)

okay, thank you very much

I’m not sure if this is a bug, if all disk is ‘TOSHIBA’, then the display is correct, the ‘TYPE’ column display ‘SCSI’, I have replaced a BAD disk with ‘SEAGATE’, there are ‘SEAGATE’ and ‘TOSHIBA’ disks on same SAS port, the ‘TYPE’ column diskplay ‘-‘ for ‘TOSHIBA’ disks, all disks are SAS

here is the screen output pastebin.com/yAypLm8w

here is othere machine screen output pastebin.com/GVW8bdd4

jbk, should I remove ‘c1t500003970852BA8Ed0’ or ‘c1t5000039708526F5Ed0’ for this case? I’m a little bit confuse, which one should be ‘removed’ ? thanks in advance

and last question, I have 10x U.2 NVMe disks, and want to get best balance between ‘reliability’, ‘capacity’ and ’disk lifetime’ ; what’s the level is adviced? spare disk and log disk is adviced ? best wishes, is any advice?

Ultimately it depends on the performance and capacity tradeoffs you want to hit.

rmustacc, I want to get the performance and reliability

is the mirror adviced? need to create slog and spare disk for the case?

neirac: Which image did this happen to you with?

and, I have a disk can’t not remove / repaire, don’t know how to repare it, any idea/advice? here is the details pastebin.com/Wmt5W7tf thank you very much

I perfer mirror, but not sure if slog disk and spare disk is needed if all disk is U.2 disk, someone told me, the big size is for long life for SSD

tozhu: If you have all SSDs there's not much point in having a dedicated log device.

if long life require big size, the raid-z2 is better than mirror, but mirror is more reliability in tranditional known, so I’m confuse

There's always a ZIL. If you have HDDs then you can get some benefit by having a dedicated SSD log device. But if it's all SSD already, a dedicated log device doesn't get you anything.

But you should *always* have spares if you're doing anything serious. Even with mirrors.

bahamat, rmustacc thank you very much

Is there a video / doc on how to build triton service images?

I think I remember there being a "build smartos" thursday talk

Smithx10: I think the office hours was recorded.

Do you by chance know the URL to the video/

would you please advice how to repare this? pastebin.com/Wmt5W7tf I have tried to remove the disk, but it reports Pool busy, so I don’t know how to do next step

a big thank :-)

Smithx10: No, but I can check

tozhu: I think this might be your clue: removal may already be in progress

if so, how to check the removal progress ?

I'm not sure. You might be able to get something out of zdb or mdb.

but run ‘zpool remove -s zones’ command, it reports message: cannot cancel removal: operation is not in progress

try zpool detach zones c1t5000039708526F5Ed0 (assuming the zpool status output hasn't changed since your pastebin)

jbk, thanks for the advice, I’ll try it, thank you

jbk, it works, thank you very much for the help

@bahamat looks like they are on youtube youtube.com/@TritonDataCenter

:)

I figured something like that

np

bahamat: is the jenkins that does all these builds public?

Smithx10: The hvm/lx images aren't built in Jenkins yet, but our Jenkins is public, yes.

nice, I'm trying to get a build pipeline setup internally for sdc-service images So we can experiment in our test DC before nagging folks.

Was hoping to maybe expand the sdc-cloudapi plugins to cover more of the api

I'm not against it.

The plugins used to be able to modify anything. We'd pass in the entire JSON object and let them do whatever before sending it for provisioning.

I think it was rearchitected for docker

Yeah, having our own plugin and keeping some of the custom behavior their is easier to maintain vs keeping a fork of cloudapi

there*

what do you want to expose?

bahamat, just download yesterday latest iso and installed today

neirac: So you're saying that it's pkgsrc-tools that's broken?

bahamat what I did was to install python311 then after that pkgin was broken

in the global zone?

bahamat correct in the gz

Smithx10: if you done the build process for Triton from source, Please record the step to doc, and share it, Thank you, :-)

bahamat: is there a jenkins file or github action for sdc-cloudapi that I can reference to see what image / build zone image we use?

github.com/TritonDataCenter/sdc-cloudapi/blob/master/Jenkinsfile :) Looks like it uses triton-origin-x86_64-21.4.0 master-20220322T012137Z-g9382491

yeh a newer pkgsrc bootstrap will include the latest pkgin that has a boatload of fixes for upgrade scenarios

Tried a make all and hit a node-gyp import sys; print "%s.%s.%s" % sys.version_info[:3];

Quick guess is the python version ?

Yeah, you'll need python2 for node-gyp

Is triton-origin-x86_64-21.4.0 master-20220322T012137Z-g9382491 a build zone? It looks like it's missing a few things.

Should I use this old version of pkgsrc.joyent.com/packages/SmartOS/…trap//bootstrap-2019Q4-tools.tar.gz

Were you just trying to test a change to cloudapi or setup something close to our Jenkins setup?

similar to your jenkins, I looked through the groovy lib, we use gitlab

I think I got a make all to run cleanly, trying to find where the image + manifest are output too

jperkin, neirac: OS-8500

Smithx10: We have special jenkins-agent images on updates.tritondatacenter.com

Smithx10: From your headnode: updates-imgadm list name=~jenkins

nice

Smithx10: We use the GitHub Branch Source Jenkins plugin to hook into our GH org and auto import everything.

make all worked without issue on the latest jenkins image. Will that create artifacts like the image / image json / tar.gz ?

Smithx10: This is what most builds use: github.com/TritonDataCenter/jenkins…s/joyBuildImageAndUpload.groovy#L20

when run from Jenkins.

The `bits-upload` part will want to upload it to manta, so you'll want everything except that.

unless I override the env vars for the MANTA_URL etc right?

Some builds don't use BuildImageAndUpload, so you'll need to check the Jenkinsfile for what they do instead.

gotcha

Correct, if you supply your own manta environment variables then it will go to whatever you've configured instead.

Thank You

Got stuck on pkg_add gist.github.com/Smithx10/b5a5fab267f226d2445fdcd024ea963b

chroot resolvers :( our company blocks 8.8.8.8 lol

is that hard coded into the chroot?

not sure, I'm grepping now seeing if I can find an override

yea

bahamat: I'm trying to edit buildimage to copy over my resolv.conf but it doesn't seem like changes to that node file are taking effect

Confusingly, there's multiple copies of buildimage, you need to make sure you're using the right one.

ahhh ok

Or possibly all of them.

bingo, /opt/tools/bin/buildimage

success