No hockey tomorrow, just let me know what to do, im a bit confused

maybe ill try zdb -dddd zones/$dataset

for that dataset, us-east-storage.solutions.iqvia.com/bruce_dev/public/zdb-output-dd and us-east-storage.solutions.iqvia.com/bruce_dev/public/zdb-output-dddd

so it looks like..... from that output.... there are a lot of ZFS plain file and ZFS directory that write 512 bytes but take up a 128k block "47087 1 128K 512 0 512 512 100.00 ZFS plain file"

Yeah, that's what it looks like.

Percent empty: 78.749621

So..... what is writing out the large amount of /system/contracts?

is that normal?

has a list of du --inodes sorted

Well a contract is a group of processes

SMF, for example, each service runs in a contract.

What's that behavior like?

Does every process that spins up in lx create a directory / file in contracts?

No. Having 918k of them seems like way too many

This box has been up 00:42:06 up 224 days, 4:37, 4 users, load average: 0.23, 0.26, 0.25

I mean maybe...

but for example one of my LX zones, I've zlogin'd to it. The zlogin is under a contract

prstat shows Total: 33 processes, 301 lwps, load averages: 0.23, 0.26, 0.26

dhcpagent, which is running out of /native and is spawned by hypervisor is in another. All other processes are grouped into a single contract.

20220505T001410Z

Any ideas on what could change that behavior ? Is it safe to rm -rf /system ?

No

But if you reboot the zone I think it'll clean it all up

I'll ask them to bounce

Guessing /system/contracts doesn't clean up ?

I think it should.

Any logs I should check to see if its having issues doing that?

I don't think so. IIUC that whole directory is managed by the kernel.

/system/contract is backed by ctfs.

So it's entirely synthetic.

So I don't think there's anything to do from a user perspective there.

But it seems odd to me that there'd be 918K files in it.

Not disagreeing there.

But being ctfs backed, a reboot should clear that up, shouldn't it?

Probably, yes.

Is this causing a cascading problem?

Though probably I would try to figure out what's in there or take a dump if you're going to reboot.

Well, this stemmed from trying to find out where all the space went.

I do not believe that any zfs space is used by that file system.

The zone is using about 13G, but is consuming about 20G

But we can check the underlying impl.

But I think every ctfs node is anonymous kernel memory.

Above there is a link to the sorted inodes on the lx zone

But since you transition to another mount point I don't think that would be backed by zfs.

I guess we could just inode count x 128 k and see if it even makes sense

The ctfs_mount doesn't have a hold to the underlying file system that's immediately obvious.

The zdb output has a lot of empty looking files that don’t show the path

Errr not files *. Objects

I've spent very little time so could be something else ultimately.

was on my phone, jsut got back on the lt

zomg. since it's a very self-contained golang binary program, I can just pull in the package from nearly any existing zone:

pkg_add pkgsrc.smartos.org/packages/SmartOS…2Q4/x86_64/All/tailscale-1.34.2.tgz

Got the paths of the blocks with zdb us-east-storage.solutions.iqvia.com…ruce_dev/public/zdb-output-dddddddd

Does a symlink take up a recordsize in zfs?

Yeah, it's a file

Still not 100% certain whats taking up all the space, heading to bed tho

Worked with some folks over in #openzfs and found out that a lot of space was stuck in the "on delete queue" via zdb

looked into that, and restarted "promtail"

which is some grafana logger rotatter thingie

im guessing promtail was keeping them open

Not sure if its LX, or Just Illumos, but some windows folks grafana/loki #3668#issuecomment-1252441103 ran into this

sigh

I run grafana on native but not promtail, sorry, so can't vouch for it being an issue on illumos

Smithx10 - do you have the promtail config?

neuroserve: gist.github.com/Smithx10/e598adf082d366c4f4fd883b602fb9b0

thats basically two logs (/var/log/messages and /var/log/haproxy.log)

Smithx10: i wonder if it periodically would stat the fd it's tailing and if it sees the link count drop, if it'd stop tailing the file or not..

it seems like that would solve the issue... but also not something i've ever tested...

Smithx10 : do you know the promtail version, too?

version 1.6.1

ok - I've got 2.5.0 - current seems to be 2.7.1

Whats the command to get the SN for a disk ? SCSI c3t5000CCA27077CE29d0 HGST HUH721212AL4200 11176.00 GiB no no

The most reliable way is iostat -En

diskinfo -P ?

`diskinfo -c` can, but often times can't find it. It was never clear to me why, but it's apparently because they enumerate disks in a different way.

Anyway to light it up from within SmartoS?

trying to blink it so it can eb replaced

Yeah, but it's not straightforward.

yeah, i notice with my internal nvme disk that diskinfo doesn't show the serial.. but i think that might be because i'm using a custom topo map that doesn't enumerate the location right now

Smithx10: Ok, so use iostat -En to get the serial number of the device

Then use /usr/lib/fm/fmd/fmtopo to dump devices. Find your serial number to get the enclosure and bay number

Then use /usr/lib/fm/fmd/fmtopo | grep indicator and look for your enclosure/bay to get the fmri of the device

Then use `/usr/lib/fm/fmd/fmtopo -P facility.mode=uint32:1 <fmri>?indicator=ok2rm` to turn the light on

to turn the light off, do the same command but use uint32:0 instead

Then use /usr/lib/fm/fmd/fmtopo to dump devices. Find your serial number to get the enclosure and bay number dont see the serial number, do I need to turn up the output?

As far as I know, there's not a way to increase the verbosity.

The serial is usually part of the FMRI.

I have an m2 SSD in my headnode, and I don't even see anything that might be it in the fmtopo output :-(

this is what i get from topo

So starting on line 567 is scsi-devices

But that doesn't even have serial numbers in the FMRI...

So, no idea...

./shrug wonder if I can do this in super micro oob

Yeah, I don't know. rmustacc would probably be the best person to ask to better understand this stuff.

He's the one that showed me how to drill down to light up the light.

But fmtopo often defies my expectations. I really don't understand it well.

I gotta take off. Back in about an hour.

lol, this seems like it shouldnt be so hard rofl

Smithx10: Can you add -V to your fmtopo command?

rmustacc: us-east-storage.solutions.iqvia.com/bruce_dev/public/topo-v

So this is a platform Joyent had a map for. So diskinfo -P should have told you the slot.

Did it not?

rmustacc: gist.github.com/Smithx10/8adfff35cc52a1ec8088365be5f64c88

Curious, that's odd.

Right, this is a system we specifically designed all this support for back in the day.

Is ses not reporting anything on that system?

what is "ses"?

scsi enclosure services

How do I tell?

basically it's a device that lives on the SCSI bus/fabric that manages an enclosure.. so it's what handles things like indicator/fail lights, power for a disk bay, fans, ...

Do you have anything like a /dev/es/XXX? IIRC.

You can ask sestopo for more info there. I don't know why fmtopo doesn't have the ses enumeration.

rmustacc: us-east-storage.solutions.iqvia.com/bruce_dev/public/sestop-ses0 us-east-storage.solutions.iqvia.com/bruce_dev/public/sestop-ses1

i've been told that the SES devices can be a bit loose with the spec

and that there's a lot of 'we'll just document how it works today vs. how it should work'

:(

gotta boot up linux to run lldp during our next expansion :(

you use aggrs, right?

Smithx10 : still that lldp problem?