[illumos-gate] 14548 snoop has strange bedfellows -- Joshua M. Clulow <josh⊙so>

hey gang, just curios if anyone knows if its possible to modify on the fly the zfs_arc_max? we realised that some of our problems are due to the system running out of memory (even though the vm's only account for 90Gb of 256GB, roughly 150Gb are being used by zfs arch cache. When a new provision is attempted a dladm process will hang (because i40e required 8gb per vnic of non-contigious memory; at least thats our understanding so far) ... thanks

for the help!

Hi all. I have a short question regarding ZFS: is it possible to force a release of the used used memory by ARC Cache without risking data corruption? On our systems, we see that ZFS File Data is using up to 57% of RAM.

GeorgePocky: you can try zinject -a

GeorgePocky - create a large file in /tmp - something like `mkfile 100g /tmp/fill && rm /tmp/fill`

However, generally it is better that memory is being used for ARC than sitting spare - the system will release it when there is memory pressure

in theory yes, it should be released, but from what i saw, RAM usage is 97.3% used and memory isn't filled up

and in combination with i40 10G network cards when if you need to create a vnic it will use up to 8GB of contiguous RAM, all the zones running (kvm/bhyve) will start to have intermittend access issues

Ouch, that's a big overhead for a VNIC. Which distribution are you using?

sensille: It looks like it will not release also the memory used by ZFS File Data on the system

andyf: we are using smartos

and yes, large file creation in /tmp does the trick

GeorgePocky : which PI are you using?

thanks for the idea!

I used to have to do that to boot a new VM when the system had been up a while, but there were changes that improved things there a lot and I haven't had to do it since. I don't have 10G network cards though.

physically contiguous RAM ... the i40e driver really needs to be improved

multiple PIs, even the latest ( 20220825T001415Z)

If you are starting memory hogs (like bhyve etc), then you should have max arc setting in place to reserve memory for those apps.

With default, the arc should still get freed, but this will take time...

In illumos, how can I admin down up an aggregation?

I have 2 interfaces in the down state apart of the same aggr... it was flapping in dmesg

tsoome - bhyve has the memory reservoir facility. I'd use that before starting to tune the ZFS ARC.

i have some VMs that have a reservation, so they always boot

...straight away, without having to wait for ARC reclaim

ou

Smithx10 you want to check why interfaces are down, you may need to use fmadm to clear them

Would I see them down in fmadm faulty?

if they are faulted, it would mean they are disabled, yes

I only have a "Oct 17 2020 3c342e58-1556-c667-d207-fcfda8c8b835 SUNOS-8000-J0 Major"

hm, and dladm show-phys ?

I see. that fibre?

yea

The probem in my situation is, I have a network team at work "_"

that may be problem, indeed;)

both links down at the same time hints about human error

So I'm just trying to make sure the OS isn't sitting in some weird spot waiting for me to clear something or reset the aggregation or something

Yea... at my job.... they seem to replace cables at an extremely high rate (and never test them after pulling them) lol

aggregation should get online as soon as you have physical link

I can understand why folks end up on the Public Cloud :(

I'd get physical inspection first - are cables attached, what is the status of leds

Cool, I sent that e-mail / ticket in already

from that log, I'd say your ixgbe1 was offline before

and then something did happen about ixgbe3 - and as you see, when ixgbe3 is online, aggregation is also "up"

yeah

BTW your only fmadm faulty is for overlay, which fires whenever your headnode reboots and your CN does not.

Smithx10: ^^^

And ixgbe1 is Just Down... (and yeah I remember your "network team" stories. WTF they aren't together with systems is _beyond_ me.)

Thanks danmcd

(When I flush Kebecloud's nodes, I reboot all the CNs first, then once they're happy, I reboot the HN, and once the HN's happy I `fmadm clear ...` on the CNs.)

andyf: did the reservoir facility ever get added to the omnios bhyve brand / zadm? Or did you just set that up manually for a few VMs

Yes, it's in the omnios brand - I think it made it into r151042

ty

Folks from the DC are thinking the optics are getting too hot. Do we have a way to measure an interfaces temp?

Manually, but not automatically.

I never rigged that part of the transceiver parsing up, sorry.

But you can manually dump it and figure it out.

Along with getting the other errors.

What's the transceiver? That should be in topo.

[illumos-gate] 15036 portfs wears inadequate pollcache disguise -- Patrick Mooney <pmooney⊙pc>

I'm not too familiar with fmtopo, what should I run?

Run /usr/lib/fm/fmd/fmtopo -V $FMRI , where one of those FMRIs in your gist should be used for $FMRI.

The -V will expand the goodies, and likely give you some answers.

Just leave out the $FMRI and gist that?

Or run /usr/lib/dl/dltraninfo -v

rmustacc here is the output of dltraninfo -v , gist.github.com/Smithx10/db4aad342671078c4b94e209a44c3292

us-east-storage.solutions.iqvia.com/bruce_dev/public/topo-out topo out

I don't see any temp info in -V, but I assume there is a switch for dumping all of the data?

It's like your transceivers aren't reporting temperature (at least by my reading... rmustacc may see something I'm missing).

danmcd: As I mentioned the parsing isn't there for diagnostic info there.

Based on the transceiver type we can look up where in the raw info it'll be and get it manually.

If you use -x, you'll get the raw hex and we ca go look at the spec.

rmustacc: us-east-storage.solutions.iqvia.com/bruce_dev/public/topo-x-output

I mean -x to dltraninfo, not topo.

That dumps the raw hex and you can go double check the specs on what offsets that'll be at.

Where do you usually grab the spec from?

In this case you want SFF-8472.

But get the dltraninfo -x and we'll take it apart.

Smithx10: So by my read of the data and SFF-8472 that doesn't support digital monitoring.

ahhh ok

So it's possible it's just lying, but because it doesn't claim it does implement the monitoring of SFF-8472 we don't try to get that i2c page.

The output of dltraninfo -x, what kind of table / or how would I know how to parse this?

Because this is an SFP+/SFP28 device you would go to SFF-8472 which is publicly available and look at the bytes.

That's what I did internally in the nvlist.

is there a way for svcs(1) to list ALL dependencies for a given service? -d seems to list only ones explicitly listed in the manifest, i would like it to show dependencies of dependencies too (i hope that made sense)

yuripv: I think, today, you would need to recurse yourself

But that would be a neat RFE, a "svcs -rd" that was able to create a ptree-like output

ok, or is there another way of checking if some service is supposed to start after some other? need to check if my service starts after identity:node

rmustacc: We have this optic in a linux machine and I read here: docs.nvidia.com/networking-ethernet…-hardware-information-using-ethtool that "Optical diagnostics support" would come back with yes, but I guess thats where the optic may be lying

yuripv: Do you want to know if it _should_ start after, or if it _did_ start after

ethtool on linux didn't return a "Optical diagnostics support " field at all with ethtool -m

both, i guess :) just seeing as if my service is starting too early, and wondering if i should just add explicit dependency here

Smithx10: So if it did have support then I wouldn't expect the field not to exist.

yea

:( shucks

So we are seeing the same thing from the transceiver.

Sorry.

I'm wondering how valuable this info is

Maybe in the future we should by optics that do DOM

yuripv: svcprop '*' | awk '$1 ~ /start_method_timestamp$/ { gsub("/:properties.*", "", $1); print $3, $1 }'

There are definitely ones that don't implement the digital monitoring interface. While traditionally most did that are active optical stuff, a lot don't.

a list of start method timestamps might be interesting to you

jclulow: great, thank you!

Smithx10: I also think because this is an active optical cable, it's less likely to have that unfortunately.

yuripv: They're probably not sorted, btw; you'll need something that can sort decimals; e.g., I think I have used "gsort -g" because it does ~floats

it's only 2 services, so i can compare the numbers myself; but thanks for the hint

Thanks Again rmustacc Much Appreciated !

[illumos-gate] 15056 Missing pcie cap leads to panic in pcie_fabric_feature_scan() -- Robert Mustacchi <rm⊙fo>