04:48:52 [illumos-gate] 14548 snoop has strange bedfellows -- Joshua M. Clulow 08:57:03 hey gang, just curios if anyone knows if its possible to modify on the fly the zfs_arc_max? we realised that some of our problems are due to the system running out of memory (even though the vm's only account for 90Gb of 256GB, roughly 150Gb are being used by zfs arch cache. When a new provision is attempted a dladm process will hang (because i40e required 8gb per vnic of non-contigious memory; at least thats our understanding so far) ... thanks 08:57:04 for the help! 09:15:48 Hi all. I have a short question regarding ZFS: is it possible to force a release of the used used memory by ARC Cache without risking data corruption? On our systems, we see that ZFS File Data is using up to 57% of RAM. 09:24:32 GeorgePocky: you can try zinject -a 09:25:40 GeorgePocky - create a large file in /tmp - something like `mkfile 100g /tmp/fill && rm /tmp/fill` 09:26:04 However, generally it is better that memory is being used for ARC than sitting spare - the system will release it when there is memory pressure 09:26:43 in theory yes, it should be released, but from what i saw, RAM usage is 97.3% used and memory isn't filled up 09:28:11 and in combination with i40 10G network cards when if you need to create a vnic it will use up to 8GB of contiguous RAM, all the zones running (kvm/bhyve) will start to have intermittend access issues 09:33:00 Ouch, that's a big overhead for a VNIC. Which distribution are you using? 09:33:16 sensille: It looks like it will not release also the memory used by ZFS File Data on the system 09:33:29 andyf: we are using smartos 09:35:28 and yes, large file creation in /tmp does the trick 09:35:43 GeorgePocky : which PI are you using? 09:35:43 thanks for the idea! 09:36:27 I used to have to do that to boot a new VM when the system had been up a while, but there were changes that improved things there a lot and I haven't had to do it since. I don't have 10G network cards though. 09:36:49 physically contiguous RAM ... the i40e driver really needs to be improved 09:36:51 multiple PIs, even the latest ( 20220825T001415Z) 11:02:38 If you are starting memory hogs (like bhyve etc), then you should have max arc setting in place to reserve memory for those apps. 11:03:40 With default, the arc should still get freed, but this will take time... 12:31:59 In illumos, how can I admin down up an aggregation? 12:33:16 I have 2 interfaces in the down state apart of the same aggr... it was flapping in dmesg 12:39:44 tsoome - bhyve has the memory reservoir facility. I'd use that before starting to tune the ZFS ARC. 12:40:42 i have some VMs that have a reservation, so they always boot 12:40:56 ...straight away, without having to wait for ARC reclaim 12:44:46 ou 12:46:47 Smithx10 you want to check why interfaces are down, you may need to use fmadm to clear them 12:47:34 Would I see them down in fmadm faulty? 12:48:48 if they are faulted, it would mean they are disabled, yes 12:49:02 I only have a "Oct 17 2020 3c342e58-1556-c667-d207-fcfda8c8b835 SUNOS-8000-J0 Major" 12:49:25 hm, and dladm show-phys ? 12:50:04 https://gist.github.com/Smithx10/f2998bafd626ff7e2d14847dbf6949ef 12:50:37 I see. that fibre? 12:50:51 yea 12:51:06 The probem in my situation is, I have a network team at work "_" 12:51:28 that may be problem, indeed;) 12:52:06 both links down at the same time hints about human error 12:52:08 So I'm just trying to make sure the OS isn't sitting in some weird spot waiting for me to clear something or reset the aggregation or something 12:52:37 Yea... at my job.... they seem to replace cables at an extremely high rate (and never test them after pulling them) lol 12:52:38 aggregation should get online as soon as you have physical link 12:53:22 I can understand why folks end up on the Public Cloud :( 12:53:28 I'd get physical inspection first - are cables attached, what is the status of leds 12:53:51 Cool, I sent that e-mail / ticket in already 12:54:38 from that log, I'd say your ixgbe1 was offline before 12:55:21 and then something did happen about ixgbe3 - and as you see, when ixgbe3 is online, aggregation is also "up" 12:57:22 yeah 13:44:30 BTW your only fmadm faulty is for overlay, which fires whenever your headnode reboots and your CN does not. 13:44:39 Smithx10: ^^^ 13:45:31 And ixgbe1 is Just Down... (and yeah I remember your "network team" stories. WTF they aren't together with systems is _beyond_ me.) 13:47:25 Thanks danmcd 13:48:03 (When I flush Kebecloud's nodes, I reboot all the CNs first, then once they're happy, I reboot the HN, and once the HN's happy I `fmadm clear ...` on the CNs.) 14:20:20 andyf: did the reservoir facility ever get added to the omnios bhyve brand / zadm? Or did you just set that up manually for a few VMs 14:20:40 Yes, it's in the omnios brand - I think it made it into r151042 14:21:09 https://man.omnios.org/man7/bhyve#memreserve 14:21:55 ty 15:28:42 Folks from the DC are thinking the optics are getting too hot. Do we have a way to measure an interfaces temp? 15:29:49 Manually, but not automatically. 15:29:59 I never rigged that part of the transceiver parsing up, sorry. 15:30:07 But you can manually dump it and figure it out. 15:30:14 Along with getting the other errors. 15:30:24 What's the transceiver? That should be in topo. 15:31:59 [illumos-gate] 15036 portfs wears inadequate pollcache disguise -- Patrick Mooney 15:32:09 I'm not too familiar with fmtopo, what should I run? 15:35:14 https://gist.github.com/Smithx10/db4aad342671078c4b94e209a44c3292 15:40:47 Run /usr/lib/fm/fmd/fmtopo -V $FMRI , where one of those FMRIs in your gist should be used for $FMRI. 15:41:53 The -V will expand the goodies, and likely give you some answers. 15:41:56 Just leave out the $FMRI and gist that? 15:42:22 Or run /usr/lib/dl/dltraninfo -v 15:47:39 rmustacc here is the output of dltraninfo -v , https://gist.github.com/Smithx10/db4aad342671078c4b94e209a44c3292 15:49:57 https://us-east-storage.solutions.iqvia.com/bruce_dev/public/topo-out topo out 15:53:43 I don't see any temp info in -V, but I assume there is a switch for dumping all of the data? 16:07:32 It's like your transceivers aren't reporting temperature (at least by my reading... rmustacc may see something I'm missing). 16:10:18 danmcd: As I mentioned the parsing isn't there for diagnostic info there. 16:10:34 Based on the transceiver type we can look up where in the raw info it'll be and get it manually. 16:24:01 If you use -x, you'll get the raw hex and we ca go look at the spec. 16:30:21 rmustacc: https://us-east-storage.solutions.iqvia.com/bruce_dev/public/topo-x-output 16:39:27 I mean -x to dltraninfo, not topo. 16:39:42 That dumps the raw hex and you can go double check the specs on what offsets that'll be at. 16:51:08 Where do you usually grab the spec from? 16:51:34 In this case you want SFF-8472. 16:51:44 But get the dltraninfo -x and we'll take it apart. 16:52:38 https://gist.github.com/Smithx10/6543e8eae237c9f5b714fcf7d53ca456 17:08:28 Smithx10: So by my read of the data and SFF-8472 that doesn't support digital monitoring. 17:08:47 ahhh ok 17:08:49 So it's possible it's just lying, but because it doesn't claim it does implement the monitoring of SFF-8472 we don't try to get that i2c page. 17:09:54 The output of dltraninfo -x, what kind of table / or how would I know how to parse this? 17:10:20 Because this is an SFP+/SFP28 device you would go to SFF-8472 which is publicly available and look at the bytes. 17:10:29 That's what I did internally in the nvlist. 17:24:09 is there a way for svcs(1) to list ALL dependencies for a given service? -d seems to list only ones explicitly listed in the manifest, i would like it to show dependencies of dependencies too (i hope that made sense) 17:25:33 yuripv: I think, today, you would need to recurse yourself 17:26:08 But that would be a neat RFE, a "svcs -rd" that was able to create a ptree-like output 17:26:57 ok, or is there another way of checking if some service is supposed to start after some other? need to check if my service starts after identity:node 17:28:04 rmustacc: We have this optic in a linux machine and I read here: https://docs.nvidia.com/networking-ethernet-software/cumulus-linux-40/Monitoring-and-Troubleshooting/Troubleshooting-Network-Interfaces/Monitoring-Interfaces-and-Transceivers-Using-ethtool/#monitor-switch-port-sfpqsfp-hardware-information-using-ethtool that "Optical diagnostics support" would come back with yes, but I guess thats where the optic may be lying 17:28:16 yuripv: Do you want to know if it _should_ start after, or if it _did_ start after 17:28:43 ethtool on linux didn't return a "Optical diagnostics support " field at all with ethtool -m 17:28:59 both, i guess :) just seeing as if my service is starting too early, and wondering if i should just add explicit dependency here 17:30:32 Smithx10: So if it did have support then I wouldn't expect the field not to exist. 17:30:40 yea 17:30:44 :( shucks 17:30:46 So we are seeing the same thing from the transceiver. 17:30:47 Sorry. 17:31:04 I'm wondering how valuable this info is 17:31:17 Maybe in the future we should by optics that do DOM 17:31:20 yuripv: svcprop '*' | awk '$1 ~ /start_method_timestamp$/ { gsub("/:properties.*", "", $1); print $3, $1 }' 17:31:24 There are definitely ones that don't implement the digital monitoring interface. While traditionally most did that are active optical stuff, a lot don't. 17:31:27 a list of start method timestamps might be interesting to you 17:32:36 jclulow: great, thank you! 17:32:48 Smithx10: I also think because this is an active optical cable, it's less likely to have that unfortunately. 17:40:33 yuripv: They're probably not sorted, btw; you'll need something that can sort decimals; e.g., I think I have used "gsort -g" because it does ~floats 17:41:06 it's only 2 services, so i can compare the numbers myself; but thanks for the hint 18:06:34 Thanks Again rmustacc Much Appreciated ! 21:04:43 [illumos-gate] 15056 Missing pcie cap leads to panic in pcie_fabric_feature_scan() -- Robert Mustacchi