02:00:02 perhaps you can automatically generate 'bloody releases' (for OmniOS CE or others with teams) daily, weekly, or monthly? At times I've seen them older than the current stable (as now) or even LTS releases... 02:00:36 or even let people get it from a version control system (VCS) and make their own .IMG/.ISO? 02:05:11 Is it preferable to ask questions about dtrace on illumos in this channel, or in the #dtrace channel? 02:42:53 I'm trying to help someone port ghc 9.12.1 to OI, and because of limitations in a build tool I need to capture the stderr of gcc (I can't get it from the build tool, it's discarded if gcc exits nonzero and fixing that in the build tool will be fairly involved) 02:44:01 my understanding from the dtrace handbook was that /execname == "gcc" && args[0] == 2/ would be the correct condition, but dtrace is telling me I can't reference args in a predicate because it's variable. the manual tells me there's no if-then-else. is there some other way to limit output to only fd 2? 02:50:17 "dtrace: failed to compille script gcc-stderr.d: line 2: args[ ] may not be referenced because probe description syscall::write:return matches an unstable set of probes" 02:50:40 (uh, typo, I had to retype that out of my VM until I figure out how to make cut and paste work) 03:31:51 you could probably instead use the libc wrapper around the syscall.. just stash the pointer to the buffer on entry, then use it to dump the contents (using the return value for the length) 11:10:09 Is this chat active? 11:15:27 sometimes 11:16:44 a lot of channels are somewhat less active right now because of the holiday 13:44:59 my system is showing near 100% memory usage but htop/ps/etc don't show any processes using any large amounts of memory. how could i track this down? 13:46:48 probably a better question for a linux-related channel. it can mean e.g. open but unlinked files, but usually those get flushed to swap as needed 13:47:26 that said, well, the goal of memory management is to put memory to productive use, so near 100% memory usage without using swap is not necessarily a bad thing 13:48:10 why a linux related channel? 13:48:15 (in my case, when memory usage gets high it's either some leaky MATE program or vmware, currently the latter because I've been using it a lot) 13:48:30 and i wouldn't mind high memory usage... if it was attributed to something 13:48:37 oh wait, I'm in the wrong channel, sorry 13:48:57 #ilumos inserted itself before #xmonad in my IRC client 🙂 13:49:25 ah but i use i3, sorry :P 13:50:35 meh. it's not like we're at war 13:51:42 about the only wm xmonad competes with is dwm, and that not very much 15:10:35 if you have lots of unused memory then you have wasted your money:P 15:12:37 in htop you can enable the "ZFS ARC" and/or "ZFS CARC" meters. which will show you how much memory zfs is using for its ARC. mind you, this memory will be released if something else needs it 15:52:43 Hello, is anyone active? I need help with building illumos 16:00:11 have you read https://illumos.org/docs/developers/build/ ? 16:02:30 Yeah 16:03:03 but when i runned illumos.sh it immidiately turn off 16:03:03 so what problem are you facing? 16:03:51 1. I downloaded source code, 2. i added# 16:03:52 # Set a package version number which is greater than the current OpenIndiana 16:03:52 # build number. 16:03:53 # 16:03:53 export PKGVERS_BRANCH=9999.99.0.0 16:03:54 # 16:03:54 # Set Perl related variables: 16:03:55 # 16:03:55 PERL='/usr/perl5/bin/perl' 16:03:56 export PERL_VERSION="$($PERL -e 'print $^V =~ /^v(5\.[^\.]*).*$/')" 16:03:56 export PERL_PKGVERS="$($PERL -e 'print "-", $^V =~ /^v(5)\.([^\.]*).*$/')" 16:03:57 export PERL_ARCH="$($PERL -MConfig -e 'print $Config{archname}')" 16:03:57 export PERL_ARCH64="$PERL_ARCH" 16:03:58 export BUILDPERL32="$($PERL -MConfig -e 'print $Config{ptrsize} == 4 ? "" : "#"')" 16:03:58 export BUILDPERL64="$($PERL -MConfig -e 'print $Config{ptrsize} == 8 ? "" : "#"')" 16:03:59 ah, you run it with ./usr/src/tools/scsripts/nightly illumos.sh 16:06:50 nightly.sh? Should i edit something in it? 16:17:54 no 16:18:19 which distro are you using? 16:19:20 OpenIndiana 16:20:01 i can't find nightly illumos.sh script somehow 16:21:50 i ain't gonna lie i don't really understand what to do after getting source 16:23:40 :') 16:23:46 you might have missed the "cp usr/src/tools/env/illumos.sh ." step from https://illumos.org/docs/developers/build/#configuring-the-build 16:24:07 then you can edit the copy if needed 16:27:39 @Guest8: The easiest way to build illumos-gate on OpenIndiana is to use its build system: 16:27:39 git clone https://github.com/OpenIndiana/oi-userland ; cd oi-userland/components; gmake setup; cd openindiana/illumos-gate; gmake publish 16:28:28 wacki and that's it? 16:35:08 I'm trying it 16:36:32 Yes 16:36:55 your user needs to be able to install missing packages (if there are any) 16:38:50 You can read about it at https://docs.openindiana.org/dev/userland/ 16:43:38 Thanks 16:44:34 hm. HTTP Error 504: Gateway Time-out from git pbchk 16:46:08 ah. https://www.illumos.org/issues is not responding. 17:43:27 I have a bit of a strange situation, wondering if any dev here wants to take a look at it. ... 17:44:05 Summary: tar of a .gz file exits without error but doesn't complete the process. Extracting the file returns error. Uncompressing the file results in a clean tar, as does recompressing the file. 17:44:30 Since I have a workaround (uncompress, recompress) I'm not overly fussed, just wanted to know if anyone wanted to take a look before I do it. 17:44:49 We are leaving it compressed because this is for a filesystem backup and we want things to be in the state they're in in the tarfile itself. 17:45:07 OS in question is OmniOS LTS (latest patch set) 17:47:05 tsoome: also seeing slowness in gerrit and permission errors trying to review one of your proposed changes. 17:49:10 nomad: jbk was looking at tar decompression recently (see https://code.illumos.org/c/illumos-gate/+/3672 when gerrit is happy again) 17:49:43 nomad: can you be specific about what you mean by "doesn't complete the process"? only extracts some of the files at the beginning of the compressed archive? 17:49:51 code review '3672' not found... 17:56:10 sommerfeld, when creating the tarfile it makes a much smaller file than expected. When extracting it reports "tar: directory checksum error" and exits with 2. 17:56:23 I have a full script that can show what I'm talking about, let me post it. 17:57:19 https://pastebin.com/2LCWJdaU 17:57:53 that script copies the .gz file into a scratch directory, tries to tar it and extract it, uncompresses, tars & extracts, recompresses, tars & extracts. 17:58:05 the past includes the script and the result. 17:59:12 in case I wasn't clear (I frequently fail at that) the initial tar fails but exits without error. 18:01:23 In the initial case where we discovered this problem it was obvious tar exited while dealing with this file as the resulting tarball is *way* too small to contain everything. 18:01:28 That's not in the testcase.' 18:36:44 nomad: so this is not relevant to the error you're seeing, but bzip2 compressing a gzip'ed file strikes me as an odd choice. I'd expect better compression if you gave bzip2 the .csv file. 18:38:37 As I said, this is a test case brought up by trying to tar up a filesystem. We're archiving the filesystem elsewhere. 18:38:44 Changing the files in the filesystem isn't really an option. 18:39:32 I saw the tarball was very small and narrowed the problem down to this one file not being properly handled. 18:40:40 In theory I could fix this problem by merely having the owner extract and recompress the file but I thought it might be of interest to a dev to know there's a bug... somewhere. 18:41:26 I just copied the problem file to another fileserver running the same install of OmniOS and it tar'd up fine. 18:41:46 Now I'm trying it on a filesystem on that other fileserver with nbmand set to on to see if that changes anything. 18:42:23 are you accessing the file over smb? 18:42:35 all of these tests are local to the fileserver. 18:42:38 (nbmand should only matter for smb clients) 18:43:21 hmm. Ok. Interesting that scping the file was enough to 'fix' the problem. 18:43:37 The only difference I could find otherwase was nbmand=on in the source filesystem. 18:45:30 it might be interesting to dump the contents of that uncompressed block and the expected/calculated checksums 18:45:45 unfortunately, i think that'd require a modified tar binary to do so (at least easily) 18:46:39 I was hoping someone expert would come up with a dtrace or three :) 18:48:58 hrm... tar isn't built with ctf :( 18:53:16 another wild thing I'm testing is file ownership. The file copied to the other fileserver is owned by me, not by the original owner. 18:53:36 so now I'm re-running my test script on the original fileserver but with me as the owner of the example file. 18:53:51 I seriously doubt anything will change but . 18:54:59 FTR: testing on the second fileserver with nbmand made no difference. The tarball created and extracted correctly. 19:00:48 as expected, changing file ownership on the original fileserver made no difference. 19:12:20 I'm not sure why we are not building everything with ctf... 19:15:03 One detail I should have mentioned is that this file was created via a SMB/CIFS mount. Everything I'm trying is local to the host. 19:15:37 but because of ownerships and permissions on SMB/CIFS I'm doing all my tests as root unless otherwise specified. 19:15:53 The copy to the other fileserver is scp using Unix permissions/owernships. 19:16:00 and that's all done as a regular user. 19:17:12 it might be worth doing an ls -lV to compare 19:18:50 if created via SMB, there's likely ACLs attached to the file 19:18:57 well, the reason I bring that up is I just tested on the original fileserver by sudo cp the file to another filesystem (as all of my tests have done) then chown'ing the file to me. 19:19:09 and tar worked when I did everything else as me instead of as root. 19:20:11 cp -p that is. 19:20:36 the permissions on that file in situ match the permissions of other files in the same location. 19:20:56 (All of which tar just fine.) 19:40:18 you're using the E ("extended headers") and "p" (save acls) options. I wonder if they play nice together. 19:40:52 while we're looking at weird tar options, there's also an "i" ("ignore directory checksum errors") which might sweep this one under the rug. 19:41:53 chances are the "recompression" part is a red herring; the real difference is an ACL change which affects what it does for "p" 19:42:57 also.. the way acls are saved via smb can vary a bit.. i was wondering if maybe there's some bad interaction there 19:44:21 IIRC, the 'subject' of an ACL can be a windows sid, or a uid/gid, but it can depend on your idmap setup 19:46:03 or i guess maybe to say it slightly differently, the same acl can potentially be expressed into different ways depending on idmap 19:46:09 err in two 19:50:42 that's undelightfully confusing. 19:51:11 At this point I've recopied the file, verified root tar/extract fails, then chown'd the file to me and am trying again as root. 19:51:31 Next I'll try changing permissions on the file. 19:52:42 basically if you have a windows user foo@domain that is also mapped to a unix user 'foo', and there's an acl targeting that user, it might be saved as either usersid:foo@domain:.... or user:foo:.... 19:53:15 Here's the ls -lV on the file in question (in situ): 19:53:18 ----------+ 1 sdaniel sysadmin 4958477085 Jul 24 2019 CHS_methylation_SWAN_2019July23.csv.gz 19:53:18 group:11706:rwxpdDaARWcCos:------I:allow 19:53:19 group:13116:rwxpd-aARWc--s:------I:allow 19:53:19 group:13115:r-x---a-R-c--s:------I:allow 19:53:19 which are stored as distinct things in zfs -- they'll get evaluated the same, but aren't byte equivalent on disk 19:53:44 (similarly iwth groups) 19:54:07 I could see a bug where an ACL could get implicitly 'converted' from one form to the other 19:54:37 which doesn't alter the effective permissions, but would change the actual encoding of it on disk 19:55:06 but why would it tar *part* of the file, then fail silently? 19:55:20 I could see it failing to open the file, or such. 19:58:38 or, why would it extract part of the file then error out with 'tar: directory checksum error' 19:58:42 also, it might be worth (for testing) trying without E and p flags (or just one and not the other) -- just to see if maybe it can be narrowed down to behavior with a specific flag 19:59:08 and mind you, the initial tar -c doesn't add any files after the questionable one. 19:59:20 I'll add that to my queue. 20:00:36 I think (but not 100% sure) the 'directory checksum' might be better read as 'header/entry' checkum 20:00:58 makes sense 20:01:28 it failed with just the chown, now testing with 'chmod u+r'. 20:01:52 then I'll try it without the two flags and see if anything changes. I'll narrow it down after. 20:02:22 for every file, it stores (mostly as more or less text) the mode, uid, gid, size, mtime, etc 20:02:39 and does a checksum of all of those 20:04:07 It wouldn't re-evaluate those in the middle of extracting the file, though, would it? It's dying 633M into the file. 20:06:46 Looking back at my original test, I am reminded of this: 20:06:56 adding the file: a CHS_methylation_SWAN_2019July23.csv.gz 4842263K 20:07:07 extracting the file: x CHS_methylation_SWAN_2019July23.csv.gz, 663509789 bytes, 1295918 tape blocks 20:07:32 the extracted file is the same size extracting says it will be... but maybe it prints that after extract finishes instead of before. I don't remember now. 20:12:31 nope, it prints it before extracting. 20:12:45 and the changed permission was sufficient if the extraction message is to be believed. 20:12:50 x CHS_methylation_SWAN_2019July23.csv.gz, 4958477085 bytes, 9684526 tape blocks 20:13:06 that's very different from "x CHS_methylation_SWAN_2019July23.csv.gz, 663509789 bytes, 1295918 tape blocks" 20:17:53 and sure enough, the extraction succeeds. 20:18:21 ok, time to try w/o -pE but with the original permissions. 20:21:42 I feel like -- possibly -- you might want to try to debug this after the new year too 20:21:48 when more folks are around/at work/etc. 20:21:53 richlowe, most likely, yes. 20:22:12 so at least maybe don't do anything destructive yet :) 20:22:16 but maybe, just maybe, I can find a way to generate a file that will fail so more people can test against. 20:22:19 I can't share this file. 20:22:23 that would be good, yes 20:22:37 personally, I've never methylated a swan, so I wouldn't know what it meant anyway 20:22:38 but I understand 20:22:46 oh no, I'm doing everything with 'cp -p' of the original file into a test filesystem. 20:22:54 I'm leaving prod as untouched as I possibly can. 20:23:31 It's genomic data of somesort. 20:26:34 I suppose I could try creating a huge file in windows and see if it has the same problem. 20:27:44 though that would mean creating a new filesystem with proper exports and then logging into a windows box. 20:28:30 * nomad is not going to go creating new filesystems today, nope nope nope 20:30:04 x CHS_methylation_SWAN_2019July23.csv.gz, 4958477085 bytes, 9684526 tape blocks 20:30:14 that's without -pE in the create or extract flags. 20:30:34 I'll test the extract with -pE as soon as this finishes. 20:40:17 extracting with pE flags is successful. Now to create the tarball with just the E flag. 20:43:17 there are also some uninitialized variable warnings that are currently suppressed 20:43:42 i haven't looked at them closely, so might not be actual issues or relevant, but something else that could be contributing 20:44:05 I presume I'm not seeing them because they're suppressed. :) 20:44:12 at least, I hope that's why I haven't noticed them. 20:44:45 they're build warnings that are currently suppressed 20:45:00 whew. I was afraid I'd ignored them. 20:51:08 looks like 'p' is the flag that triggers the problem. Makes sense given the interplay with permissions. 20:51:30 I'm waiting for the tar xjvlpEf to finish so I can verify but it says it is extracting a file of the correct size. 20:55:53 yep, extraction is good (though, of course, permissions are wrong). 21:37:14 well damn. Now I have no faith at all in any of the tarballs created in the past 6 months. 21:37:26 I was able to replicate this problem fairly trivially. 21:38:12 source: 21:38:15 ----------+ 1 lvd sysadmin 5369578871 Dec 31 13:24 hugefile.gz.hold 21:38:15 group:11706:rwxpdDaARWcCos:-------:allow 21:38:18 extracted result" 21:38:24 ----------+ 1 lvd sysadmin 1074611575 Dec 31 13:24 hugefile.gz 21:38:24 group:11706:rwxpdDaARWcCos:-------:allow 21:38:33 one of these things is not like the other. 21:45:32 Here's my proposed problem ticket submission. Did I miss anything important? https://pastebin.com/KArbQSxU 22:03:56 seems fine to me 22:04:41 Should I submit it now or wait for Thursday? I presume now is fine. 22:04:55 yeah, no reason to wait 22:05:37 jbk, nomad: well, the illumos bug tracker appears to be off the air at the moment. 22:05:49 I was just about to ask if it were still down 22:05:57 oh hrm 22:05:58 since I'm timing out I guess I know the answer. :) 22:05:59 that sucks :( 22:06:13 Good thing I have this written up as a text file I can just cut & paste :) 22:07:21 apparently it decided it needed the holiday off as well :P 22:09:30 504 gateway timeout. How exciting. 22:09:32 methylation is described here: https://en.wikipedia.org/wiki/DNA_methylation - it looks to be an "off" switch for certain genes or some such. 22:09:34 (not) 22:52:07 Happy new year everyone 22:52:23 you too 22:56:07 Happy New Year!