perhaps you can automatically generate 'bloody releases' (for OmniOS CE or others with teams) daily, weekly, or monthly? At times I've seen them older than the current stable (as now) or even LTS releases...

or even let people get it from a version control system (VCS) and make their own .IMG/.ISO?

Is it preferable to ask questions about dtrace on illumos in this channel, or in the #dtrace channel?

I'm trying to help someone port ghc 9.12.1 to OI, and because of limitations in a build tool I need to capture the stderr of gcc (I can't get it from the build tool, it's discarded if gcc exits nonzero and fixing that in the build tool will be fairly involved)

my understanding from the dtrace handbook was that /execname == "gcc" && args[0] == 2/ would be the correct condition, but dtrace is telling me I can't reference args in a predicate because it's variable. the manual tells me there's no if-then-else. is there some other way to limit output to only fd 2?

"dtrace: failed to compille script gcc-stderr.d: line 2: args[ ] may not be referenced because probe description syscall::write:return matches an unstable set of probes"

(uh, typo, I had to retype that out of my VM until I figure out how to make cut and paste work)

you could probably instead use the libc wrapper around the syscall.. just stash the pointer to the buffer on entry, then use it to dump the contents (using the return value for the length)

Is this chat active?

sometimes

a lot of channels are somewhat less active right now because of the holiday

my system is showing near 100% memory usage but htop/ps/etc don't show any processes using any large amounts of memory. how could i track this down?

probably a better question for a linux-related channel. it can mean e.g. open but unlinked files, but usually those get flushed to swap as needed

that said, well, the goal of memory management is to put memory to productive use, so near 100% memory usage without using swap is not necessarily a bad thing

why a linux related channel?

(in my case, when memory usage gets high it's either some leaky MATE program or vmware, currently the latter because I've been using it a lot)

and i wouldn't mind high memory usage... if it was attributed to something

oh wait, I'm in the wrong channel, sorry

#ilumos inserted itself before #xmonad in my IRC client 🙂

ah but i use i3, sorry :P

meh. it's not like we're at war

about the only wm xmonad competes with is dwm, and that not very much

if you have lots of unused memory then you have wasted your money:P

in htop you can enable the "ZFS ARC" and/or "ZFS CARC" meters. which will show you how much memory zfs is using for its ARC. mind you, this memory will be released if something else needs it

Hello, is anyone active? I need help with building illumos

have you read illumos.org/docs/developers/build ?

Yeah

but when i runned illumos.sh it immidiately turn off

so what problem are you facing?

1. I downloaded source code, 2. i added#

# Set a package version number which is greater than the current OpenIndiana

# build number.

#

export PKGVERS_BRANCH=9999.99.0.0

#

# Set Perl related variables:

#

PERL='/usr/perl5/bin/perl'

export PERL_VERSION="$($PERL -e 'print $^V =~ /^v(5\.[^\.]*).*$/')"

export PERL_PKGVERS="$($PERL -e 'print "-", $^V =~ /^v(5)\.([^\.]*).*$/')"

export PERL_ARCH="$($PERL -MConfig -e 'print $Config{archname}')"

export PERL_ARCH64="$PERL_ARCH"

export BUILDPERL32="$($PERL -MConfig -e 'print $Config{ptrsize} == 4 ? "" : "#"')"

export BUILDPERL64="$($PERL -MConfig -e 'print $Config{ptrsize} == 8 ? "" : "#"')"

ah, you run it with ./usr/src/tools/scsripts/nightly illumos.sh

nightly.sh? Should i edit something in it?

no

which distro are you using?

OpenIndiana

i can't find nightly illumos.sh script somehow

i ain't gonna lie i don't really understand what to do after getting source

:')

you might have missed the "cp usr/src/tools/env/illumos.sh ." step from illumos.org/docs/developers/build/#configuring-the-build

then you can edit the copy if needed

@Guest8: The easiest way to build illumos-gate on OpenIndiana is to use its build system:

git clone github.com/OpenIndiana/oi-userland ; cd oi-userland/components; gmake setup; cd openindiana/illumos-gate; gmake publish

wacki and that's it?

I'm trying it

Yes

your user needs to be able to install missing packages (if there are any)

You can read about it at docs.openindiana.org/dev/userland

Thanks

hm. HTTP Error 504: Gateway Time-out from git pbchk

ah. illumos.org/issues is not responding.

I have a bit of a strange situation, wondering if any dev here wants to take a look at it. ...

Summary: tar of a .gz file exits without error but doesn't complete the process. Extracting the file returns error. Uncompressing the file results in a clean tar, as does recompressing the file.

Since I have a workaround (uncompress, recompress) I'm not overly fussed, just wanted to know if anyone wanted to take a look before I do it.

We are leaving it compressed because this is for a filesystem backup and we want things to be in the state they're in in the tarfile itself.

OS in question is OmniOS LTS (latest patch set)

tsoome: also seeing slowness in gerrit and permission errors trying to review one of your proposed changes.

nomad: jbk was looking at tar decompression recently (see code.illumos.org/c/illumos-gate/+/3672 when gerrit is happy again)

nomad: can you be specific about what you mean by "doesn't complete the process"? only extracts some of the files at the beginning of the compressed archive?

code review '3672' not found...

sommerfeld, when creating the tarfile it makes a much smaller file than expected. When extracting it reports "tar: directory checksum error" and exits with 2.

I have a full script that can show what I'm talking about, let me post it.

that script copies the .gz file into a scratch directory, tries to tar it and extract it, uncompresses, tars & extracts, recompresses, tars & extracts.

the past includes the script and the result.

in case I wasn't clear (I frequently fail at that) the initial tar fails but exits without error.

In the initial case where we discovered this problem it was obvious tar exited while dealing with this file as the resulting tarball is *way* too small to contain everything.

That's not in the testcase.'

nomad: so this is not relevant to the error you're seeing, but bzip2 compressing a gzip'ed file strikes me as an odd choice. I'd expect better compression if you gave bzip2 the .csv file.

As I said, this is a test case brought up by trying to tar up a filesystem. We're archiving the filesystem elsewhere.

Changing the files in the filesystem isn't really an option.

I saw the tarball was very small and narrowed the problem down to this one file not being properly handled.

In theory I could fix this problem by merely having the owner extract and recompress the file but I thought it might be of interest to a dev to know there's a bug... somewhere.

I just copied the problem file to another fileserver running the same install of OmniOS and it tar'd up fine.

Now I'm trying it on a filesystem on that other fileserver with nbmand set to on to see if that changes anything.

are you accessing the file over smb?

all of these tests are local to the fileserver.

(nbmand should only matter for smb clients)

hmm. Ok. Interesting that scping the file was enough to 'fix' the problem.

The only difference I could find otherwase was nbmand=on in the source filesystem.

it might be interesting to dump the contents of that uncompressed block and the expected/calculated checksums

unfortunately, i think that'd require a modified tar binary to do so (at least easily)

I was hoping someone expert would come up with a dtrace or three :)

hrm... tar isn't built with ctf :(

another wild thing I'm testing is file ownership. The file copied to the other fileserver is owned by me, not by the original owner.

so now I'm re-running my test script on the original fileserver but with me as the owner of the example file.

I seriously doubt anything will change but <shrug>.

FTR: testing on the second fileserver with nbmand made no difference. The tarball created and extracted correctly.

as expected, changing file ownership on the original fileserver made no difference.

I'm not sure why we are not building everything with ctf...

One detail I should have mentioned is that this file was created via a SMB/CIFS mount. Everything I'm trying is local to the host.

but because of ownerships and permissions on SMB/CIFS I'm doing all my tests as root unless otherwise specified.

The copy to the other fileserver is scp using Unix permissions/owernships.

and that's all done as a regular user.

it might be worth doing an ls -lV to compare

if created via SMB, there's likely ACLs attached to the file

well, the reason I bring that up is I just tested on the original fileserver by sudo cp the file to another filesystem (as all of my tests have done) then chown'ing the file to me.

and tar worked when I did everything else as me instead of as root.

cp -p that is.

the permissions on that file in situ match the permissions of other files in the same location.

(All of which tar just fine.)

you're using the E ("extended headers") and "p" (save acls) options. I wonder if they play nice together.

while we're looking at weird tar options, there's also an "i" ("ignore directory checksum errors") which might sweep this one under the rug.

chances are the "recompression" part is a red herring; the real difference is an ACL change which affects what it does for "p"

also.. the way acls are saved via smb can vary a bit.. i was wondering if maybe there's some bad interaction there

IIRC, the 'subject' of an ACL can be a windows sid, or a uid/gid, but it can depend on your idmap setup

or i guess maybe to say it slightly differently, the same acl can potentially be expressed into different ways depending on idmap

err in two

that's undelightfully confusing.

At this point I've recopied the file, verified root tar/extract fails, then chown'd the file to me and am trying again as root.

Next I'll try changing permissions on the file.

basically if you have a windows user foo@domain that is also mapped to a unix user 'foo', and there's an acl targeting that user, it might be saved as either usersid:foo@domain:.... or user:foo:....

Here's the ls -lV on the file in question (in situ):

----------+ 1 sdaniel sysadmin 4958477085 Jul 24 2019 CHS_methylation_SWAN_2019July23.csv.gz

group:11706:rwxpdDaARWcCos:------I:allow

group:13116:rwxpd-aARWc--s:------I:allow

group:13115:r-x---a-R-c--s:------I:allow

which are stored as distinct things in zfs -- they'll get evaluated the same, but aren't byte equivalent on disk

(similarly iwth groups)

I could see a bug where an ACL could get implicitly 'converted' from one form to the other

which doesn't alter the effective permissions, but would change the actual encoding of it on disk

but why would it tar *part* of the file, then fail silently?

I could see it failing to open the file, or such.

or, why would it extract part of the file then error out with 'tar: directory checksum error'

also, it might be worth (for testing) trying without E and p flags (or just one and not the other) -- just to see if maybe it can be narrowed down to behavior with a specific flag

and mind you, the initial tar -c doesn't add any files after the questionable one.

I'll add that to my queue.

I think (but not 100% sure) the 'directory checksum' might be better read as 'header/entry' checkum

makes sense

it failed with just the chown, now testing with 'chmod u+r'.

then I'll try it without the two flags and see if anything changes. I'll narrow it down after.

for every file, it stores (mostly as more or less text) the mode, uid, gid, size, mtime, etc

and does a checksum of all of those

It wouldn't re-evaluate those in the middle of extracting the file, though, would it? It's dying 633M into the file.

Looking back at my original test, I am reminded of this:

adding the file: a CHS_methylation_SWAN_2019July23.csv.gz 4842263K

extracting the file: x CHS_methylation_SWAN_2019July23.csv.gz, 663509789 bytes, 1295918 tape blocks

the extracted file is the same size extracting says it will be... but maybe it prints that after extract finishes instead of before. I don't remember now.

nope, it prints it before extracting.

and the changed permission was sufficient if the extraction message is to be believed.

x CHS_methylation_SWAN_2019July23.csv.gz, 4958477085 bytes, 9684526 tape blocks

that's very different from "x CHS_methylation_SWAN_2019July23.csv.gz, 663509789 bytes, 1295918 tape blocks"

and sure enough, the extraction succeeds.

ok, time to try w/o -pE but with the original permissions.

I feel like -- possibly -- you might want to try to debug this after the new year too

when more folks are around/at work/etc.

richlowe, most likely, yes.

so at least maybe don't do anything destructive yet :)

but maybe, just maybe, I can find a way to generate a file that will fail so more people can test against.

I can't share this file.

that would be good, yes

personally, I've never methylated a swan, so I wouldn't know what it meant anyway

but I understand

oh no, I'm doing everything with 'cp -p' of the original file into a test filesystem.

I'm leaving prod as untouched as I possibly can.

It's genomic data of somesort.

I suppose I could try creating a huge file in windows and see if it has the same problem.

though that would mean creating a new filesystem with proper exports and then logging into a windows box. <sigh>

x CHS_methylation_SWAN_2019July23.csv.gz, 4958477085 bytes, 9684526 tape blocks

that's without -pE in the create or extract flags.

I'll test the extract with -pE as soon as this finishes.

extracting with pE flags is successful. Now to create the tarball with just the E flag.

there are also some uninitialized variable warnings that are currently suppressed

i haven't looked at them closely, so might not be actual issues or relevant, but something else that could be contributing

I presume I'm not seeing them because they're suppressed. :)

at least, I hope that's why I haven't noticed them.

they're build warnings that are currently suppressed

whew. I was afraid I'd ignored them.

looks like 'p' is the flag that triggers the problem. Makes sense given the interplay with permissions.

I'm waiting for the tar xjvlpEf to finish so I can verify but it says it is extracting a file of the correct size.

yep, extraction is good (though, of course, permissions are wrong).

well damn. Now I have no faith at all in any of the tarballs created in the past 6 months.

I was able to replicate this problem fairly trivially.

source:

----------+ 1 lvd sysadmin 5369578871 Dec 31 13:24 hugefile.gz.hold

group:11706:rwxpdDaARWcCos:-------:allow

extracted result"

----------+ 1 lvd sysadmin 1074611575 Dec 31 13:24 hugefile.gz

group:11706:rwxpdDaARWcCos:-------:allow

one of these things is not like the other.

Here's my proposed problem ticket submission. Did I miss anything important? pastebin.com/KArbQSxU

seems fine to me

Should I submit it now or wait for Thursday? I presume now is fine.

yeah, no reason to wait

jbk, nomad: well, the illumos bug tracker appears to be off the air at the moment.

I was just about to ask if it were still down

oh hrm

since I'm timing out I guess I know the answer. :)

that sucks :(

Good thing I have this written up as a text file I can just cut & paste :)

apparently it decided it needed the holiday off as well :P

504 gateway timeout. How exciting.

methylation is described here: en.wikipedia.org/wiki/DNA_methylation - it looks to be an "off" switch for certain genes or some such.

(not)

Happy new year everyone

you too

Happy New Year!