[illumos-gate] 15611 rtld: suggest explicit braces to avoid ambiguous 'else' -- Toomas Soome <tsoome⊙mc>

hmmm.. i do wonder if it might be useful to make pageout related i/os a higher priority than other writes

+1, to avoid pageout_deadman panic

it looks like (from looking at one such dump) that they just get lumped into the sync write queue

so they could compete with other sync writes

Hello all. Sorry if this has already been asked, but is there documentation that explains the directory structure of illumos-gate? I know that "uts" is the directory that contains the kernel sources. However, I don't recall what "uts" stands for & I have no idea what "cb" means for the directories named "ucbcmd" & "ucblib". A little help?

Try illumos.org/books/dev/layout.html

I believe it's 'unix timesharing system' and 'university of california berkeley

^^^

originally sun shipped with a BSD based os (back in the 80s I guess?) -- aka SunOS 4.x, then in the 90s, they moved (or I guess worked with AT&T?? something like that) on SystemV which became 'Solaris 2', and because it would be too simple otherwise, SunOS 4.x because solaris 1.x and the system-v based OS became solaris.2 (but the uname bits showed SunOS 5.x)

something like that

and so for compatability reasons, versions of commands that were different between sunos 4.x and solaris had the older sunos 4.x bits put into the ucb directories

so the ucb stuff is more or less the BSD versions (from that era -- don't confuse with modern BSDs) of commands

danmcd (or anyone else that looks at dumps)... I've written a little dcmd that takes an hrtime_t as an input and basically gives you the delta with mdb_gethrtime() (so 'now' on a live system, more or less the panic time on a dump -- mdb_gethrtime() is often a few ms later than panic_hrtime in a dump)

Oooh.

e.g.

> 0x2862546361::hrdelta -h

00:01:36.15787773

it seems like maybe useful as an mdb built-in ?

(i've been deep, deep insides the bowels of zfs's vdev queueing and the zio pipeline and i found myself doing the math constantly and though it might be useful as a command)

nothing like scratching a real-world itch.

You may want a way to specify what the comparison is against, but seems like it'd be useful depending on what you're doing.

[illumos-gate] 15590 smbd becomes unresponsive on memory allocation failures -- Gordon Ross <gwr⊙rc>

[illumos-gate] 15655 Build scripts should tolerate make -w -- Bill Sommerfeld <sommerfeld⊙ame>

[illumos-gate] 15605 elfedit: clean up warning gags -- Toomas Soome <tsoome⊙mc>

if you have a need for mdb stuff, I think it's always good to have it, it can be extended, etc.

(I think explicit comparison is good now though, and probably easy to add?)

ADS_Sr: 'uts' - UNIX timesharing system, what the business side of AT&T called what everyone else called 'sys/', 'ucb*' - UC Berkerely, but in this case really SunOS 4.x, 'psm' - platform specific module, once was the future, but then stopped being the future, kind of sits there half baked, 'stand' - standalones, like the booter etc, that run without an OS (on system's where that's more useful), not a thing on x86.

I have no idea how I came to spell Berkeley like that, sorry alanc.

[illumos-gate] 15651 bhyve mishandles periodic RTC timer -- Patrick Mooney <pmooney⊙pc>

You might as well do "sgs" for compleness.

It is in the dev guide.

It's the software generation suite, which was the AT&T name for the collection of utils at various points.

system, not suite, I think.

as a name, it made more sense when it had the compiler in it, etc, too.

but then Sun decided the compiler was a separate product.

and here we are

(I'm not sure whether they were the first to do that?)

sommerfeld_: I have an observation I didn't want to derail your 'make -w' for. The massive towers of egrep we form in nightly are -- in circumstances -- _really_ slow.

I _think_ because during the build obviously we fill nightly.log sequentially, and never read it, so the head of it is long gone from any cache before we get to reading it back, and then we effectively throttle it by the speed of each successive egrep, but I'm not sure, because I've not been able to catch it being awful except during a build.

where my worst result is in the minutes.

(and that might just be innately how long it will take because of circumstances)

I just enjoy sharing a mystery around.

at least nightly.log compresses well, if on a dataset with it so configured

yes, I noticed that recently while impatiently awaiting a build's completion.

I think the slow one is actually the "grep : $SRC/${INSTALLOG}.out"

sommerfeld: would fgrep/`grep -F` help there?

actually the several such. danmcd: not quite because there are regexps involved

You mentioned $SRC/${INSTALLOG}.out... thought that was a pattern.

I know the other egreps need the 'e'.

start looking at line 193

I see 5 passes over ${INSTALLOG}.out

And there's this: `egrep ":" $SRC/${INSTALLOG}.out |`

THAT can be fgrep, no?

fgrep is not necessarily faster than egrep

benchmark it

it comes with less semantic overhead

Fair enough. But the 5 passes... THAT could be one egrep expression if done right, no?

(ugg, prob. not)

or one awk?

The output goes to a bunch of different places, though. I'm wondering if we wouldn't be better off with "make install | tee -a $LOGFILE | awk

with the awk script splitting the bits & pieces off as needed

(build errors, build warnings, build noise, and the outright go-directly-to-mail-msg checks for fatal link errors and oversized bootblocks.

that would probably be more cache friendly

certainly more cache friendly.

Oh, I bet the 'grep "bootblock image is .* bytes too big" $SRC/${INSTALLOG}.out' is dead code.

(on a different subject, and nothing new, but again disappointed that the various allocation functions in zfs either don't allow the caller to specify allocation flags, nor change underlying allocation behavior based on what's requested

aside from data vs. metadata

That line in nightly is the only hit in the gate for "bytes too big" and "bootblock image".

nbjoerg: less semantic overhead but (at least historically) egrep was just plain more efficient even for fixed strings.

I wouldn't be surprised if some old versions are not using BM or so ;)

sommerfeld: a lot are dead code, <PSTAMP> etc.

If anyone is looking for places to optimize in the full build, consider the huge set of make pattern rules that come from $SRC/uts/.../Makefile.rules etc.

the only thing stopping me dealing with that is the merge problems.

Most of those only needed to be centralized like that so that we could run a global lint on all the kernel code. Now that we don't do that anymore, I'd love to see all the Makefile.files and Makefile.rules stuff split up and distributed around somewhere more local to the sources they related to.

my plan is to split things up so that common bits are actually _common_ for the ARM gate, and hopefully reduce a lot of that in the process.

but everyone downstream would get a potentially hellish merge, so I've been thinking about it in detail first

because it doesn't actually hurt build times as much as it seems (or didn't, when I measured it)

The current situation cause make to stat a _whole_ _lot_ of files that won't exist. (that ain't free)

Depend how well your FS does with ENOENT.

and honestly, the idea of writing up an IPD -- which I'd have to do -- terrifies me, because everyone and their dog has a peanut gallery opinion about this stuff

and I have no energy to talk about gmake, cmake, ninja, or whatever.

and yet "Why do all this and not ..." is a perfectly valid question

it's just one I'd rather throw myself into a lake than answer.

Incremental improvement is really the only sensible way forward, not "Let's switch to (insert favorite framework)'

anyway, it depends heavily on my cross-compilation IPD/work

which you'll notice I haven't sent out yet either :)

chicken.

:)

yeah, setting aside the possibilities of a build system overhaul or replacement, fixing the target rule stuff in uts has been a serious temptation

(because of all the unecessary stat()s)

there are also some VFS improvements we could make on that front as well, I think

well, I can shuffle some steps of things I want to do around maybe so that that's closer to the top

but I'm not sure how closer to the top is enough.

the critically important bit to me is the absolute lack of sharing in there stops

didn't mean to make it an ask for re-prioritization

yes

but the 2nd stage is probably not much worse after that.

most of the makefiles know what objects they need, and where they are

we don't need to put all the rules in one place so they're shared by literally everything

well, that's the fun bit, they're shared by "literally everything", except across platforms, where that sharing is useful.

and instead they're duplicated.

so I want to shared the shared bits, and then slice the files up as step 2

for various reasons, but especially that no matter how you do step 2, everyone downstream doing actual work has one hell of a merge

I can talk about it more another time, if you're interested (and I'd have to write it up anyway, so you'd definitely have time for an opinion)

looking at the egrep thing, it appears that " egrep 'warn:|warning:' " is a pathological regexp for egrep.

compared to /warn(ing)?:/ even?

trying desperately to remember finite automata

time egrep -i 'warn:|warning:' /z/ws/illumos-wformat/usr/src/install-nd-i386.out

[2] 801286 exit 1 egrep -i 'warn:|warning:' /z/ws/illumos-wformat/usr/src/install-nd-i386.out

egrep -i 'warn:|warning:' /z/ws/illumos-wformat/usr/src/install-nd-i386.out 147.50s user 0.15s system 99% cpu 2:27.66 total

time egrep -i 'warn(ing)?:' /z/ws/illumos-wformat/usr/src/install-nd-i386.out

[2] 801299 exit 1 egrep -i 'warn(ing)?:' /z/ws/illumos-wformat/usr/src/install-nd-i386.out

egrep -i 'warn(ing)?:' /z/ws/illumos-wformat/usr/src/install-nd-i386.out 1.40s user 0.07s system 99% cpu 1.476 total

wow

yeah.

I filed illumos.org/issues/15662

→ BUG 15662: nightly uses pathological regexp to find warnings. (New)

.oO

We should probably also look at improving egrep there at some point. That is pretty bad.

gnu grep does much faster (with "bad" regexp)

it's not using spencer regex

yep, reading cached log does not help our grep:(

I suspect that Apple fixed a CVE in DTrace and did not share the fix with anyone. Correct me if I am wrong.

I will send an email to the security email.

Thank you @ryao

[illumos-gate] 15648 Attempting to create multiple link-local IPv6 addresses leaves subinterfaces behind -- Andy Fiddaman <illumos⊙fn>

rmustacc: it looks like grep just uses the libc regcomp/regexec