00:16:25 <xaero> is there a list of the last remaining closed source bits (if any) required to build illumos?
00:19:03 <andyf> https://illumos.topicbox.com/groups/developer/T9694d83e5bcf84ed-M076701df50a68fff011b8962/remaining-closed-bits is probably the most recent collated list
00:23:10 <xaero> andyf: thanks!
02:29:39 <sommerfeld> Hmm, I think I have some more work to do on the python backtraces - line numbers are sometimes quite wrong..
05:14:53 <jclulow> jbk: That's why src.illumos.org stopped indexing the other things :P
06:10:14 <jbk> i can totally understand :)
06:11:45 <jbk> of the 10-30s / file it was taking on my system, I am curious to know what the breakdown of what it's doing
06:53:09 <gitomat> [illumos-gate] 16268 libadutils: open_conn could print additional ldap error information -- Matt Barden <mbarden⊙rc>
19:32:32 <sommerfeld> so it's probably the wrong way to do it but I have a kludge to work around the .cold thing.   https://code.illumos.org/c/illumos-gate/+/3287
19:32:34 <fenix> → CODE REVIEW 3287: XXXXX CTF lookup kludge for .cold symbols (NEW)
19:44:34 <rmustacc> So is there no dwarf being generated for the .cold functions?
19:47:24 <rmustacc> Put differently who generates the .cold symbols?
19:48:12 <rmustacc> If this is going to end up in libproc, we're going to want to constraint this assumption as the python logic is definitely not a universal assumption.
19:48:21 <rmustacc> (Unless there is some broader generalization)
19:56:40 <sommerfeld> GCC is generating the .cold symbols when run with some options that enable profile-guided optimization.
19:57:02 <sommerfeld> I'm not sure how to check the dwarf for that
19:57:57 <sommerfeld> yes
19:57:59 <sommerfeld> oops
19:58:26 <rmustacc> OK, so I'd start with dwarfdump I guess.
19:58:41 <rmustacc> And you're still doing the ctfconvert/ctfmerge style pass right?
19:59:17 <rmustacc> So presuming that you aren't stripping this, I'd use the dwarfdump either from the libdwarf or llvm utils and see if there's an entry related to that with type information.
20:00:38 <sommerfeld> I switched to the single ctfconvert at the end on every elf file in DESTDIR 
20:02:23 <rmustacc> Does that occur before or after pgo?
20:02:29 <sommerfeld> After
20:02:59 <rmustacc> OK. I guess then the thing we'll need to figure out is how gcc is representing the relationship between the functions in dwarf.
20:03:47 <sommerfeld> PGO process: build with one set of options to generate a profile; run a set of tests to exercise the first build and dump a profile; then rebuild with a different set of options to generate a new binary influenced by the profile. 
20:06:58 <rmustacc> Maybe there's a DW_AT_abstract_origin.
20:07:15 <rmustacc> Though not sure if that'll be in dwarf 2.
20:08:30 <sommerfeld> I switched to dwarf 4 at your suggestion
20:10:34 <rmustacc> So I think the question is what dwarf if anything is gcc generating for .cold.
20:11:55 <sommerfeld> what I see for the function of interest is two entries in DW_AT_ranges, one which has a low bound matching _PyEval_EvalFrameDefault, and the other matching _PyEval_EvalFrameDefault.cold
20:13:47 <rmustacc> Yeah, though we don't really look at the ranges as usually that's for a different part.
20:13:57 <rmustacc> I'd exepct there's a DW_TAG_subprogram for the function.
20:15:06 <rmustacc> Able to toss one of these somewhere and maybe I can look in parallel a bit later today?
20:15:20 <sommerfeld> here are the two DW_TAG_subprogram entries for the function of interest: https://gist.github.com/Bill-Sommerfeld/e41f630ede7dd5e15d966e3cd5866a26
20:15:47 <sommerfeld> the whole dump is large (80MB)
20:16:11 <sommerfeld> I don't see any mentions of .cold in the dwarfdump output
20:16:14 <rmustacc> Sure, I just meant the raw binary.
20:17:14 <rmustacc> Is the .cold really going to be a 1:1 entry point?
20:17:24 <rmustacc> Or is it just a portion of it that's cold?
20:17:39 <sommerfeld> it's just part of the function that has been classed as cold
20:17:51 <rmustacc> If that's the case I'm not sure we can assume the arguments are the same.
20:18:18 <sommerfeld> it's part of the code of the function that has been relegated to the cold path.
20:18:31 <rmustacc> Yes, I get that.
20:19:04 <rmustacc> I just mean that if you were to lie about that in libdtrace, it would do the wrong thing.
20:19:11 <rmustacc> Well, libproc to libdtrace.
20:19:22 <rmustacc> The pid provider would think oh on entry to this function I have these args which it doesn't.
20:19:30 <sommerfeld> rather than all the function being in one contiguous range of addresses, it's in two discontiguous pieces.
20:19:39 <rmustacc> Yes, I understand.
20:19:50 <rmustacc> I'm just saying that lying about the argument types is going to confuse things.
20:20:35 <rmustacc> Particularly if there's ever a call into that other symbol.
20:20:42 <sommerfeld> it would be wrong for dtrace to think the .cold thing is an entry point -- because it's the middle of the function. 
20:20:54 <sommerfeld> I wouldn't expect any *calls* to foo.cold.  Just jumps.
20:21:04 <sommerfeld> from within the .cold segment and the main segment of the function.
20:21:06 <rmustacc> Sure, but that's not what your change told it.
20:21:23 <rmustacc> It's a valid entry in the symbol table, so how can it know those special semantics?
20:21:37 <rmustacc> The pid provider can happily instrument it. Maybe it won't find a synthetic entry probe.
20:22:28 <sommerfeld> from the point of view of read_args(), if PC is inside either range, you're in a function of 3 args of these types..
20:22:40 <rmustacc> Sorry, I realize this is just me coming off as saying no. Not my intent. Just want to figure out how we get there.
20:22:49 <rmustacc> But that's only because you know that in this case .cold has those semantics.
20:23:14 <rmustacc> That is you know that this binary was produced with pgo and that gcc has promised that, I think?
20:23:19 <sommerfeld> so the trick is to how to get this from the elf/dwarf records into CTF, that this function has two extents
20:23:31 <sommerfeld> rather than the usual one
20:23:53 <rmustacc> The problem is that it's two different symbols.
20:24:15 <rmustacc> And there is no real .cold function based on everything you described.
20:25:45 <rmustacc> What
20:25:57 <rmustacc> Just for my own curiosity, what's the disassembled cold version look like?
20:27:19 <sommerfeld> basically in foo, some conditional branches to foo.cold+offset (for varying offsets)
20:27:37 <rmustacc> If you can't give me the binary can I get the disasm?
20:27:53 <sommerfeld> I can get you the binary - give me a sec
20:31:58 <rmustacc> Take your time.
20:37:50 <sommerfeld> https://drive.google.com/file/d/1lFy_XFm-kcwCbTwuLrk4caRRdKRFrR5s/view?usp=drive_link
20:39:25 <sommerfeld> sha256 should be a103b7431ad8f8be82faa0c53cd25c3e57da633ffe09b49d168d2f431ba6d32b
20:41:18 <sommerfeld> note that there are many foo / foo.cold pairs and _PyEval_EvalFrameDefault is probably one of the most complicated ones in there
20:42:10 <sommerfeld> as it's the core of the bytecode engine
20:42:17 <rmustacc> Thanks, I have it.
20:42:47 <sommerfeld> which is why it's interesting for pstack -- every python function evaluation goes through it.
20:43:48 <rmustacc> So there are definitely some challenges writ large here as we look at as even if none of these have entry probes, they do definitely have return paths.
20:45:13 <rmustacc> It sems likely that gcc won't trick us up by pushing a frame pointer in these, though I'm not sure it makes this promise.
20:45:19 <rmustacc> As if it did, that would throw off saveargs.
20:45:26 <sommerfeld> yes, you'd expect the unlikely return paths to be relegated to the .cold section.
20:45:58 <rmustacc> I've learned to try not to expect too much from a compiler.
20:47:53 <sommerfeld> Indeed. 
20:48:39 <sommerfeld> give it an unrepresentative profile and I'd expect the hot return paths to be sent to .cold...
20:54:22 <sommerfeld> I've seen FDO cause some really unpredictable performance regressions build to build because sometimes it nailed it and sometimes it didn't.
21:10:38 <sommerfeld> actually, given the documentation for gcc's -fshrink-wrap I can totally believe there might be frame creation in the .cold extent of the function.
21:10:54 <sommerfeld> https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html#index-fshrink-wrap
21:16:34 <sommerfeld> ah, -freorder-functions enables this splitting even absent profile feedback.   minimal example: https://godbolt.org/z/9rYbzoh4x
21:28:58 <sommerfeld> minimal example without compiler warnings: https://godbolt.org/z/GKdzzvj88
21:30:41 <rmustacc> Yeah, that's why we disable this stuff in illumos.
23:20:57 <gitomat> [illumos-gate] 16250 use zfs_nicebytes() to print human-readable sizes -- Toomas Soome <tsoome⊙mc>