[illumos-gate] 16211 fexecve tests fail if output is redirected -- Andy Fiddaman <illumos⊙fn>

rmustacc: Follow up to my nvme question the other day, they were on a PI before that issue you pointed me to.

OK, great. Hopefully if they can update that should work.

Yeah, they did and reported that it's resolved.

Even better.

Glad it was that simple bahamat.

i was just saying earlier, the simples fixes are great :)

... as i'm about to get dragged into iscsi hell (which is somewhat of a redundant term :P)

though speaking of which, i should probably add decoding of it to snoop

Ouch. The fexecve() change looks a lot more damaging than originally expected

When I looked at the review it was a simple wrapper, but turn your back for a moment and it's quite a different thing entirely

What's wrong?

And/or why is it damaging?

Special casing parts of the VFS namespace in the kernel isn't exactly a simple wrapper.

Given you're changing the semantics of what exec() meant.

If I understand the fallout on OI correctly, then a zone or chroot with a pre-change libc will fail

And one of my common zone usages is to run older/different illumos distros in zones

And while we've never said that would work, for it to fail without warning would be a surprise

I haven't actually tested this, just catching up on the mailing lists and looking at the final commit.

(Been slightly busy on the day job, which is currently porting Java back to Solaris/SPARC.)

ptribble: only if you use the OI-special "nlipkg" zones

which let libc in zones get out of sync with the kernel.

Uh, wait, you're letting libc and the kernel get out of sync?

I guess we've been operating under different assumptions. We definitely treat the constraint as libc and the kernel must be in sync.

You have ABI compat for user software definitely.

But if I'm running an OI or OmniOS or XStreamOS zone on Tribblix (or an older version of any on a current version), which has worked perfectly for years, then I get into trouble

I know this doesn't happen often, and it's certainly frustrating regardless.

Generally, as long as the zone is older than the host compatibility is just fine

That's definitely not always true for core software.

That is, the expectation is that the consolidation is in sync. At least from a design constraint perspective.

Otherwise you basically are requiring us to make the system call ABI and not libc static.

I get that it doesn't break often and I don't want to say won't fix as it's important we figure out stuff here.

But at least from a constraints we've told people in the past and committed to it's been around libc/kernel and similar things with unstable interfaces being in sync.

Admittidly the last super obvious case of this was the pipe system call changes many, many years ago.

There was a more recent one, when a structure used over a door between the kernel and dlmgmtd changed.

I thought that nlipkg zones (and ipkg in OmniOS, which is the equivalent), enforced a minimum set of packages to be in sync with the GZ

was completely unaware of this until today but OI has a "nlipkg" brand which evidently doesn't enforce enough: docs.openindiana.org/handbook/systems-administration/#zones

Yeah that's totally unsafe

I expect lots of subtle changes we've made in the last few years will _seem_ like they're working, too, while potentially causing a bunch of silent data corruption

(any time you rearrange a struct for an ioctl, etc)

I need to refresh my memory, but that's what these are all about:

Until right now, I thought these in the gate manifests too

*were

I think if you want to be able to use zone images as-is you probably need to be a lot more like Solaris 10, basically: in your distribution, tightly manage releases and binary compatibility. Then you can make a brand like the s10 brand which is a mixture of replacing components that have changed, and doctoring system calls to create a kind of partial syscall stability against a specific era

But it is not really feasible to do that for an arbitrary set of foreign distributions that may have their own local patches, etc

What we're prototyping at Oxide is a little different from that. We have zone images, but we're treating them more like layers. The base system provides things like libc, and your zone image is an overlay on top of that of files that are _not_ base system files; e.g., your application, maybe a PostgreSQL client library, etc.

It's a bit like Docker, basically, except that instead of just depending on the kernel as an unseen artefact that happens to be outside your container, you're really depending on quite a lot of files and processes that are _inside_ your zone, but which didn't come from your image

The layered approach is how I've built production zones since day 1, essentially.

SmartOS does that too, with the caveat that you can't modify /usr so all of the zone-delivered software has to be in /opt instead etc

But the other use case is for building software against an older release, essentially the zone is like a sysroot rooted at /, so I can't modify the local libc

I think for that, you either need a built that understands how to use "gcc --sysroot" or similar: essentially a cross-compiler. Or, you need VMs running the older software release.

I'll have to look at the s10 change to see how hard it would be to mirror that

s/built/build process/

I definitely tripped over just assuming that the old libc would run in the past with software with the pipe change.

It definitely sucks compared to the convenience, but I don't know a good way to solve it.

If you just want to unwind this specific change inside zones, the s10 brand is a good model, I think. e.g., here is a filter on pwrite() -- github.com/illumos/illumos-gate/blo…_brand/common/s10_brand.c#L711-L736

so it seems that the reason why you might want to do that is so that the build doesn't see new features in libc that wouldn't exist in the older system.

sysroots for that kind of thing are fine. The hard part is being able to *run* it.

but even if you had a magic linker that produced a filter that limited the new libc to just the endpoints visible in the old libc, there could still be new kernel behavior detectable by configure which would give you something that wouldn't work on the old kernel.

Yeah you have to cross-compile. Your software needs to know it's cross-compiling and cannot do run-time tests.

That's why we have github.com/illumos/sysroot -- originally so that the Rust project can build rustc/cargo/etc binaries that run on illumos, but because of their infrastructure requirements they do that build on a Linux machine.

So they have a cross-gcc and cross-binutils configured to use the headers and libraries from illumos/sysroot for building and linking.

(and their build does not use autotools)

I could imagine actually some option to ld that would make a binary as something you're not allowed to execute on this system

*mark a binary

yeah, either cross-compile or run in a VM.

so that if a cross build _accidentally_ tries to do a runtime tst

*test

you'd know about it; a core file etc

Or I suppose just mount your build directory noexec maybe, even?

Filesystem W^X mode - at mount time, choose either ro or noexec for all filesystems..

haha

Yeah I guess you would need to make all the /tmp and /var/tmp stuff noexec too

alternatively, a procfs-based supervisor for a build that monitors all execs

or dtrace

You could do DTrace but you'd have to be careful to make sure there were no drops

Which are more likely the harder the machine is working on the build etc

just to get people on the same page, see: openindiana.org/pipermail/oi-dev/2024-January/033531.html and my (somewhat speculative) response at: openindiana.org/pipermail/oi-dev/2024-January/033532.html

I think that if OI add those self dependencies to package manifests, then nlipkg zones would also have received at least the key packages like system/library. It looks like danmcd added them to OmniOS when he added linked zones, but I don't know the history there.

sommerfeld - thanks for those emails links, I wasn't aware there was fallout from the change.

just as a sanity check, is there any good reason to use a semaphore like a mutex (e.g. init to 1)?

(speculative = I haven't reproduced it myself)

andyf: is there any good reason to not add those self-dependencies to illumos-gate?

it seems like the fact that you can't (easily) track who actually has the lock would be a downside, and I can't really think of any upside

I don't think so, no. Except that it becomes no longer possible to shoot yourself in the foot quite as easily. I'm hoping danmcd might have a bit more context on how they came to be in OmniOS, since I know the linked images landed there some time after OI

jbk: some people learned about semaphores first, and didn't move on to newer abstractions?

this is in the iscsi initiator code, i guess they didn't look at any of the other code in the module?

but, yeah, hba_sendtgts_semaphore looks like it could be a mutex.

(assuming that's what you're referring to..)

actually iscsid_config_semaphore

that one too.

new response from Stephan Althaus at: openindiana.org/pipermail/oi-dev/2024-January/033533.html

So early on ipkg zones (even upstream in Oracle Solaris) didn't properly link ngzs with GZs.

Once linked-images came along I asked Theo: "Do I keep ipkg unlinked or change it to match upstream?" He told me to keep them (presumably for least-surprise reasons). So I introduced `lipkg` into OmniOS.

see also hyrumslaw.com/: "With a sufficient number of users of an API,

it does not matter what you promise in the contract:

all observable behaviors of your system

will be depended on by somebody."

danmcd - I think I remember you asking us all on the mailing list too, at the time.

Oooh, there's receipts then.