I could imagine it might be for folks using zstd compression in OpenZFS

there've been at least two brief panics where we thought we had it.

the writeup says that at least the corruption is caught by checksums, as long as you haven't disabled them

Does anybody know of any door clients or servers that ever send more than one descriptor at a time?

Looking through illumos-gate, most of the examples I see send at most one descriptor in any given door_call/door_return

i can't think of any that i've written

okay cool, thanks jbk that's helpful. Just wanted to make sure I'm not missing an obvious use case.

sometimes i've done things where i've passed stdout to the door server so it can just write the output, i could see maybe also doing stderr

yeah okay, I could see that!

I could also see too, like, if a socket server wanted to upgrade without severing connections, it could maybe pass all its open connections to the new version over a single door call. Or something? But that feels really convoluted.

I'm hoping to find examples of such use cases, if there are examples available, rather than just guess at them.

Does any other descriptor-passing system call ever share multiple descriptors in one action? I guess fork would do that. But using ioctl/SEND_FD to send a descriptor over a pipe just does one at a time.

I like the idea over just passing a client's stdout/stderr to a door server, that really may be enough of a motivating example.

obviously you want to be careful about using it -- since those may block (may sure you account for that)

ah okay, that raises an interesting point -- could a client and server disagree about the value of O_NONBLOCK for a given descriptor?

Like say the client opens a file, uses fcntl to mark it nonblocking, and then sends it (without DOOR_RELEASE) to a server, could the server use fcntl to mark it as blocking?

I guess what I'm asking is, do you know if that behavior is per-resource, or per-process-per-resource?

In this case the question is to ask what is the resource and where is it set. Flags are kept on the file_t, not an fd. So then the question is what is the granularity of the file_t. I'm rereading to remind myself what happens in the send fd case.

rmustacc: thank you! I appreciate that. Can I assume that the same reasoning would apply for sockets? Like, the flags are kept on the socket_t, and that a socket_t would have the same granularity as a file_t? Or is that likely to be a whole different ball of worms?

I believe in most cases you're basically always sending the contents of a file_t when you're doing descriptor sharing like that.

A socket is no different than any other file.

You have a vnode from sockfs (or a STREAMS) device.

oh no shit? I thought that was just an illusion of libc.

You have a file_t that points to a vnode and fd's that point to file_t's.

It's pretty common on most systems. That's part of what the VFS does.

So it's all different at the vnode layer.

But above that it's just a file more or less.

so then when I am sending an fd between processes, I end up with two file_t structs that may not necessarily stay in sync after creation. Is that right?

I'm not certain.

I think we're passing the file_t by value effectively rather than as a reference.

When you fork(), I think you still refer to the same file_t. But it's been a while since I had to think about this at this level.

ahhh, that's interesting!

That's really interesting. I never considered that sending descriptors between unrelated processes might play out differently than forking, but that makes sense.

fork() is just duplicating file descriptors by definition, so it makes sense that they'd point to the same file_t, I think.

Yeah, fork() just bumps the reference on the file_t.

So I'm less certain on these other cases what's happening.

Fair enough. I bet I can cook up some examples and test to see if there is a difference. Like, if I fork(), I should be able to set flags on the file_t in the child and see those flags in the parent. But for unrelated processes, maybe not? That would be interesting to see.

But it's probably worth noting some flags exist on a per-fd basis, others on a per-file_t basis.

oh!

FD_CLOEXEC is on the fd.

ah yeah, okay, that makes sense

Now, that's the only fd-specific flag. Calling things ala F_SETFL or similar are on the file_t.

ah okay, so if I stick to things like O_NONBLOCK, O_APPEND, etc then I should be able to see if there's a differece between (say) fork() and door_call() when it comes to sharing vs duplicating the file_t structs

thank you both! That is super helpful.

So doors are going to be pretty different if you look at the impl.

No, it looks like they are sending the file_t by reference.

So yes, I expect that to be a way to cause them to be the same and that an fd in the descriptor is sending the file_t.

door_insert() makes this pretty clear.

I believe the same is true of a UDS sending an fd if you look at fdbuf_extract().

ah okay! Cool, I have not studied door_insert yet. I will take a gander at that.

Thank you again, I really appreciate this!

one of these days i'd like to fix the rather toxic doors()/fork() interaction

[illumos-gate] 15143 bhyve pause/resume should include vCPUs -- Patrick Mooney <pmooney⊙pc>

andyf is the vmm module for mdb needs more work I just remember we had this still on review code.illumos.org/c/illumos-gate/+/2286

→ CODE REVIEW 2286: 14525 Would like kmdb module for vmm (NEW) | illumos.org/issues/14525

[illumos-gate] 15518 Update various AMD CPU revision and socket data -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 15520 Ensure we can't hit AMD xsaves erratum 1386 -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 15521 Use Automatic IBRS when available -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 15522 Want support for Zen 4 CPU Performance Counters -- Robert Mustacchi <rm⊙fo>

[illumos-gate] 15523 zen_umc should capture memory speeds -- Robert Mustacchi <rm⊙fo>

pmooney: bhyve viona is getting about 4-6 gbps and a native zone is getting about 14-16gbps in iperf3 with mlxcx (connectx-6 dx).... curious if there are any tunables or something simple I may be able to do to increase viona perf?

It's possible adding TSO support to mlxcx so you can pass that through to a guest will probably help.

rmustacc: yea, we've got some remote storage that we want cloud users to be able to use in their bhyve machines, the storage boxes are 200gbps on linux, and was hoping to get clients that could do 20 gbps or so....

kicking ourselves in the ass right now.... we didn't get the chelsio cards because of the breakout cables we are using didn't work with sfp28, but we should just ate the ports.... I did warn my manager though and have the CMA email :P

Depending on packet sizes and rates (because iperf may not be similar to your actual workload), I think it'll be a bit challenging.

There are other challenges with the T6. It doesn't have full PCIe bandwidth for 2x100G.

the nvme-tcp client was limited by it

So, this is something where TSO may apply if you're doing 4K or greater writes.

Also, the idea of viona exposing and using multiple rings to the guest. There's a feature bit for multiple queues.

That doesn't help a single connection, but it's possible it may help. Unfortunately it's a lot of work for the experiment.

we could mount these directly from the GZ but i think we only have support for iscsi, we can expose iscsi from the server.... wonder if nvme-tcp vs iscsi perf is that much different*

No idea.

grr... annoying when bugs are hard to reproduce reliably

Heisenbugs are nobody's friend.

is there an easy way to map a datalink_id_t to a link name?

Smithx10: multi-queue viona support may help for larger instances (not so much on single-stream), but that work hasn't been done yet

also wouldn't be a big priority for me on c-bhyve

since I believe it requires virtio 1.0 compliance for the device

(which seems more within reach for propolis than it does on c-bhyve)

furthermore, I suspect that we'll have to switch viona to copy packets by default soon, thanks to the loopback issues if packets get queued somewhere in the host

which will further impact perf

(negatively)

pmooney: I know the product hasn't shipped yet, but how is propolis looking for home use these days?

propolis-server is definitely targetted for use with the rest of the control plane

propolis-standalone could be made into something easier to run by hand

Was there ever a Solaris on MIPS?

not that I know of

x86, sparc, ppc, ia64 (not sure how far that got out of the company)

[illumos-gate] 15449 bnxe MAC_PROP_MEDIA support -- Robert Mustacchi <rm⊙fo>

I wish I could arrange an "are you sure" before I do cat log/latest/nightly.log for the millionthng time, because I missed mail_msg

I never heard of a mips port either

as far as I remember IA64 was only done for ON, got killed before the rest of the OS was ported

MPK17 never had locks on peoples' office doors. (Young'uns might be thinking, "YOU HAD OFFICES?!?" but thats another time.) Once the IA64 bringup started, folks on those teams needed to install locks on their office doors and keep them locked when they were not in the office.

I think we inherited one of those locked offices when we moved into MPK17 a few years later and had to have a locked room to keep the Display PostScript sources in

^^^ wow!

danmcd: why? re: locks

Intel said, essentially, "IA64 docs must be kept under lock and key."

Don't know if you remember this far back, but Intel bet big on IA64/Itanic^Hium.

I know the bringup team had to keep any docs on pre-release Intel products locked up even long after Itanium died - they really didn't want them leaking

heh