Can bhyve hot swap interfaces/ disks?

jclulow: it's been a while since I thought about this issue, but one idea towards that direction is to add a new mblk flag that is only set when a packet crosses mac-loopback, as that's the problem datapath. And then something higher up the Rx path could look for that flag and decide to copy the mblk. I had something like that in mac for a while (HW_LOCAL_MAC) in SmartOS to serve other purposes, but ripped it out later.

But honestly I think I like sommerfeld's idea a lot.

mac-loopback is really a double edged sword

because we pretend its like going across the wire, but of course its not in subtle ways

Also, for a long time I've been of the mind we should just have a tunable to disable mac loopback either globally, per link, or both

because at least then when you hit Weird Shit (TM) you have a way to get out of trouble in production

'help! i'm trapped in a stack deeper than the marianas trench'

rzezeski: which? the adaptive copy?

jbk: yeah, there are cases where "punt to a queue that someone will look at while their stack is shallower" is the right answer.

So now that I am awake again: 1) close the current man page cr and bugas the consensus is this is not the right thing to do? 2) Open a new bug linked to 13463 to change the default in viona to always copy (but leave the logic for bnx in place) ?

That gets us in a safe place without the sysadmin having to do anything, then when someone has $time_to_spend look at a proper fix for 13463?

s/bugas/bug as/

I didn't really do a/b benchmarks but for me the performance impact was not really noticeable, might be because without copying even when the vm completely locking up, it would have random mini stalls (I guess because it also ran out of tx buffers but got them back before completely locking up) so the overal performance I'm seeing now (mostly on NFS) is better because it can now consistently hit my disk performance bottle neck.

s/vm completely/did not completely/

Trying to compile github.com/meetecho/janus-gateway on SmartOS, don't know if this is gonna work 😄

first ./configure error: checking for dlopen in -ldl... no - configure: error: libdl not found.

Any clues what it's trying to tell me here?

dig into config.log to see what the actual error is

sjorge: I think documenting the tunable is useful if we change the default. Someone might notice the performance regression, be using it in a config that works with the buffer loaning, and want the speed back.

sjorge: that's for the viona buffers stuck in queues elsewhere problem?

yes

sommerfeld OK, so I guess that needs to go in in tandem with the change of default. I'm not sure how to do the later.

As we still want the logic for bnx to always copy

And I'd need to reword it so it's the inverse of what it says now

is there anyone with the relevant bnx hardware who is also using bhyve?

it would be nice to collect some actual data around that

*bnxe

I only have intel hardware myself, nic wise at least

only ixgbe tbh, the i40e is a non working garbage fire for me

That logic came from pluribus, and they were reportedly on an old OpenSolaris-y codebase of an unknown vintage

I'm willing to bet that they were using the old closed driver, and that the open source one we have is not plagued by the same issue(s)

jperkin (LIBERA-IRC): thanks, will do

[illumos-gate] 15438 dlled fails to build if fprintf() is marked _PRINTFLIKE() -- Robert Mustacchi <rm⊙fo>

I'm fairly sure I've seen #15461 (fenix?) and thought I was somehow an idiot or going crazy

BUG 15461: Deletion of IPv6 addrconf address before static IPv6 address on same interface results in undeletable addrconf address (New)

↳ illumos.org/issues/15461

I'm so glad someone got to the bottom of it

I think I've run into that before too.

i've got a fun one i have to chase down w/ zfs...

resumable receives can fail at the _very_ end with 'dataset has been modified'

(using zfs recv -F fixes it, but would be nice to figure out the problem)

Does anybody know what could cause dladm: warning: failed to activate varpd instance: Bad file number when creating an overlay? I tried checking all files being present and all, but all seems to be as it should

is /var/run/varpd/varpd.door there, etc?

Yep and varpd is online and working

it only fails with the files search plugin

What file did you specify?

I was trying with "/etc/vpcs/search/testnet1.json" as files/config property the file exists

Huh, I found the problem. create-overlay must have an empty json object inside the file for it to work

What were the contents of the file otherwise?

Empty

after a simple touch

We need to improve the error message, but that won't work.

I mean, an empty object also is going to not help you very well at all. Nothing will work.

Well at that point no ZOnes are configured yet :) Or are you telling me I can't modify the file at all after?

It is read and used at the time the overlay is created.

It's intended where you know the design up front for experimenting.

Oh

It's been too long so I don't remember if you can ask it to change the property and it'll do something useful there, but I'd guess probably not.

So we have no multihost support other than direct at the moment?

I mean, files works for that.

But the direct bit is the most common thing that you'd use in a cross-platform env.

I could probably re load varpd

Otherwise the directory implementation has to come from somewhere.

varpd doesn't read the file every time you try to use it.

It reads it when it sets things up.

Anyways, things could definitely be made better here, just wanted to give you context for what to expect.

Thank. It will help set the API leayer I am making up proper

*Thanks

Yeah, if you're trying to build something more dynamic, I would write your own plugin.

That was what we did at Joyent when I did this originally.

I can get away with direct but was thinking files would be more scalable before I make my own plugin :)

We would go to the directory service that hosted this information and get it dynamically.

Haven't played with rusts c export

yet

Just remember it's a private interface.

But the thing to think about are what semantics you're trying to achieve.

pretty much my idea aswell.

I am ok with private interfaces :)

The files backend as the manual page mentions is for static configurations.

Otherwise, you could fix multicast groups and just use that to get going in a small-scale depolyment.

Otherwise if it's API driven, your directory store needs to be able to issue invalidations.

Yeah adding and removing zoney dynamically is required.

Yeah, files is wrong for this then.

At least, as I implemented a long time ago.

github.com/TritonDataCenter/illumos…/svp/common/libvarpd_svp.c#L18-L337 is the starting point for the dynamic module we actually used as part of this.

Yep :) I'll stick with direct for now. I'tll get me started and then I can go from there.