that also not possible as then you can't set the secondary-macs for the 2nd instance

has anyone tried to compile vitess on smartos?

sjorge, in omnios a guy created the interfaces in the global zone and then he attached the interface using { "physical": "test1vip0" } in the regular zone ...but there no such option in smartos

neuroserve: go stuff is usually pretty good at just working on SmartOS, and if not there are plenty of examples of where we've fixed things, it's just unfortunate that go makes that so difficult compared to other languages

jperkin, speaking of Go, the current Minio can no longer be compiled under SmartOS due to a library that has removed support for Solaris/Illumos... It would be great to try to get them to reinstate the support

is that the kubernetes one or am I thinking of something else?

no kubernetes

ah no, that was grafana: kubernetes/kubernetes #120498

-> minio/minio #17398

"yeh we don't support illumos so we're going to gratuitously break all dependents on our code" :/

ok, I think that might get messy with how go modules work and will likely just mean lots of patching dependents again, sigh

xmerlin might work for a zone, don't think it would work for a KVM/Bhyve VM

sjorge, it's a starting point but actually smartos doesn't support it so ...it works only in omnios

VRRP VNIC works fine if you precreate them and not depend on the zone scripts to do so on OmniOS

there no way to use it in a zone created using vmadm

Theoretically if you don't want to/cant pass a extra nic to the VM via ppt, you could probably setup a extra netwerk that does not use VLAN tagging and have the switch deal with it

Then promisc will just see the normal untagged traffic only

That SHOULD work

But I just ended up passing the nic to the VM as i would be dedicated it for that purpose anyway

It would be better if the wiki example worked and it was possible to set up with VRRP inside the zone... I suspect it used to work properly before the changes.

sjorge, I cannot pass the nic to the vm ...so I need another solution

on my servers I have only 2x25Gbit cards

danmcd: you know a bit about the netstack, can we somehow configure a vnic with a vlan set to only receive traffic for that vlan in promisc mode? (so no other vlans and all traffic for the vlan it has configure arrives untagged)

that would do the trick i think, but i dont think its possible atm

Have you tried not using vnic, but a *vlan* network object?

i.e. `dladm create-vlan ...` instead of `dladm create-vnic... ` ?

It might be syntactic sugar, but I wanna say things might be a hair different under the hood.

Also... what actual-NIC are you using for this?

(If you're using i40e with the promisc workaround, e.g. I'd be VERY VERY interested in knowing if this problem only exists there...)

IIRC, it should be

i think vnic doesn't allocate new rings

err vlan

while vnic does

i.e. I think a vnic object just uses the existing rings but inserts/removes vlan tag

Where a vlan object allocates a ring, huh?

a vnic definitely does

which is why some people have problems with slow-booting VMs on i40e

if they're using jumbo frames

(probably other NICs too, but i40e seems to be the most notable)

also means more kernel memory used

granted, i think mlxcx might be worse in that regard

had a system w/ 2 4-port aggrs + a couple vnics on top of them

w/ jumbo frames

kernel needed about 76GB of RAM basically just to get started

well userland would come up, but the instant you tried running anything, you'd run out of ram (and likely trigger a pageout deadman) within minutes

We can help that along as I start digging through mlxcx for CX-6.

having fixed i40e and vmxnet3... and likely most other drivers supporting jumbo frames likely have a similar issue (more acute of a problem if they support large numbers of hw rings)

i'm working on prototyping some opt-in bits to mac that should take care of all of this (and simplify drivers a fair amount)

and will probably present it as an IPD

Sounds IPD-worthy for sure.

ideally, when advertising ring support, the driver can specify some additional info, as well as some alternate functions

and then for TX, their callback gets passed a struct w/ checksum/offload info, and an array of ddi_dma_cookie_ts that already have everything in the packet copyied/bound/split as necessary

so all they need to do is write the paddr + size + driver specific metadata (e.g. SOP, EOP bit, checksum offload info, etc) to the descriptor entries

and then whatever needed to kick the ring

plus a callback to notify when entries have finished transmitting and can be reclaimed

and the (again opt-in, drivers not doing this see no changes, and hopefully don't even hit any changed code paths)

mac layer handles all of that

plus something similar for RX

so all that buffer management could go away

all the complex logic for mapping descriptor<->mblk largely goes away

and with mac handling the buffers, initially it'd be fairly simple, but it'd allow for future improvements

(one thing is that it allows TX/RX bufs to share a common pool, but could add things like dynamically adjusting size, or maybe have another pool of smaller bufs

e.g. 128 bytes or such (at least for our stuff, probably 30% of the packets on the wire fit into that size (or maybe 256bytes)