05:08:40 that also not possible as then you can't set the secondary-macs for the 2nd instance 06:43:45 has anyone tried to compile vitess on smartos? 06:44:07 https://github.com/vitessio/vitess/releases 09:42:04 sjorge, in omnios a guy created the interfaces in the global zone and then he attached the interface using { "physical": "test1vip0" } in the regular zone ...but there no such option in smartos 09:44:23 neuroserve: go stuff is usually pretty good at just working on SmartOS, and if not there are plenty of examples of where we've fixed things, it's just unfortunate that go makes that so difficult compared to other languages 10:17:34 jperkin, speaking of Go, the current Minio can no longer be compiled under SmartOS due to a library that has removed support for Solaris/Illumos... It would be great to try to get them to reinstate the support 10:19:26 is that the kubernetes one or am I thinking of something else? 10:19:55 no kubernetes 10:19:58 ah no, that was grafana: https://github.com/kubernetes/kubernetes/pull/120498 10:20:16 -> https://github.com/minio/minio/issues/17398 10:20:20 "yeh we don't support illumos so we're going to gratuitously break all dependents on our code" :/ 10:23:40 ok, I think that might get messy with how go modules work and will likely just mean lots of patching dependents again, sigh 13:06:58 xmerlin might work for a zone, don't think it would work for a KVM/Bhyve VM 13:07:54 sjorge, it's a starting point but actually smartos doesn't support it so ...it works only in omnios 13:08:02 VRRP VNIC works fine if you precreate them and not depend on the zone scripts to do so on OmniOS 13:08:31 there no way to use it in a zone created using vmadm 13:09:04 Theoretically if you don't want to/cant pass a extra nic to the VM via ppt, you could probably setup a extra netwerk that does not use VLAN tagging and have the switch deal with it 13:09:15 Then promisc will just see the normal untagged traffic only 13:09:20 That SHOULD work 13:09:37 But I just ended up passing the nic to the VM as i would be dedicated it for that purpose anyway 13:10:09 It would be better if the wiki example worked and it was possible to set up with VRRP inside the zone... I suspect it used to work properly before the changes. 13:10:50 sjorge, I cannot pass the nic to the vm ...so I need another solution 13:11:16 on my servers I have only 2x25Gbit cards 20:36:11 danmcd: you know a bit about the netstack, can we somehow configure a vnic with a vlan set to only receive traffic for that vlan in promisc mode? (so no other vlans and all traffic for the vlan it has configure arrives untagged) 20:36:37 that would do the trick i think, but i dont think its possible atm 20:36:54 Have you tried not using vnic, but a *vlan* network object? 20:37:36 i.e. `dladm create-vlan ...` instead of `dladm create-vnic... ` ? 20:38:13 It might be syntactic sugar, but I wanna say things might be a hair different under the hood. 20:38:31 Also... what actual-NIC are you using for this? 20:39:13 (If you're using i40e with the promisc workaround, e.g. I'd be VERY VERY interested in knowing if this problem only exists there...) 20:41:59 IIRC, it should be 20:42:15 i think vnic doesn't allocate new rings 20:42:20 err vlan 20:42:22 while vnic does 20:42:42 i.e. I think a vnic object just uses the existing rings but inserts/removes vlan tag 20:42:59 Where a vlan object allocates a ring, huh? 20:43:17 a vnic definitely does 20:43:38 which is why some people have problems with slow-booting VMs on i40e 20:43:44 if they're using jumbo frames 20:45:01 (probably other NICs too, but i40e seems to be the most notable) 20:45:07 also means more kernel memory used 20:45:25 granted, i think mlxcx might be worse in that regard 20:45:44 had a system w/ 2 4-port aggrs + a couple vnics on top of them 20:45:49 w/ jumbo frames 20:46:29 kernel needed about 76GB of RAM basically just to get started 20:47:18 well userland would come up, but the instant you tried running anything, you'd run out of ram (and likely trigger a pageout deadman) within minutes 21:10:19 We can help that along as I start digging through mlxcx for CX-6. 21:15:15 having fixed i40e and vmxnet3... and likely most other drivers supporting jumbo frames likely have a similar issue (more acute of a problem if they support large numbers of hw rings) 21:16:26 i'm working on prototyping some opt-in bits to mac that should take care of all of this (and simplify drivers a fair amount) 21:16:39 and will probably present it as an IPD 21:17:07 Sounds IPD-worthy for sure. 21:17:51 ideally, when advertising ring support, the driver can specify some additional info, as well as some alternate functions 21:18:42 and then for TX, their callback gets passed a struct w/ checksum/offload info, and an array of ddi_dma_cookie_ts that already have everything in the packet copyied/bound/split as necessary 21:19:21 so all they need to do is write the paddr + size + driver specific metadata (e.g. SOP, EOP bit, checksum offload info, etc) to the descriptor entries 21:19:35 and then whatever needed to kick the ring 21:20:04 plus a callback to notify when entries have finished transmitting and can be reclaimed 21:20:35 and the (again opt-in, drivers not doing this see no changes, and hopefully don't even hit any changed code paths) 21:20:47 mac layer handles all of that 21:20:53 plus something similar for RX 21:21:03 so all that buffer management could go away 21:21:31 all the complex logic for mapping descriptor<->mblk largely goes away 21:22:08 and with mac handling the buffers, initially it'd be fairly simple, but it'd allow for future improvements 21:22:55 (one thing is that it allows TX/RX bufs to share a common pool, but could add things like dynamically adjusting size, or maybe have another pool of smaller bufs 21:23:43 e.g. 128 bytes or such (at least for our stuff, probably 30% of the packets on the wire fit into that size (or maybe 256bytes)