tozhu: The nic tags shouldn't affect anything with regard to logging in on the console, unless your config was unparsable.

bahamat: do you mean, it’s okay to change the tag in config file? I just switched the MAC address for external_mac and admin_mac

Yeah, there shouldn't be anything wrong with that.

okay, I will try, thank you very much

bahamat: I have rsync-ed a new platform on USB os dir, and changed the entry in boot/loader.conf, may I know how to change it back? seems it stopped on boot prompt

You should not be rsyncing platforms into the usb image. You need to use sdcadm for that.

Your best bet is to get a new known good usb image and copy your config onto it.

I use this script for migrating from grub to loader based USBs: gist.github.com/bahamat/b23e9180cd51a16a735ddb26774da165

You should be able to use that as a basis for reconstructing your necessary config on a brand new USB.

But I can't say how to fix your existing one because I don't know how badly it's broken.

bahamat: Thanks for the help, thank you

How do you repush vnics to a triton node? We've migrated a box between DCs and it seemed to have an issue bringing up the second aggr. We think this may be an issue with MACs and keys being the same on all LACP links on the switches. But now I'm trying to work out how to get all the vnics on the same aggr in triton.

It's been a while since I've done this, but that would nominally be the network tag.

So is the network tag on the aggr?

I've got external, internal, sdc_underlay and admin tags on the aggr now. It was admin with the others on the aggr that wouldn't come up.

The box seems to have picked up all these tags correctly against the first aggr after reboot but doesn't have any vnics brought up corresponding to them.

One vnic that was on the primary aggr that never had issues has survived all the reboots.

Seems this is a lot of custom config piled on top. Think I;ve found most of it now.

Thanks for that update.

So i think most of that was a combination of bobytrapped stuff that may never have survived an inplace reboot as well as customisation.

The multiple aggrs issue doesn't seem to be that though.

Anyone had experience running multiple aggrs against mellanox switches running nvidia cumulus?

I detest aggrs as a rule, BUT I'm pretty sure someone here (who I will not throw under the bus) runs aggrs with at least the Mellanox *cards*.

Unsure about that someone's *switches* however.

Do you use something else instead of aggrs or do you not run HA networking and mitigate it at other layers?

we have a few customers using mlxcx cards in an aggr, but i'm unsure of what sort of swtich they're connected to

(though not running smartos, but different illumos-based distro)

aside from the high kernel memory requirements, it seems fine