maybe a remedial question, but my instances (I'm working with bhyve at the moment) don't pick up the DNS servers of the network they start up with. Instead, they seem to pick up the default nameservers from the GZ (8.8.8.8 and 8.8.4.4). I need them to come up with the internal nameserver that I have defined on the network in Triton. Isn't that how it should work?

I'm reading about changing the nameservers globally, but it's pretty clear from the documentation that that's discouraged.

(i.e., the nameservers on the headnode USB key)

jdt: This may be a bug with the image. Someone else reported the same thing but I haven't had a chance to look into it yet.

I'll be looking into that next week.

Awesome, thanks bahamat.

That's the latest debian-12 image, fwiw.

bahamat: I think that there is a possibility that I found a bug in cloud-firewall. I am not sure if it is in the API, agent, or in ipfilter/fwadm. It is specifically pertaining to writing rules with tags as the source and destination.

I have never found an issue with writing rules based on the concept of tagged resources in triton before but this is kind of peculiar and I wonder if it is by design or a bug.

Provision 2 instances with fabric NICs and 'ip_spoofing: true', enable firewall on both, set tags on both like: 'environment=development'. Write a rule that says "From tag "environment" = "development" TO tag "environment" = "development" ACCEPT

Add an IP alias on both fabric nics within the guest. Assuming NIC IP is 192.168.128.11 and 192.168.128.12 respectively then (assuming linux bhyve btw)...ip addr add 10.20.10.11/32 dev net1 src 192.168.128.11 and

ip addr add 10.20.10.12/32 dev net1 src 192.168.128.12

then add routes to each other

ip route add 10.20.10.12/32 via 192.168.128.11 dev net1 src 192.168.128.12

and vice vursus

Try to ping either IP from either guest and it will work as expected

(ICMP by default allowed with no source and destination restrictions globally in Triton)

But you open a port on one like: `nc -l 10.20.10.11 4000` and try to connect from the other `nc -vz 10.20.10.11 4000` sourcing 10.20.10.12...nothing will ever pass and even worse if you run tcpdump in the guest you will never see the traffic.

That is what I'd expect.

To get around it I had to set a rule like so: FROM subnet 10.20.0.0/24 TO tag "environment" = "development" accept

You have added addresses inside the instance that triton cannot see.

TBH myself as well

I would not expect aliases added inside a guest with ip spoofing to function with the set of firewall rules described above.

But I conceptually I understood that if I set source and dest as tags, that triton wouldn't care about the src/dst headers

That's not quite right.

At least with spoofing disabled on the guest

The two don't really interact.

I understand from the OS/NAPI/CNAPI/FWAPI perspective that they do not

When you're using tags like that, implicit in it is the addresses that the system knows about.

But I think the true lack of understanding is in how fwapi permits and denies based on tags to begin with maybe?

so it is dependent on napi then?

and writes base static rules?

It is definitely dependent on napi and is using the information about the state of the instances / nics that the control plane knows about to write out rules.

Ahhhhh it is statefule within triton

It is not implicitly tagging packets on the network to assosciate it with an instance.

This is a good lesson learned

It doesn't have to be stateful in the calssic firewall sense.

right

But it does rely on the information triton knows to create the rules.

but in the api service sense

Thank you sir! Always a pleasure.

Oh but definitely not a bug then

I probably wouldn't call it one, but it can be surprising.

I'll see if I can't update docs to include this caveat and send over a PR if MNX will approve.

I would just make it clear that firewall rules based on tags, instance names, etc. only ever operate on information that triton knows about.

I was testing some kubernetes CNI's for a customer and spent wayyyyy too many hours trying to figure out why my cluster wouldn't come up correctly before I realized that the firewall was blocking.

I had assumed that I enabled an impliceit permit for all instances tagged as such, regardless of what the CNI was doing to traffic.

It's best to basically think that ip spoofing doesn't really work with anything.

To do what you'd want you'd need to write a custom header on each packet.

Not bad advice at all traditionally per triton operator documentation, concepts, or best-practices. I was just trying some "hacky" new stuff that I thought was cool and learned about this the hardway. Thought mabye I could spread some knowledge in-case this was misunderstood by anyone besides myself.

What do you mean on hte customer-header comment? Like a masquerading the ip?

No, I mean to make it semantically what you want and ignore the IPs as where this is originated from then you'd need some other way to id the packet's originator.

So that would be something like a geneve header.

Aha, I was specifically avoiding encapsulation protocols

I mean, you said it was a fabric NIC, so it's there anyways.

I'm referring to the triton side.

because I mean, vxlan over vxlan and MTU, etc, doesn't sound like too much encapsulation or packet forwarding overhead at all haha

You wouldn't trust the gest for that.

*guest

Right, it is a fabric.

I had the option of geneve or vxlan encapsulation

What I'm saying is for that to be implemented, the fabric would have to support it. It doesn't matter what the guest does.

I understand completely now.

I have a customer who uses the default calico vxlan tunneling CNI between numerous instances without spoofing enabled, and it works fine.

Even with cloud-firewall.

I just chose not to do it that way.

Maybe an RFD feature request is more appropriate here, rather than a documentation update, to support spoofing only on fabrics, and maybe some form of IP discovery on existing NICS. Not that it would ever actually get implemented if a big customer didn't need it.

Or maybe I'm over thinking this and encapculating network fabrics over existing overlay's isn't that big of a deal

Sorry to blow the room up :)

I think I would be a bit concrete about what it is you need explicitly. The things you listed just now don't immediately relate back to the issue we were discussing.

But obviously, I'm not at MNX.

Yeah, I went in circles a bit sorry about that. IMHO I am torn between thinking that "tagging" shouldn't care about "src" or "dst" at l3 when "tagging" is set as both src and dst but I also support the current flexibility, supporting both l3 src/dst and the concept of resource "tagging" together in the same src/dest rule. So basically I have no idea what I am looking for ATM except to share my

experience and maybe add some updates to documentation for future users I think.

I mean, you shouldn't have to think about the implementation of how it does it, except what's pertinent is to know that Triton will only do it based on what it believes should and shouldn't come from the instance.

It doesn't play well with IP spoofing and a bunch of that is just evolution.

And the fact that not all traffic in Triton is encapsulated so it gets more nuanced.

Triton is a beast. I read an RFD regarding multiple availability-zone support in NAPI recently that gave me a headache. So I totally understand the apprehension in making it anymore difficult. Your 100% correct. I'm just going to remember this in the future and move on lol

That's not saying one shouldn't try to improve things. But just useful to understand what's going on and yeah, sometimes you need to peer beneath the abstractions.

I just thought about a way that this could also be worked around through automation accessing the metadata API as well. Will post results at a later date.

One last question, if I were to spoof an IP in an instance, kind of like the example above but using NAT, masqerade all traffic sourced from it, as its interface IP, do you know if SmartOS detects this? I know that if you enable forwarding and masqerade out of a different interface it will, like in terms of a router, but I haven't tried spoofing like this on the same interface

barfield: Tags are only used to derive the IPs so that you don't have to hard code the IPs. It's a bit like DNS names. It doesn't filter based on DNS names. It uses the name as a lookup to find the IPs to filter on.

And enabling ip_spoofing breaks a lot of guarantees that Triton provides. You're taking things into your own hands in that case and you can't count on fwapi.

Ohhhhh so tagging like this, even if you make up your own tags, is effectively is tied to CNS kind of like the "tritoncli.ssh.proxy" or "triton.cns.services" tags

That actually explains a lot, even if I am not understanding it exactly correctly, and makes me understand why so many of my tagged rules work just fine and sometimes other rules are a bit difficult.