Traffic Routing in Ambient Mesh
In September 2022, Solo.io and Google announced a new Istio data plane mode called Istio ambient mesh.
Istio ambient mesh is a new, open source data plane deployment model that doesn’t use sidecar proxies. Instead, it uses a per-node component called ztunnel for mTLS, identity, and L4 traffic management and a waypoint proxy (Envoy instance) for L7 traffic management.
The ambient mesh deployment model is much leaner than the sidecar data plane deployment model, allowing for incremental adoption of service mesh features and making it less risky. As ambient mesh includes fewer components, this leads to reduced infrastructure costs and performance improvements, as captured in this blog post. Ambient mesh does all this while retaining all the service mesh critical features, including zero trust security.
In this blog post, we’ll discuss how ambient mesh works behind the scenes and how the components work together to achieve the above-mentioned features without sacrificing the zero trust security, telemetry and traffic management features built into Istio.
How Is Traffic Intercepted?
Ambient mesh relies on the Istio Container Network Interface (CNI) plugin that configures each node in the cluster with appropriate rules and configuration needed to intercept inbound and outbound traffic and to correctly route the packets to the resident ztunnel proxy.
There are two mutually exclusive redirection mechanisms the Istio CNI plugin uses to achieve this: the iptables rules and eBPF (Extended Berkeley Packet Filter).
While both mechanisms achieve the same goal, the eBPF approach is simpler and more performant as we can send the packets directly to their destination without going through multiple routing configurations, which is the case with the iptables rules.
The eBPF program part of the Istio CNI plugin is attached to the ingress and egress hooks of the Linux kernel, where it intercepts and directs the traffic to the resident ztunnel instance.
You can read more about how both eBPF and iptables mechanisms work in this post, or join us for one of the upcoming Ambient Mesh — In-Depth Routing Analysis workshops at Solo Academy.
Ztunnel Proxy
The new Rust-based ztunnel proxy is responsible for mTLS, authentication, L4 authorization and telemetry in the ambient mesh. Its job is to proxy the traffic between ambient mesh pods. Optionally, the ztunnel proxies to L7 waypoint proxies, ingress and, in the future, egress proxies.
Ztunnels on different nodes establish a tunnel using HBONE (HTTP-Based Overlay Network Environment). Similarly, the tunnel gets established between the ztunnel and the waypoint proxy, if one exists. The tunnel that’s established between the ztunnels allows the source ztunnel to connect to the destination workload on behalf of the source workload.
On startup, ztunnel connects to Istio’s control plane and retrieves its configuration, which contains information about all pods running on that node. This configuration called the workload xDS configuration, has two parts.
The first part contains workload information, including the namespace name, IP address, workload name, service account name, etc. This is where the optional waypoint addresses are specified, so ztunnel can decide where to route the request.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
"10.244.0.9": { "workloadIp": "10.244.0.9", "waypointAddresses": [ "10.244.0.19" ], "gatewayAddress": null, "protocol": "HBONE", "name": "httpbin-2-67647575fc-x9vcz", "namespace": "default", "serviceAccount": "httpbin-2", "workloadName": "httpbin-2", "workloadType": "deployment", "canonicalName": "httpbin-2", "canonicalRevision": "v2", "node": "kind-control-plane", "nativeHbone": false, "authorizationPolicies": [ "default/httpbin-2-policy" ], "status": "Healthy" } |
The second part contains information about the L4 authorization policies, the information the mesh operator provides using the AuthorizationPolicy API. Any authorization policies are referenced in the workload configuration, so the ztunnel knows which workloads to enforce the policies.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
"default/httpbin-2-policy": { "name": "httpbin-2-policy", "namespace": "default", "scope": "WorkloadSelector", "action": "Deny", "groups": [ [ [ { "principals": [ { "Exact": "cluster.local/ns/second/sa/sleep" } ] } ] ] ] }, |
Waypoint Proxies
While ztunnel is Rust-based and responsible for L4 traffic, the waypoint proxies are Envoy proxy instances that handle L7 traffic management.
Anytime you want to apply HTTP-based policies, such as request routing, fault injection or authorization policy using HTTP concepts, you’ll have to use a waypoint proxy. As we don’t typically have a waypoint proxy on both sides of the communication as with sidecar mode, L7 policies in ambient are applied at the destination waypoint proxy.
To deploy a waypoint proxy, we use the Kubernetes Gateway resource. By default, the waypoint proxies get deployed per namespace and act as a gateway into that namespace. The second option is to deploy the waypoint proxy to act as a gateway for a specific service account. In both cases, the actual proxy deployment is created in the namespace.
The namespace-scoped proxy enforces all policies for the namespace. Any traffic entering the namespace will go through the waypoint proxy. For scenarios where we want to apply policies for specific workloads (or workloads using the same service account), we’d deploy a per-service account waypoint proxy.
The ztunnel uses the configuration and determines whether the request should be sent to a waypoint proxy or directly to the destination workload.
Waypoint proxy is configured to listen on port 15008 and uses a combination of Envoy filter chain matchers, different filter chains and a concept of internal listeners to correctly match the traffic and route it to the destination workload. As waypoint proxies are instances of Envoy proxies, the requests go through a set of filters similar to the ones in the sidecar deployment mode. For example, the CORS (cross-origin resource) filter, RBAC (role-based access control) filter, fault filter, stats filter, etc.
The amount of configuration at the waypoint proxy is significantly less compared to the configuration in the sidecar mode, where each sidecar proxy is configured and knows about every other destination in the mesh. The design using waypoint proxies is much more scalable. Although the configuration is still verbose, debugging is easier as all configuration is at the destination waypoint instead of being spread between the client and server proxies.
Conclusion
Istio ambient mesh offers advantages over the traditional sidecar data plane deployment model. Its noninvasive and leaner design allows for a less-risky and incremental adoption of the service mesh. As fewer components are involved — per-node ztunnel proxies and optional waypoint proxies — the adoption of the ambient mesh directly leads to reduced infrastructure costs and performance improvements. The design also allows for a better separation of concerns between L4 and L7 and, consequently, easier debugging.
As a co-founder of the project and development leader in the Istio community, we have launched an advanced workshop called Ambient Mesh-In-Depth Routing Analysis, alongside our beginner workshop “Get Started with Istio Ambient Mesh” at Solo Academy. This workshop goes into detail about how ambient mesh routing works. It includes hands-on labs and teaches you about the role of the Istio CNI plugin and how it configures the node, ztunnel and waypoint proxies for traffic interception and how the requests flow from source to destination workloads.
