Deploy a Multicluster Ingress on Google Kubernetes Engine

In the previous part of this tutorial for Federated Kubernetes, we have configured a geographically distributed Kubernetes cluster through federation. Now, we will deploy an application in the federated cluster, and expose it via a global ingress endpoint.

While the federated control plane orchestrates the deployments across clusters, we need to configure an ingress that can route the traffic to the nearest cluster based on user’s location. In case one of the clusters suffers from an outage, the ingress must be intelligent enough to route the request to another cluster that’s healthy.

On the Google Kubernetes Engine (GKE), the global ingress is configured as an HTTP load balancer that talks to the backend NodePort on each cluster.

The Google Cloud Platform (GCP) load balancing provides global load balancing for HTTP(S) requests. Customers can configure URL rules that route some requests to one set of instances and route other to other instances. Requests are always routed to the instance group that is closest to the user, provided that group has enough capacity and is appropriate for the request. If the closest group does not have enough capacity, the request is sent to the closest group that does have the capacity.

Google took advantage of the features exclusively available in GCP load balancer.  Even though it is not natively integrated with gcloud SDK — the official command line interface for GCP — Google’s engineers published kubemci, the tool that can do global, multi-cluster load balancing.

Once you have kubemci downloaded to your Mac or Linux box, you are ready to deploy and expose the application.

Since kubemci needs application credentials to access GCP resources, we will run the below command.

Application Default Credentials (ADC) provide a method to get credentials used in calling Google APIs. The gcloud auth application-default command group allows you to manage active credentials on your machine that are used for local application development.

We also need to pass each cluster configuration to kubemci. For this, we will create a new configuration file with the credentials. This will act as the custom kubeconfig file exclusively used by the tool.

Don’t forget to replace the project id with your GCP project id.

We will also create a static IP address for our global ingress.

The basic configuration is done, and we are ready to deploy and expose the workload.

Google’s engineers have built a nice application that prints the details of the zone where it is being served. We will deploy this sample application in our federated cluster.

Create a file called zone-printer.yaml with the following markup:

This creates a deployment with six replicas and also exposes that through a NodePort service.

Let’s go ahead and submit this to the federated control plane.

This simple command kicks off an orchestration that pushes the deployment into each registered cluster. Since we asked for six replicas, each cluster runs two pods.

You can verify it by checking the control plane followed by each cluster.

The global federated control plane reports 6 pods – 2 pods in each cluster.

Let’s switch the context to each of the clusters.

It’s the same case with the service. The below screenshot shows that each cluster has a NodePort assigned to the zoneprinter service.

Now, it’s time to create the ingress object. First, we will define the ingress, and then submit it via the kubemci tool. Note that kubemci is independent of the federated cluster concept. It can be used with multiple standalone clusters that are not federated. I am demonstrating how these two can be combined for exposing federated workloads.

The ingress definition has an annotation that associates the object with the static IP address we created earlier. The annotation, gce-multi-cluster is a hint to GKE to provision a global HTTP(S) load balancer.

Run the below command to create the ingress object.

This single step results in a series of configuration tasks involving the creation of load balancers, firewall rules, health checks, and proxies.

Keep checking the GCP Console for the health status. Once all the clusters are registered with the ingress, it should show all the clusters as healthy. Wait till you see 3/3 in the Healthy column of Backend services section. It typically takes a few minutes for the health check to pass.

It’s time for the final test. To verify the geo-routing feature, I am using Opera browser with VPN enabled. You can choose the location of the VPN to emulate the traffic originated from a different place.

Depending on your VPN location, you will end up seeing a different output in the browser. The flag and the location indicate the GKE zone serving the request.

Kubernetes is slowly but steadily becoming the preferred choice for running highly-available and distributed workloads. Features such as federated clusters and global ingress bring additional capabilities to the platform. Combined with globally-distributed, and strongly consistent database services such as Google Cloud Spanner, it is now possible to run applications that are deployed across the globe.