Data / Kubernetes / Open Source / Sponsored / Contributed

Maximize K3s Resource Efficiency with Calico eBPF Data Plane

28 Oct 2021 8:08am, by

Reza Ramezanpour
Reza is a developer advocate at Tigera, working to promote adoption of Project Calico. Before joining Tigera, Reza worked as a systems engineer and network administrator.

Amazon’s custom-built Graviton processor allows users to create Arm instances in the AWS public cloud, and Rancher K3s is an excellent way to run Kubernetes in these instances. By allowing a lightweight implementation of Kubernetes optimized for Arm with a single binary, K3s simplifies the cluster initialization process down to executing a simple command.

In an earlier article, I discussed how Arm architecture is becoming a rival to x86 in cloud computing, and steps that can be taken to leverage this situation and be prepared for this new era. Following the same narrative, in this article I’ll look at an example of the Calico eBPF data plane running on AWS, using Terraform to bootstrap our install to AWS and Rancher K3s to deploy the cluster.

A few changes to Calico are needed for Arm compatibility, including updating parts, enabling eBPF and compiling operators for the ARM64 environment:

  • Tigera Operator – The recommended way to install Calico.
  • go build – A container environment packed with all the utilities that Calico requires in its compilation process.
  • Calico-node – The pod that hosts Felix, the brain that carries control plane decisions to your cluster.

Let’s discuss how to run an Arm-powered Kubernetes cluster equipped with Calico for security and eBPF as the data plane.

What is eBPF?

eBPF is a virtual machine embedded within the Linux kernel. It allows small programs to be loaded into the kernel and attached to hooks, which are triggered when some event occurs. This allows the behavior of the kernel to be customized.

Calico’s eBPF data plane makes use of BPF functionality to allow source IP preservation, direct server return (DSR) and even better performance. A full explanation of eBPF and its capabilities is out of the scope of this article, but if you are interested in learning more you can check out this post, which talks about eBPF capabilities and why you should use it.

Performance

Calico eBPF data plane is an alternative to Calico’s standard Linux data plane (based on iptables) that pushes the performance limits further. Calico uses BPF programs to process packets rapidly and efficiently without ever leaving the packet-processing context of the Linux kernel. The efficiency achieved by this method is close to natively compiled code in the kernel.

Note: AWS has a limit on single flows of traffic between EC2 instances that restricts this type of traffic to 5 Gbps.

The chart below shows the total CPU utilization consumed by the benchmarks, measured in vCPUs. This is the sum of both client and server CPU utilization.

These statistics were collected using two A1.xlarge instances, this K8s benchmark suite, and multiple instances of iperf with parallel connections to each other. Note that AWS has a limit on single flows of traffic between EC2 instances that limits this type of traffic to 5 Gbps.

Demo

This section provides you with the steps necessary to deploy a Rancher K3s Arm-based cluster equipped with Calico’s eBPF data plane on AWS public cloud infrastructures.

Before We Begin

This demo section uses A1 instances from Amazon. In addition to an Amazon account, make sure the following applications are installed on your system:

Cluster Preparation

Use the following link to download the Terraform template cluster:

Browse to the calico-k3s-aws folder within demo-cluster:

Inside the calico-k3s-aws folder, you will find an example variable file called terraform.tfvars-example. Removing the example suffix from this file provides an easy way to modify some of the attributes of the cluster that will be deployed using Terraform.

For this example, since our document focuses on setting up an Arm environment, you need to modify the following values accordingly:

Note: At the time of writing, AWS us-west-2 region only offers ARM64 instances in us-west-2a and us-west-2c availability zones.

Terraform uses “providers” to connect to a variety of environments. Use the following command to download the “providers” related to this demo:

Now that our Terraform project is ready, use apply to populate the resources. Two vanilla Ubuntu EC2 VMs, one virtual private cloud (VPC), one internet gateway (IGW), two subnets and one default route will be created after running the following command.

Note: Executing the following command will create resources in your AWS Account. Take a minute to review the resources before approving the deployment.

There should be an output similar to the following after a successful deployment:

K3s Server Installation

K3s server is the process that plays the role of control plane. This installation process creates important files that are crucial in controlling and maintaining the cluster.

Use instance_1_public_ip value and the calico-demo.pem file to connect to instance one.

Note: In Windows, use PuTTY to complete this step. More information can be found here.

K3s’ single binary approach provides a nearly magical experience for installing a cluster. Simply run the following command and the cluster will be ready in a matter of seconds!

Despite its tiny size, K3s is packed with a lot of features. In this demo, we are going to disable Flannel, Traefik, and K3s default network policy to simplify the process. (If you would like to know more about these features, please check out this link.)

Let’s check the state of our cluster using kubectl:

You should see a result similar to the below. Notice that all of the pods are in pending state. This is because there is no container network interface (CNI) installed in our cluster.

Installing Calico

Tigera Operator is a great way to install Calico. Operators are a great way to interact with custom resources in a controlled manner.

Note: Calico ARM64 eBPF support is in its early stages. You’ll need to modify the tigera-operator manifest and add -arm64 to the end of the image: quay.io/tigera/operator:master line. In the below command, I’ve used sed to automatically do the procedure. However, if you are using Windows or having trouble with the following command, consider doing this step manually using a text editor and then applying your modified manifest using kubetl create -f <<myfile.yaml>> .

Use the following command to install Tigera Operator on the cluster:

It is possible to verify operator installation using the following command:

Next, you should see an output similar to the following:

Calico is packed with a lot of features and tunable to its basic components. After installing the operator, it will constantly check for a configuration in the default namespace that contains the kind: Installation header to configure Calico in the cluster.

Use the following command to begin the installation:

Use the following command to verify Calico deployment has finished:

There should be an output similar to the following:

At this point, we have a working node in our K3s cluster equipped with Calico using the standard Linux data plane.

K3s Agent Installation

Adding K3s worker nodes is as easy as installing the server. Simply copy the token:

You should see an output similar to the following:

Use the instance_2_public_ip value and calico-demo.pem that was created earlier by our Terraform project and SSH into the second instance:

Next, use the following command to install the agent and join the worker node to the cluster.

(Note: Please change the IP address, port and token in the following command to suit your environment. You need to specify the internal IP instance_1_private_ip of the first node, not the external IP.)

Back at the server, execute the following command to verify the newly installed worker:

There should be an output similar to the following:

Enable eBPF

By default, Calico is set to use the iptables data plane, which provides a service named kubernetes in the default namespace and proxies it to the API server using kube-proxy pods.

Since Calico can take over the kube-proxy responsibilities in eBPF mode, we can safely remove these pods. However, we have to allow Calico to directly talk to the api-server to prevent any interruption.

Use the following command to determine the API server information:

By using a ConfigMap, we can tell Calico how to directly contact the cluster API server.

Note: Address and port might differ. Make sure these values are typed correctly, since an incorrect value can trigger a crash loop for tigera-operator pod.

Use the following command to create the required ConfigMap:

Use kubectl apply to apply the file and create the ConfigMap:

It might take 60 seconds for the ConfigMap to be picked up by the cluster. After that, use the following command to restart the operator in order to pick up the changes:

K3s embeds the kube-proxy process, which makes it hard to disable. Since both kube-proxy and eBPF are trying to interact with cluster data flow, we must change the Felix configuration parameter BPFKubeProxyIptablesCleanupEnabled to false. If both kube-proxy and BPFKubeProxyIptablesCleanupEnabled are enabled, kube-proxy will write its iptables rules and Felix will try to clean them up, resulting in iptables flapping between the two.

Export the cluster credentials file that was created by the server installation process:

Use the following command to disable iptables cleanup:

Execute the following command to change the cluster data plane with eBPF:

Let’s verify our cluster’s health and node architecture by executing the following command:

That’s it! You now have a multinode K3s cluster secured with Calico that uses the eBPF data plane.

Clean Up

In the demo folder, issue the following command to remove all the resources created for this demo:

Conclusion

K3s’s minimalistic approach makes it invaluable for resource-constrained environments, and its single binary approach allows for faster deployment across any infrastructure. Users can push this resource efficiency even further by using the Calico eBPF data plane, while also benefiting from Calico’s other features such as its feature-rich network policy language, BGP routing and more.

In this article, I explored how to install a multinode K3s cluster equipped with Calico and how to swap the standard Linux data plane with eBPF. I’ve only scratched the surface of Calico’s eBPF capabilities here. For more information, take a look at our eBPF documentation.

If you enjoyed this blog post, you might also like:

Did you know you can become a certified Calico operator? Learn Kubernetes and container networking and security fundamentals using Calico in this free, self-paced certification course.

The New Stack is a wholly owned subsidiary of Insight Partners, an investor in the following companies mentioned in this article: Tigera.

Featured image via Pixabay.