Cloud Native / Data / Kubernetes

Tutorial: Set up a Secure and Highly Available etcd Cluster

14 Aug 2020 7:55am, by

In this second part of the tutorial series on Rancher Labs’ K3s project, we will explore how to configure a three-node etcd cluster that will be used by a highly available, multinode K3s cluster.

The etcd project is one of the most popular open source projects in the cloud native ecosystem. It’s a Cloud Native Computing Foundation-incubating project that has already been a core building block of Kubernetes infrastructure.

By the end of this tutorial, you will have a TLS enabled, three-node etcd cluster deployed which acts as an external datastore of a highly available K3s cluster with multiple masters.

Make sure you have three Linux hosts with static IP addresses. In my lab, I am running four Intel NUC mini PCs running Ubuntu 18.04 with IP addresses ranging from 10.0.0.60 to 10.0.0.63. We will install etcd on the hosts with IP addresses 10.0.0.60, 10.0.0.61, and 10.0.0.62. You will have to replace these IP addresses with your own set of addresses.

Downloading etcd Binaries

On each Linux host, run the below commands to download and install the latest version of the binaries:

Generating and Distributing the Certificates

We will use cfssl tool from Cloudflare to generate the certificates and keys. If you using Mac as your workstation, you can install it using Homebrew.

Make a directory called certs and run the below commands to generate the CA certificate and server certificate and key combinations for each host.

First, let’s create the CA certificate that’s going to be used by all the etcd servers and the clients.

This results in three files – ca-key.pem, ca.pem, and ca.csr

Next, we will generate the certificate and key for the first node.

Repeat the steps for the next two nodes.

Don’t forget to replace the IP addresses and node names with your own combination.

At this point, we have the certificates and keys generated for the CA and all the three nodes.

It’s time to distribute these certificates to each node of the cluster.

Run the below command to copy the certificates to the respective nodes by replacing the username and the IP address.

SSH into each node and run the below commands to move the certificates into an appropriate directory.

We are done with the generation and distribution of certificates on each node. In the next step, we will create the configuration file and the Systemd unit file per each node.

Configuring and Starting the etcd Cluster

On node 1, create a file called etcd.conf in /etc/etcd directory with the below contents:

For node 2, use the below content.

Finally, create the configuration file for the last node.

Please don’t forget to replace the IP addresses specific to your network.

With the configuration in place, it’s time for us to create the systemd unit file on each node.

Create the file, etcd.service at /lib/system/systemd with the below content:

Since the configuration per node is moved into the dedicated file (/etc/etcd/etcd.conf), the unit file remains the same for all the nodes.

We are now ready to start the service. Run the below command on each node to start the etcd cluster.

Make sure that the etcd service is up and running with no errors.

Testing and Validating the Cluster

SSH into one of the nodes to connect to the cluster through the etcdctl CLI.

We inserted a key into the etcd database. Let’s see if we can retrieve it.

Next, let’s use the API endpoint to check the health of the cluster.

Finally, let’s ensure that all the nodes are participating in the cluster.

Congratulations! You now have a secure, distributed, highly available etcd cluster that’s ready for a production-grade K3s cluster environment.

In the next part of this tutorial, I will walk you through the steps of installing and configuring a 4-node K3s cluster with a highly available control plane. Stay tuned!

Janakiram MSV’s Webinar series, “Machine Intelligence and Modern Infrastructure (MI2)” offers informative and insightful sessions covering cutting-edge technologies. Sign up for the upcoming MI2 webinar at http://mi2.live.

The Cloud Native Computing Foundation is a sponsor of The New Stack.

Feature image by analogicus from Pixabay.

A newsletter digest of the week’s most important stories & analyses.