Implement Postgres on Kubernetes with Ondat and SUSE Rancher

We previously explored the challenges of running public cloud DBaaS and benefits of running your own Database as a Service (DBaaS) on Kubernetes using SUSE Rancher and Ondat. Now, we’re going to take the theory and make it practical.

This article will take you step by step through how to implement a PostgreSQL DBaaS using the excellent operator by the folks at Percona. We’ll be deploying this on a Rancher Kubernetes Engine (RKE) cluster on DigitalOcean, but the instructions should work fine for almost every Kubernetes distribution.

Prerequisites

This tutorial was written and tested on a macOS and Linux workstation, so we will assume you are using one or the other. In addition, you’ll need the following utilities installed locally through your preferred package manager:

• kubectl
• git
• Terraform
• Helm

In this tutorial, we will be using DigitalOcean as our cloud provider to provision droplets, using Terraform, that will be used as master and worker nodes for our RKE workload cluster. If you don’t have a DigitalOcean account, you can sign up for one and start your free trial with $200 credit for 60 days.

Lastly, I would also recommend a beverage to go with this tutorial. I’ll be drinking tea, but feel free to substitute it with the drink of your choice.

Once you have met the tutorial prerequisites, are sitting comfortably, and have a nice drink, let’s begin!

Ondat delivers persistent storage volumes directly onto any Kube node. If your data is locked-in to one platform, so is your cluster. With Ondat you are free to put your applications and your data storage where you want. Ensuring industry-leading levels of performance, resilience and dependability.

Learn More

The latest from Ondat

Step 1: Provision a SUSE Rancher Server and RKE Cluster on DigitalOcean

Before we go any further, we need an RKE cluster. We could do this with lots of manual keyboard mashing, but that’s tedious; instead, we will use Terraform to automate this process. We’ve already created the Terraform code you’ll need for this tutorial, so let’s go ahead and clone it with the following commands:

• This directory has Terraform configuration files that will be used to provision the following elements:
  • a single node K3s cluster for the Rancher server,
  • and a highly available RKE cluster with three master nodes and five worker nodes.

• There is also a Terraform variable file for you called terraform.tfvars.example. We want Terraform to use it when we are provisioning our resources, so go ahead and rename it with the following command:

• Next, insert your DigitalOcean personal access token and set a password for Rancher. Open the file with your preferred text editor, and find and set the following values:

• Once you’ve saved the terraform.tfvars file, initialize Terraform with the following command:

• This command downloads the various modules and providers it needs to perform the automation. Next, we will validate our Terraform code and then create a plan. The terraform plan command allows us to preview the changes that Terraform will make to your platform when you apply it. Validate and run a plan against the Terraform code using the following commands:

• This will create a list of all the resources that Terraform will create when you apply it, and it’s important to check you are happy with what it’s about to do. Once you’ve reviewed the plan, use the following command to apply it:

• This might take some time, so this is an excellent point to top up your chosen beverage! Once provisioning is complete, you will see an output similar to the following:

• At this point, you should have a fully working Rancher and RKE cluster. Go ahead and test them by first going to the Rancher UI. To access it, copy and paste the value from your terraform apply, named rancher_server_url. This should open the Rancher login page. To log in, use the username of admin and the password you set for the rancher_server_admin_password in the terraform.tfvars file earlier.

• Once you’ve finished admiring the Rancher UI, you can check that your RKE cluster is also working correctly using kubectl. First, however, you need your kubeconfig file to ensure that kubectl knows which cluster to interact with. You can put your kubeconfig file in the correct place with the following commands:

• Now, we can inspect the nodes and pods running in our RKE cluster using the following command:

• This should bring back a list of master and worker nodes and a few core system pods. You may find that not all your nodes are showing; it can take some time for the nodes to register with the Rancher server, so check back later if you are missing some.

• You can also check the status of the node registrations through the Rancher UI by reviewing the “Cluster Management” tab. Alternatively, periodically execute kubectl get nodes until all eight nodes are in a Ready state.

Step 2: Deploy and Configure Ondat

To install Ondat onto your newly provisioned RKE cluster, log in or create an account on Ondat’s Portal, which will generate the correct installation commands and allow you to register your cluster to get your free Ondat Community Edition license.

• In the Ondat Portal UI Dashboard, select the Install Ondat on your cluster option. You will be redirected to the Cluster tab where under the Cluster Name text box, name your cluster and select the Rancher option for your Kubernetes distribution.

• You will notice that the portal will display prerequisites that will need to be applied before moving onto the next page. Be sure to satisfy the prerequisites, especially the installation of the Local Path Provisioner utility, which is used to provide storage for Ondat’s dedicated etcd cluster inside your RKE cluster. When you have met the prerequisites for deploying Ondat, click the Next button to continue to the next step to get the installation commands.

• The next page will provide you with helmcommands that will add the Ondat Helm chart repository to your local index, update the repository local index and then install Ondat onto your RKE cluster. Go ahead and copy the commands, paste them into your terminal and press Enter to execute them.

• The installation will take a few minutes. When it has finished, you can use the following command to check that Ondat resources are running:

• This will return a list of all the components and resources that Ondat has installed into the cluster.

• Now that you have an Ondat cluster, we want to create an Ondat StorageClass and make the default StorageClass for our cluster. We’re going to use feature labels to enable the following capabilities:

• Replication
• Topology-Aware Placement (TAP)
• Data Encryption At Rest

• First, use the labels to create custom regions. Let’s create them with the following command:

• Next, create a new Ondat StorageClass. Note the parameters section; this is where we’re setting some crucial configurations such as the number of replicas and our topology keys:

• Finally, we’re going to set our new Ondat StorageClass to be the default within our RKE cluster. This ensures that any Kubernetes workload that requests storage gets an Ondat volume by default:

Step 3: Deploy Percona’s Distribution for PostgreSQL Operator

• In Step 1 and Step 2 we created an RKE cluster, then installed and configured Ondat. Now, we’re going to create the service part of our DBaaS using the PostgreSQL Kubernetes operator developed and maintained by Percona. Use the following command to create a namespace and deploy the operator into it:

• Once the operator is running, you can create a new PostgreSQL database with some simple Kubernetes native YAML. You can find some excellent examples in the Percona Operator for PostgreSQL GitHub repository, and we’ve also created an example in the code that’s included with this tutorial. Our example will create a PostgreSQL cluster with PGBouncer and PGBadger.

• Run the following command to deploy the PostgreSQL cluster included with our code:

• You now can create PostgreSQL databases at will or give developers the ability to include a PostgreSQL database with applications deployed into a Kubernetes cluster with this operator.

Step 4: Exploring Ondat’s Replication and Data Encryption Features

• We now have a DBaaS, but we need to ensure that best practices around reliability and security are in place. Ondat handles this for you, and in this section, we’re going to simulate some failures to see it in action, particularly the fast replication capability.
  • • To make interacting with the Ondat cluster easier, start by deploying the Ondat CLI into the RKE cluster. Use the following command to deploy it:

• • • To use the CLI we will need to find the CLI pod’s name. Use the following command to get it:

• • • We’re going to use the CLI a fair amount in this section, so copy the pod name somewhere you can get to easily.

Ondat Durable Replication

When you created your database back in Step 3, it requested storage from the default StorageClass in your cluster — which, thanks to the steps we carried out in Step 2, defaulted to the custom ondat-replication-encryption StorageClass. Therefore, if you used the default configuration in the example, Ondat has automatically created two replica volumes for each volume requested by the database operator.

• • • You can review the volumes created by running the following commands:

• • • You should get output similar to the following:

• • • Copy the name of the first volume; in our example, it is pvc-2115e4dc-7c06-4ab6-8d63-7e83b13bfa3b. Now, examine the volume using the Ondat CLI using the following command:

• • • This will give you a great deal of information about the volume, including the amount and location of the replica volumes. The output will look like this:

• • • This output tells you that each volume replica is deployed on a different node to ensure data protection and high availability if a node experiences a transient failure. This demonstrates how Ondat ensures that data is replicated to avoid any single point of failure. So, now our data is nicely replicated, and our database is working well, let’s try to break it. Failures are rarely subtle, so let’s go big and delete the Kubernetes node where our master volume resides.
  • • Using the output of our volume description, we can see which node the master volume is located on; in our case,rke-ondat-demo-worker-node-4. Now we’ve found our master volume, we will terminate the node on it with extreme prejudice using the following command:

• • • When rke-ondat-demo-worker-node-4 goes offline, Ondat will automatically detect that the master volume doesn’t exist anymore and elect one of the two replica volumes to become the new master volume. It will also create a new replica on a different node to keep the defined replica volume count specified in the ondat-replication-encryption config of the StorageClass created.
  • • Run the following command to see how Ondat has managed the failure:

• • • You should see output similar to the following:

• • • We have the same number of volumes, with the exact same names, but the main difference is that you will see the location of pvc-2115e4dc-7c06-4ab6-8d63-7e83b13bfa3b is now based on a different node. To investigate further, let’s describe the same volume from earlier using the Ondat CLI:

• • • We will see the following in the output:

• • • You can see that Ondat automatically elected the replica volume on node rke-ondat-demo-worker-node-3 to become the master volume since node rke-ondat-demo-worker-node-4 no longer exists in our cluster, and a new replica volume was created on node rke-ondat-demo-worker-node-1 to ensure that the replica volume count defined is consistent.

Ondat’s Encryption At Rest

Alongside ensuring out-of-the-box encryption of data in transit using Mutual TLS (mTLS) authentication, Ondat provides the capability of being able to encrypt data at rest for volumes. This is essential for any self-managed DBaaS as it ensures that data is secured without needing to trust that the applications themselves are encrypting data held in the database.

• • • Encryption at rest can be enabled by default for all volumes by adding a feature label called storageos.com/encryption=true to your Ondat StorageClass parameters, or on a per-volume basis using the PersistentVolumeClaim manifest for your application. This offers the cluster operator the flexibility of having default data encryption at rest or leaving it to applications to decide, depending on the needs of the platform.
  • • To verify that data encryption at rest is working as expected, we will attempt to access some data on an encrypted volume to check it is encrypted. First, we’re going to find which node one of our Ondat encrypted volumes, named cluster1-repl1, is located using the following command:

• • • This gives us the following, indicating the volume is on the node:

• • • Next, run a privileged container on that node with the strings utility to give us the ability to try and read data from a volume that is located on that node. Use the following command to run the pod and install the utilities:

• • • Now, we can use this pod to try and read data from the underlying node. Since it is privileged, it should have full permission to read data from the node, normally, a surefire way to read data from pods running on that node.
  • • Run the following commands to attempt accessing the cluster1-repl1 deployment data that is stored as blob files on a node under the /var/lib/storageos/data/ directory:

As demonstrated in the test above, you can see that despite having privileged access to the underlying node, we are unable to read any data from an Ondat volume as it is encrypted. From a black hat or white hat perspective, attackers will be unable to read the data even if they have gained access to nodes running the DBaaS we’ve created. Thanks to Ondat’s native data encryption at rest, they are unable to decrypt the volume data without the encryption keys.

Conclusion and Tidying Up

This brings us to the end of our tutorial on creating a DBaaS with RKE and Ondat.

Before you leave us to top up your tea, coffee or other delicious drink, follow these instructions to tear down the resources used in this tutorial. DigitalOcean is keenly priced, but if you have finished, then it’s best not to run up charges by leaving your cluster running. Use the following commands in the current directory of the Terraform configuration files to remove the resources created during this tutorial:

Please feel free to get in touch to learn more about Ondat. We look forward to releasing future tutorials outlining how Ondat can benefit your Kubernetes strategy.

Rodney is a solutions architect at Ondat, focused on helping customers run stateful applications at scale in Kubernetes. Rodney has background experience with Linux, container security, Kubernetes and working with enterprise customers to solve their cloud native challenges.

Read more from Rodney Karemba