Tutorial: Backup and Restore of Stateful Workloads with Portworx
In the last part of this series, we saw how to configure a highly available WordPress deployment with Portworx, a data and storage management platform for Kubernetes. In the current installment, we will learn how to implement the backup and restore of PVCs through local and remote snapshots feature of Portworx.
Snapshots in Portworx
Portworx can take point-in-time snapshots of a PVC which can be stored locally within the cluster or remotely in an object storage service compatible with Amazon S3 API. The remote snapshots called CloudSnaps promise higher availability due to the fact that they are stored outside of the cluster.
When running distributed databases as statefulsets in Kubernetes, taking snapshots of individual PVCs associated with each pod of the statefulset may not result in an application-consistent backup. To enable backing up a group of PVCs that logically belong to an application, Portworx has a concept called 3DSnaps.
When using 3DSnaps, Portworx allows specifying pre and post rules that are run on the application pods using the volumes. This allows users to quiesce the applications before the snapshot is taken and resume I/O after the snapshot is taken. The commands will be run within pods with attached PVC that are being snapshotted.
Whether it is a snapshot of a single PVC or a group of PVCs taken through a 3DSnap, the backup process can be either on-demand or scheduled.
Let’s see how to create a CloudSnap to backup MySQL and WordPress PVCs to a remote site. For simplicity, we will use MinIO running in a machine outside of the cluster for storing the snapshots. Since it is 100% compatible with Amazon S3 API, Portworx can be pointed to MinIO to store CloudSnaps.
Running MinIO on Docker
On a Linux or macOS machine running Docker, create a directory to mount the volume for MinIO.
|
1 |
sudo mkdir -p /mnt/data |
Run the below command to launch MinIO container with a volume pointing to the above directory.
|
1 2 3 4 5 |
docker run -p 9000:9000 --name minio1 \ -e "MINIO_ACCESS_KEY=AKIAIOSFODNN7EXAMPLE" \ -e "MINIO_SECRET_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" \ -v /mnt/data:/data \ minio/minio server /data |
We are using the example access key and secret key as mentioned in MinIO’s documentation. For production usage, replace them with your own keys.
Verify by accessing the MinIO UI available at port 9000.
Sign in with the access key and secret key to access MinIO browser.
Registering MinIO with Portworx
Before we can initiate the backup process, we need to tell Portworx where to store the snapshots. This is done by creating a credential within Portworx with the access key and secret key of MinIO.
Let’s invoke the CLI of Portworx, pxctl, running inside one of the pods of the daemonset within the kube-system namespace, and pass the parameters required to create the credentials.
|
1 2 3 4 5 6 7 |
PX_POD=$(kubectl get pods -l name=portworx -n kube-system -o jsonpath='{.items[0].metadata.name}') kubectl exec -it $PX_POD -n kube-system -- \ /opt/pwx/bin/pxctl credentials create \ --provider s3 \ --s3-access-key AKIAIOSFODNN7EXAMPLE \ --s3-secret-key wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY \ --s3-region minio --s3-endpoint http://10.0.0.5:9000 minio_creds |
Verify the credentials by listing them with the below command:
|
1 2 |
kubectl exec -it $PX_POD -n kube-system -- \ /opt/pwx/bin/pxctl credentials list |
With the object store credentials in place, we are ready to configure CloudSnaps.
Configuring Portworx CloudSnaps for MySQL and WordPress
If you recall, the application deployment resulted in two PVCs associated with MySQL and WordPress. Our goal is to backup these PVCs through CloudSnaps.
The below YAML specification creates a CloudSnap for PVC, db-pvc used by MySQL pod.
|
1 2 3 4 5 6 7 8 9 |
apiVersion: volumesnapshot.external-storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: db-snapshot namespace: wordpress annotations: portworx/snapshot-type: cloud spec: persistentVolumeClaimName: db-pvc |
|
1 |
kubectl create -f db-snap.yaml |
Similarly, let’s create a snapshot for WordPress PVC.
|
1 2 3 4 5 6 7 8 9 |
apiVersion: volumesnapshot.external-storage.k8s.io/v1 kind: VolumeSnapshot metadata: name: wp-snapshot namespace: wordpress annotations: portworx/snapshot-type: cloud spec: persistentVolumeClaimName: wp-pvc |
|
1 |
kubectl create -f wp-snap.yaml |
List and verify the snapshots with the below commands:
|
1 |
kubectl get volumesnapshots |
|
1 |
kubectl get volumesnapshotdatas |
|
1 |
kubectl describe volumesnapshotdatas k8s-volume-snapshot-76ec8970-822f-11ea-aef1-26c54787ef6a |
|
1 |
kubectl describe volumesnapshotdatas k8s-volume-snapshot-b75b3cac-822f-11ea-aef1-26c54787ef6a |
The above output confirms that the snapshots have been successfully created.
Refer to Portworx documentation on how to schedule the snapshots to enable periodic backups.
Finally, if you access MinIO browser, you notice that there are two folders created in a bucket. These folders are storing the snapshots from the two PVCs.
Restoring from CloudSnaps
Let’s create new PVCs from the snapshots stored in MinIO.
The below YAML specification results in a new MySQL PVC which is a clone of the original PVC.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: db-pvc-clone annotations: snapshot.alpha.kubernetes.io/snapshot: db-snapshot spec: accessModes: - ReadWriteOnce storageClassName: stork-snapshot-sc resources: requests: storage: 2Gi |
Notice the storage class name, stork-snapshot-sc, which is specific to Portworx’s Storage Orchestrator for Kubernetes (STORK) that manages the orchestration and operations of stateful workloads.
Let’s also create a PVC clone for WordPress.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
apiVersion: v1 kind: PersistentVolumeClaim metadata: name: wp-pvc-clone annotations: snapshot.alpha.kubernetes.io/snapshot: wp-snapshot spec: accessModes: - ReadWriteOnce storageClassName: stork-snapshot-sc resources: requests: storage: 2Gi |
When we apply these two specs, a couple of new PVCs are created.
The PVCs ending with “*-clone” with storage class stork-snapshot-sc are new PVCs restored from the snapshots.
You can now create WordPress and MySQL pods using the cloned PVCs of the original pods. This technique can be used to migrate PVCs from one namespace to another or even other clusters running Portworx.
In the next part of this series, we will explore how to migrate the entire workload from one cluster to another using Portworx and Velero. Stay tuned!
Janakiram MSV’s Webinar series, “Machine Intelligence and Modern Infrastructure (MI2)” offers informative and insightful sessions covering cutting-edge technologies. Sign up for the upcoming MI2 webinar at http://mi2.live.
Portworx is a sponsor of The New Stack.
Feature image by Paul Brennan from Pixabay.