Progressive Delivery on OpenShift
We are accustomed to having high expectations of our apps. We want a constant stream of new features and bug fixes, and yet, we don’t want these updates to affect our user experience adversely. As a result, these expectations put a tremendous amount of pressure on developers.
This is where Iter8 comes in; it helps developers progressively roll out new versions of their app, gated by a combination of performance, functionality and even custom business metrics. If a new version fails to meet all the criteria, it is automatically rolled back without users ever having to see it.
The term “progressive delivery“ is used to describe more advanced app deployment patterns — canary, blue/green, dark launch, A/B — that are designed to achieve goals such as minimizing downtime, reducing risks, and/or maximizing certain business metrics. Iter8 supports these advanced deployment patterns and can run on both vanilla Kubernetes and its enterprise version, OpenShift.
In this article, we will describe how Iter8 can be integrated into an existing DevOps pipeline running on OpenShift. If you already have an app deployed and managed by a DevOps pipeline, enabling Iter8 takes only a few minutes, and disabling it is also easy.
To cater to developers who are new to GitOps and/or OpenShift, in the first part of this article, we will walk through a scenario to set up a demo app that runs on OpenShift. We will also adhere to the GitOps practice, where manifests are maintained in a repo and changes are synced to the cluster continuously. Once this demo app is fully set up, the second part of the article describes how to integrate Iter8 into the pipeline we just deployed to enable progressive delivery. If you are already familiar with Openshift/DevOps, you can skip straight to the second part.
Part 1: GitOps on OpenShift
Set Up an OpenShift Cluster
If you don’t already have an OpenShift cluster, the easiest way to kick the tires is by running a cluster locally on your laptop via CodeReady Container. If you’re familiar with Minikube used for provisioning a local Kubernetes cluster, this is similar: an all-in-one single-node OpenShift cluster running in a VM.
To install and run CodeReady Container, follow the instructions here. After the cluster is up and running, you should be able to login to the cluster as kubeadmin, e.g.:
|
1 |
oc login -u kubeadmin -p your_password https://api.crc.testing:6443 |
Note that there should already be clear instructions on how to obtain the kubeadmin password on the command line, but, alternatively, you can also obtain the password with:
|
1 |
crc console --credentials -o json | jq .clusterConfig.adminCredentials.password |
Fork Iter8 Repo (To Install the Demo App)
Go to github.com/iter8-tools/iter8 and fork the repo, then set the following env var:
|
1 |
export YOUR_ORG=[your github org or username] |
Then run the following commands to update a few references so they point to your own fork.
|
1 2 3 4 5 |
git clone github.com/$YOUR_ORG/iter8 cd iter8 && export ITER8=`pwd` find samples/openshift/gitops -name "*" -type f | xargs sed -i '' "s/MY_ORG/$YOUR_ORG/" git commit -a -m "update references" git push origin head |
Install ArgoCD / OpenShift-GitOps
We use ArgoCD for GitOps so that manifest changes in a Github repo can be immediately reflected in a running cluster. To install it, follow the instructions here. It should be as simple as copying and pasting the following commands:
|
1 2 |
oc create namespace argocd --dry-run -o yaml | oc apply -f - oc apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml |
As an alternative, you can install OpenShift GitOps, which installs ArgoCD under the cover, directly from the admin console. This makes installing ArgoCD even easier, and you can follow the instructions here.
Deploy the Demo App
We use a bookstore app to demonstrate how its components can be progressively rolled out in a safe manner. It can be started with the following commands:
|
1 |
oc apply -f $ITER8/samples/openshift/gitops/argocd-app.yaml |
ArgoCD should detect this action shortly after and deploy the app to our cluster. Afterward, you can open another terminal to start the port forwarding:
|
1 |
oc port-forward svc/productpage -n default 9080:9080 |
Now, you can open a web browser and try out the app at http://localhost:9080/productpage?u=normal. You should see something similar to:
At this time, we have a working app that adheres to the GitOps practice. Updates to the app are done via changing the app manifests in the repo, then they will be synced to the cluster by ArgoCD.
Part 2: Enabling Iter8 for Progressive Delivery
Create a Github Token for Iter8
If you’d like Iter8 to automatically promote new versions of your app when they pass certain success criteria, you’d need to provide it with a Github token for creating pull requests (PRs). The token can be generated by going to github.com > Settings > Developer settings > Personal access token > Generate new token. This token can be added to the cluster by:
|
1 2 |
oc create secret generic github-token --from-literal=token=[YOUR_TOKEN] oc apply -f https://raw.githubusercontent.com/iter8-tools/iter8/master/samples/tasks/rbac/read-secrets.yaml |
Install Iter8
Now let’s install Iter8. You will need to install Kustomize by following the instructions here. Run the following commands to install Iter8:
|
1 2 3 4 5 |
export TAG=master kustomize build "https://github.com/iter8-tools/iter8/install/core/?ref=${TAG}" | oc apply -f - oc wait crd -l creator=iter8 --for condition=established --timeout=120s kustomize build "https://github.com/iter8-tools/iter8/install/builtin-metrics/?ref=${TAG}" | oc apply -f - oc wait --for=condition=Ready pods --all -n iter8-system |
Upgrade the Demo App
Everything is now in place for us to upgrade the demo app and see how Iter8 can help developers test a new version before promoting it to production. To make it easier to follow along, we created a Github Action. You can go to github.com/YOUR_ORG/iter8/actions, and click on “Simulate CI pipeline on OpenShift” and then “Run workflow.”
Under the cover, this will create an Iter8 Experiment custom resource and new demo app Deployment. If you run oc get experiments and oc get deployments, you will see these resources getting deployed by ArgoCD in the cluster. As Iter8 by default will use a synthetic workload to exercise new versions, it will take a bit of additional time for it to collect enough data to make a decision about whether the new version is good, and you can run the following command to monitor Iter8 experiment’s progress:
|
1 |
oc get experiments.iter8.tools --watch |
Promote the New Version
Once the experiment is declared successful, you can go to github.com/YOUR_ORG/iter8/pulls, and there should be a new PR created automatically by Iter8 to promote the new version, i.e.:
Click on “Merge pull request” and wait a bit for ArgoCD to pick up this change, and now you can use a web browser to access the app again. This time, you should see something similar to:
The subtle change in the new version is that the words “William Shakespeare’s” are highlighted in a different color than our initial version.
Part 3: What’s Next
If you would like to know more about how Iter8 can help you manage your application updates, you can access the most up-to-date version of Iter8 and its documentation at https://iter8.tools. Code and other resources are available at https://github.com/iter8-tools. We can always be reached on Slack and also have a bi-weekly community meeting. See this link for more information. Feel free to join us and ask questions!