KubeCon: Cloud Native Patterns of the GitOps Pipeline
Honeycomb is sponsoring The New Stack’s coverage of Kubecon+CloudNativeCon North America 2020.
One of the things that makes us human is pattern recognition. From a very early age, patterns help us make predictions, understand relationships and process complexity.
Part of Cornelia Davis’s 30-year career as a technical strategist has always been highlighting patterns in complex, distributed systems. After all, she wrote the book on it. As CTO at Weaveworks, she works with open source and commercial partners to leverage these cloud native patterns.
At KubeCon North America, Davis uncovered the new patterns of GitOps — repeatability, recovery from failure, and reduction of toil — and provided viewers with the practices, patterns and tools needed to take advantage of them. In a technical presentation at this year’s KubeCon + CloudNativeCon North America, she recommended you string together those patterns into a secure GitOps pipeline that leverages what she calls the Kubernetes platform of reconciliation loops.
How Do You GitOps?
In this talk which looks at cloud native from an operational perspective, Davis defines GitOps as “a set of modern best practices for deploying and managing cloud native infrastructure and applications.”
She lists the benefits of GitOps as including:
- Compliance and security with an audit trail
- Repeatability
- Recovery from failure
- Reliability
- Increased productivity with more deployments and faster feedback loop
- Reducing toil
- Multicloud and on-premise
- One platform to rule them all
“And the way that we do GitOps is going to have a direct impact on the benefits that we get to enjoy,” Davis noted.
GitOps is a portmanteau for git — the distributed code repository that can act as the single source of truth — and operations. Davis said it’s not just practices and patterns, but encompasses the tools that implement them. And, like most tech meets culture, she importantly notes that GitOps has the primary user interface on the left Git side of the whole process.
Davis explains that the true benefit of GitOps comes from proper organization and especially how you plan the automation at the center of the design.
It’s CI. And CD. Not CI/CD. And It’s Pull Not Push.
Davis says the first place you may look to enhance automation is in the most proven space — continuous integration. After all, CI has had successful automation for a long time. Why not put deployment in there? Davis says not so fast.
She contends, “Continuous integration is not the same as continuous delivery and deployment. There are reasons to separate those out.”
Firstly, Davis says there’s a need to maintain a Separation of Concerns — developers write and release code, but operators must maintain control of the rigorous processes needed to deploy securely to the production environment. She also points to an inherent need to decouple CI and CD so operations can easily deploy to different environments. Also, if there’s been a failure, recreating a deployment should be able to happen without a new build.
“There’s no CI/CD. It’s not one word. There’s CI and there’s CD, but there’s absolute benefit in separating those,” Davis said.
Crucially, if you have your CD inside your CI process, you are making that an attack service, where someone who has gotten into the CI environment can then get into the runtime environment.
This is why she says, to get the most out of your GitOps, you must draw the deployment back out of the continuous integration loop.
However, once that’s separated, you are met with a challenge of networking connectivity at scale. You could be continuously deploying to one point, a few points, — like developer, staging and production — many — like edge deployments across factories coffee shops — or tens of thousands of mobile towers.
Davis says the simple solution to this is to flip the continuous deployment process from being in the centralized environments to being in those dozens or thousands of runtime environments. Then, instead of pushing configuration out to those runtime environments, you’re pulling the configuration out into those runtime depositions.
The Power of Reconciliation Loops to Enable Other GitOps Patterns
How do you need to know when to pull? You don’t have to. You can instead run reconciliation loops inside your Kubernetes clusters that can constantly draw the confiscation in from those repositories.
“Kubernetes created this reconciliation platform,” Davis acknowledged.
The desired state is in the repository, and the reconciler is watching the actual state of the system.
Davis says this one powerful reconciliation pattern allows for other GitOps — or cloud native operational patterns — to emerge.
The next pattern she has identified is: drift, detection, and remediation. Git, importantly, has a version history. If you were to have some type of catastrophic event and wanted to recreate your environment so long as there hasn’t been any drift, you could do that easily. But Davis asks: What happens if someone applies Kubectl — what she refers to the modern version of SSHing — and your git drifts us from the desired state? Since that reconciler is constantly running, it can perform remediation, like undoing that change to go back to the state that was represented in the repository or maybe you don’t want to go that far, but rather just ping someone.
Next Davis presents the image update automation pattern. Any update to your image repository can be done automatically in the YAML code itself, leveraging another reconciler to automatically update it by pushing directly or by generating a pull request, which someone can approve or can be automatically approved. The reconciler then does the deployment of the updated version of that image.
A GitOps pipeline also allows for environment customizations via kustomize, which allows for customization of YAML files. Kustomize is used when there are differences across the various deployment endpoints, applying both a basic configuration and then overlays that override that basis. Again, the reconcilers can allow for those overrides and can allow these different runtime environments to be treated differently.
Davis pointed out, “There’s a whole bunch of detail that’s required in this process of taking these application configurations and these images from the image registries and composing them together into the actual declarative state that’s going to be running in my runtime environment. I like to call that set of controllers, that set of reconcilers, delivery controllers.”
GitOps Patterns for Progressive Delivery
At this point in Davis’s tour through a stable GitOps pipeline, she says we’ve reached the point where we are drawing configurations into Kubernetes, but not quite deploying running pods yet. This will, of course, be done via other Kubernetes reconcilers or what Davis refers to as “runtime controllers”, like for replica sets and daemon sets.
“The real happens when we draw together all of these different reconcilers, all of these different controllers, across the entire spectrum.” — Cornelia Davis, Weaveworks
James Governor coined the umbrella term progressive delivery for routing traffic to a specific subset of users before being deployed more broadly. Progressive delivery includes A/B testing, blue-green deployments and canary releases, service meshes, chaos engineering, and observability.
Davis says you can apply reconciliation loops to facilitate different progressive delivery deployment scenarios. Kubernetes allows you to extend the API so you can create a reconciler that recognizes when a deployment is happening and add some additional logic on top of it.
Weaveworks has open sourced Flagger as a way to define a deployment strategy in your GitOps pipeline. With Flagger, implemented as a set of controllers within Kubernetes, you select if you want to run a canary, A/B test or blue-green deployment, and then it interfaces with either Ingress or service mesh to provide the traffic routing. Flagger can then run runtime analysis, promotion and rollback for these strategies.
Six Cloud Native Operational Patterns Driving Your GitOps Pipeline
For Davis, GitOps is best defined as continuous delivery plus continuous operations served atop a declaratively configured Kubernetes foundation, always with the “immutability firewall” of your git.
You’re storing code in a declarative configuration in git. Then you have a set of delivery controllers that facilitate your pipeline. And then you have runtime controllers that get that into production. This is all with the user experience on the left-hand side. With this, git plays the role of the UX and the distributed immutable data store that can be rolled back or forward.
“You take those controllers and you stitch them together into the pipelines that are relevant for your organization and for your needs, and that’s what draws together GitOps,” Davis said.
Summary of Cloud Native Operational Patterns
- CD separate from CI
- Pull configuration
- Drift detection and remediation
- Image Update Automation PIC
- Environment customizations
- Progressive Delivery
“That automation in the middle has to be cloud native. Cloud native operations is what GitOps is all about,” Davis said.
KubeCon+CloudNativeCon is a sponsor of The New Stack.
