The Box Way to Microservices: Kubernetes, DevOps, GitOps and the Desired State
Cloud storage service Box is one of the older legends of startups that have made it. Now a publicly-traded company, Box is an organization that once had its own antiquated infrastructure and processes that hampered all attempts to build out a microservices architecture. Today, Box is a company that has embraced the concepts of GitOps, the git-based way to develop workflows and processes that give Box control over the “desired state,” of its cluster running Kubernetes.
Combining Kubernetes, principles rooted in DevOps principles and a pull-based open source project it developed, Box has developed a process to scale its microservices with a stronger security posture and more rapid development with better control over the management of application architectures.
Through the transition, Box now develops microservices in days compared to what once took months, said Box Senior Engineering Manager Kunal Parmar in a keynote at JFrog’s swampUp event in San Francisco, last month.
The tight coupling of its monolith architectures made every code change a task to define how the code was being used across the overall architecture, Parmar said. There were no API boundaries.
Engineers would make changes and start pushing code, Parmar said in an interview after the keynote. They would see failures that they would not catch in testing. To recover, there would be a rollback of the release and a loss of any new functionality.
“We were running most of our software in our own data centers, using Puppet for configuring servers,” Parmar said. “Over time, the configurations had evolved into a complex mess.”
Box engineers started finding themselves in this space where configurations started drifting, affected by years of building up a monolithic architecture with mutable changes. A change, for example, such as a new installation of Java, can result in cruft, which affects updates. A change would be rolled out and then have to be rolled back. It may not even be rolled back correctly. Compounded over a year and the problem becomes something that has to be taken into consideration when building out microservices.
“It created problems for us to create microservices at the pace we wanted to,” Parmar said.
In June of 2014, the Box team went to DockerCon and learned about Kubernetes, just six months after its inception, Parmar said. The way Google engineers were building Kubernetes in approaching the problem of microservices architectures jibed with the way the Box team was approaching the problem, too. They shared in the belief about immutability — that configurations could be abstracted due to the immutability of the container. The physical server becomes a commodity, allowing the containers to run by Kubernetes across multiple cloud providers.
By late 2015, the Box team started to believe that they could use Kubernetes in production. In particular, the community had developed ways to manage the desired state of the IT system. And in part that came with its adoption of GitOps as a workflow
GitOps
The experiences that come with managing monoliths for microservices led the Box team to think through how a desired state of their system could be affected by developers submitting code. They wanted the developers to have a clear knowledge of how a change would affect the desired configuration.
GitOps is a workflow that requires all changes to be done through git. Once a commit is made, it sets off an automated pipeline, to containerize and test the code, then push into production. With the Box approach, developers see the impact with the change they are making. For example, it may allow the developer to see how a change to a library might affect other services.
GitOps also offers more clear control and usability. The version control in git means the cluster can be viewed at any point in time, allowing users to revert back to a prior version if mitigation is needed.
Understanding how to achieve the desired state through GitOps led to the development of an open-source project called kube-applier, that, according to Box, provides automated deployment and declarative configuration for a Kubernetes cluster.
Box uses Jenkins for its CI/CD workflow. Originally, Jenkins pushed the changes to the clusters. The company has since changed to a pull model, an approach that provides tighter controls — the security surface is managed within the Kubernetes cluster.
The Individual Experience
Box now runs a continuous development process on Kubernetes. They did it through a combination of DevOps practices, GitOps and the quest to reach the desired state. But to achieve real succcess, Parmar said there is still a ways to go.
The company has the core tools to help them. JFrog Artificactory serves as a registry for storing binaries, as does Jenkins for continuous integration. GitOps practices serve as the tie-in to allow for rollbacks and better tracking of changes to configurations. The goal: make the experience relatively simple for the developer.
The challenge comes with the integrations of developer tools with Kubernetes and its updates. Service discovery, load balancing, monitoring, secrets management — these capabilities and more already exist for Kubernetes. As the Kubernetes ecosystem grows, so must the overall ecosystem.
