After the Pipelines Run Dry: Thriving in a Post-Jenkins World
It’s a platitude to say that software is always changing. We all know technology evolves quickly. However, as I explained earlier, the pace of innovation in certain corners of the software world since the time Wercker was founded in 2011 has been truly remarkable, even when measured by typical standards.
Six or seven years ago, few organizations were using Linux containers — and if they were, their platform was probably LXC, since Docker did not yet exist. Highly distributed applications were also uncommon. Service-oriented architectures (SOA) had made the idea of distributed services popular, but no one was doing true microservices or breaking their apps down into as many discrete services as possible. And while basic CI tools like Jenkins and Bamboo were in use circa 2011, the idea of a complete CI/CD delivery chain was still foreign.
While many of these innovations have enabled new and unforeseen possibilities over the past few years, as I have also argued, they have yet to achieve their full potential. This is largely the result of limitations involving culture and the way the tools are used. For example, while Jenkins is an excellent tool, its complicated jungle of plugins, poor support for microservices, and inability to provide complete CI/CD have left organizations struggling to leverage what functionality they do provide.
I’ve already said plenty about the problems plaguing the modern world of software development and delivery. But I’m capable of more than critique. In this article, I’d like to switch gears by suggesting how these problems may be overcome, and how organizations can take full advantage of the technologies now available to them.
Moving from Monoliths to Microservices
In 2011, SOA was a mature concept, but microservices as a technology was just being suggested. The benefits of microservices are well known. A microservices-based application can scale more easily. Updates are simpler and less disruptive. Monitoring is more precise because admins can focus on specific services.
At Wercker, we began moving the applications we run to microservices in 2015. I suppose that means we weren’t at the forefront of the microservices revolution. I’ll also admit that our microservices migration is only 30 percent complete.
When you migrate a monolithic application to a containerized environment, you have to either refactor or rebuild it for microservices. Those processes take time. Containerization alone doesn’t do the job.
The upside is, we are getting better and better at executing migrations. We expect to spend less time migrating the next 30 percent of our monoliths to microservices will as we did the first 30 percent.
Converting a monolith to run as microservices is hard, but it’s worth the effort. And you’ll get better at performing this conversion as your organization gains more experience with it. If you struggle at first, persevere: The first mile of the microservices journey is by far the most difficult.
Moving from LXC to Docker
Although we were not early adopters of containers, we at Wercker were early adopters of containers. Well before Docker’s debut, we were using LXC to create a platform where you could run an application stack of your choosing. We thought (and still believe) that it was really cool stuff.
The problem we ran into, however, was that LXC was relatively primitive at the time, and maintaining the framework we needed to enable LXC-based containerization began to be burdensome. We opened up our platform and received assistance from the community, which helped. But that still left a lot for us to maintain ourselves. Remember, we were doing all of this before the open source community at large were interested in running portable containers on platforms.
The burden for us was that we weren’t in the business of building a containerization platform. We were creating a platform for building and releasing applications, which happened to be based on containers. We didn’t want to be responsible for the platform’s underlying container technology.
Then, all of a sudden, along came Docker, with precisely the container technology that we needed. Docker did a better job than we could because that containerization was its primary focus.
Achieving full CI/CD is not something you can do overnight. It’s a process that requires adaptations whenever you adopt new practices, like adopting microservices.
Docker provided Wercker with a more mature foundation, relieving us of the responsibility to maintain our LXC framework. It helped that lots of other developers became interested in Docker at the same time and that Kubernetes — another essential tool in our technology migration journey — was released with Docker containers in mind.
Wercker now runs fully on Docker. We still like LXC, and appreciate that LXC helped make Docker possible. But we have found Docker to be a better fit for our needs.
Adopting Kubernetes
Since we’ve used containers from the start, container orchestration has always been important to us.
Originally, we used Fleet from CoreOS to provide orchestration for our services. Fleet worked well enough at first, but as we became more and more invested in containers and microservices, we became less confident that Fleet would continue to meet our needs. (As it happens, Fleet is now set to be decommissioned and will no longer be supported by 2018, so we think that moving away from it was certainly the right choice.)
So in 2015, as we began our microservices migration, we started searching for a new orchestration solution.
First, we considered Mesos. We thought the Mesos-based Data Center Operating System (DC/OS) was an impressive technology, but we concluded that Mesosphere involved more moving parts than we needed, and was overkill for our purposes.
Instead, we chose Kubernetes. It stood out for us for being straightforward to use, with a single platform and a simple interface. We also liked that Kubernetes is very “opinionated,” in the sense that it is designed for specific purposes, and to be used in conjunction with specific technologies. That made it easier for us to integrate our various existing technologies. It was obvious to us which technologies Kubernetes should be paired with, and which approach we should take to use it. And it supports microservices very well.
Today, Wercker’s entire internal and public-facing stacks run on Kubernetes.
Embracing CI/CD
The tricky thing about CI platforms is that they don’t enable continuous delivery, (CD) on their own. To achieve full CI/CD, you need complete automation that fully removes the requirement for humans to perform manual tasks when building and releasing software. Full CI/CD also implies a fully repeatable and predictable pattern for building, testing and deploying software.
Getting to complete CI/CD for our internal development has been one of our primary goals at Wercker. We’ve made great strides in that direction, largely by embracing tools like containers. They provide important advantages, such as parity between testing and production environments, which make CI and CD easier to achieve.
Of course, we still face some challenges on the CI/CD front, the greatest one being the orchestration of testing for different microservices. With a monolithic app, testing it within a CI/CD pipeline is simple because there is only one service. But with microservices, you test multiple interdependent services. You can spin up separate containers, but things get complicated when you identify a problem with one microservice that requires you to roll back others.
We have yet to identify a perfect solution to this challenge. But at Wercker, we have found ways to address it: for example, the ability to roll back software deployments. We can roll an application back to an earlier version quickly in the event that our tests reveal a problem with a specific microservice, and we need time to sort it out. With rollbacks, the production application remains stable even if issues arise while testing updates.
So achieving full CI/CD is not something you can do overnight. It’s a process that requires adaptations whenever you adopt new practices, like adopting microservices. But we’ve identified strategies for working through the challenges we face as we continue the journey toward complete CI/CD.
Conclusion
I wish I could tell you that Wercker has solved all of the software problems known to man, and we’ve learned to make perfect use of the technologies available today. That, of course, would be a lie.
What I can say, however, is that we’ve learned over the years to make adjustments to enable us to face changing challenges. Adopting tools such as Docker and Kubernetes, and striving to build a full CI/CD pipeline for our internal development workflows, have helped us convert modern software realities into assets rather than liabilities. We’re no longer stuck in the painful world of running Jenkins in isolation or trying to maintain our own container project, rather than making the most of collaboration within the open source community.
Wercker is a sponsor of The New Stack.
Feature image: A 1927 movie poster for Fritz Lang’s classic sci-fi masterpiece Metropolis by Boris Bilinsky, shared by Simon Perkins and released under Creative Commons 2.0.
