Ingress Controllers: The More the Merrier
Just like everything in the software development space, especially in today’s cloud native world, fragmentation is everywhere. As with any single category of tool — service meshes, orchestrators and observability tools — you will find multiple “brands” and variations of each tool being used in most organizations.
We can identify two main causes for such fragmentation: One is deliberate, and the other is not. Let’s talk about the non-deliberate cause first and how that relates to my own service mesh company Traefik.
Non-Deliberate Fragmentation
Reason #1: The Power of Defaults
Ingress controllers have, for the most part, become a commodity — more like a utility. Gone are the days when developers deliberately chose their ingress controller. Of course, there are exceptions. There are always exceptions.
This is not to say that developers do not have a preferred ingress controller, but it means that what they end up using is mostly whatever came with the Kubernetes distribution (or another container orchestration platform) they are using. In other words, like most people, most developers are satisfied enough with what comes “included.” This makes sense. Ripping out and replacing your ingress controller is not on most developers’ list of top priorities, especially when you have a feature or some other tight deadline (or fun project) to get started on.
Let’s say I’m an engineer working at Company X. My boss comes and asks me to start a new project. I have two weeks to get the proof-of-concept completed. My manager does not specify what tools I can or cannot use. The focus is on scoping the project and business requirements. It is not a complex project.
Between all the other projects I manage and the desire to not spend all my time working, I fire up a new development cluster, and for the purposes of this example, my chosen distribution happens to come bundled with Traefik. I’m not going to then spend precious mental cycles ripping Traefik out and replacing it with Nginx, for example. I am going to get the cluster set up and then get started on building my apps.
Imagine this happening with dozens of other teams within your organization. Each team has a slight (or not) affinity or preference for a given Kubernetes distribution. Most Kubernetes distributions come with a default ingress controller. Whatever comes with the distribution is often what the team or individual will stick with until someone finds the time and energy to change it out. Which is basically never. This is the power of defaults.
Reason #2: Development vs. Production Environments
Another important driver of ingress controller fragmentation is that in many organizations, Ops spins up a cluster using their ingress controller of choice. For the sake of argument, let’s say that is Nginx. But for my local development environment, I prefer to use K3s, a lightweight Kubernetes distribution that comes bundled with Traefik. Just in this one instance, you are using two different ingress controllers. In these scenarios, engineers often find themselves asking how they can ensure that they are able to port their application from their development environment to the production environment without hiccups.
Reason #3: Lack of (Kubernetes) Standardization
The third driver of fragmentation in the ingress controller space is due to a lack of standardization in Kubernetes itself, especially around features and functionality that users required (e.g. access control, certificate management, middleware, etc). This void of missing use cases allowed vendors to develop their own solutions. Once everyone began forking the project, fragmentation followed. Ingress controller providers like ourselves created our own implementations and unique solutions to those use case requirements, oftentimes having to rely on new and/or custom tweaks such as custom annotations or Custom Resource Definitions (CRD). Google itself is a good example and an early mover in filling the gaps left behind by Kubernetes. They made some inroads into standardizing API specifications (which Traefik Labs contributed to).
It’s important to note here that the Kubenetes’ missing functionality was intentionally designed that way. The ingress spec is a basic interface against which developers could write their code, intentionally leaving the actual handling to a third party (such as Traefik). Open source projects, by nature, make it easy to fork or build a derivative project. Kubernetes is no exception. This is actually one of the greatest joys of working as a “third-party provider” in the open source world. It makes projects like Traefik possible, and even necessary. In other words, I’m not saying that this unopinionatedness was a bad thing; if anything, it created opportunities for projects and companies like ours. The problem is more felt by users, and specifically for organizations, who have to deal with the consequences of their development teams using a variety of derivative solutions.
Deliberate Fragmentation
“Deliberate” fragmentation most often happens when you have a specific technical or business requirement that your preferred or default ingress controller does not support (or support well), which forces you to use another ingress controller. For example, one solution can be best for large streams of binary data like data sets or video streams whereas another offers robust support for web sockets. HTTP/2 lends itself better toward real-time bidirectional communications like chat apps and games.
Whether deliberate or not, the ecosystem within organizations is starting to look a lot like the ecosystem on the outside — fragmented and varied. Vendors should embrace ingress controller diversity — and some are (see the Google API Gateway example above). They should decide how ingress controllers can complement each other — or at the very least be interoperable — and make our users’ lives easier and better, rather than trying to push each other out. Because at the end of the day, vendor wars and infighting only really benefit the vendors — and even that is arguable, especially in the long run.