DevOps: Kubernetes Help Desk or Self-Driven Innovation?
Adopting cloud native technology is a journey that takes time to mature. In most cases, teams start playing with containers, Kubernetes and adopting microservices in small projects. But as the projects grow, so does the complexity. Kubernetes is still a relatively new technology, and many developers are just now starting to come up to speed on it.
DevOps teams are increasingly selecting, deploying and managing Kubernetes. They do a lot to make the platform a success by making it easy to deploy and scale microservices on Kubernetes. The DevOps teams become the bridge between security, finance and development teams. They’re the ones at the center of everything and have the task of bringing it all together and making it successful. And while they start out innovating and want to continue to try new technologies or make the platform better, too often they become the Kubernetes help desk, i.e., they are constantly responding to basic inquiries from developers and the rest of the organization.
These basic inquiries could be reviewing why an application is down and discovering that no one had set liveness and readiness probes. It could also be responding to developer questions about why an application’s build broke and having to stop everything to configure it correctly and then redeploy it. It may be because the development team never set container security or because there are other missing configurations (if a policy engine is working in the CI pipeline).
What’s so Wrong with Running a Kubernetes Help Desk?
Absolutely nothing. At some organizations, there may be enough staff and resources to dedicate a team to this. But that’s often the exception, not the rule. DevOps and SRE teams are understaffed, overworked and spend too much time tactically firefighting or in support mode for developers.
This takes away from the innovation side of DevOps — strategically driving the implementation of new tools and technologies to improve the development process. Consider how one company’s DevOps team had to drop everything each time a developer wanted to push to production: There was a manual checklist that had to be followed, and it took a week to get it done. It slowed down the DevOps teams’ sprints and also the development projects. A lose-lose for all involved as “we’re spending all of our time auditing as opposed to innovating.”
This doesn’t take into account that by acting as the Kubernetes help desk, DevOps team members often aren’t able to work on innovation or find creative solutions to problems, which results in mundane work and high levels of burnout.
It’s Fine, I got CI/CD
Many reading this will not consider this an issue because of the CI/CD pipeline in place. After all, it is there to help automate the process to build, test and deploy code. It’s an important part of the cloud native toolkit, but it doesn’t take away from the DevOps help-desk problem.
Kubernetes is not secure by default, and care must be taken to ensure the appropriate guardrails are in place to secure the environment. While Kubernetes is built to scale dynamically, auto-scaling can be abused if containers are not configured properly, creating a FinOps disaster if no one is tracking cost optimization and cloud spend. Additionally, it can have performance and reliability issues without the right configuration in place. The CI/CD pipeline doesn’t solve for these things.
Kubernetes Governance from Dev through Production
Much of the help-desk services DevOps teams provide are around configuration issues, which can and should be automated. Instead of simply being brought problems that need fixing by developers, DevOps need the ability to provide instant feedback on what isn’t working, why and how to fix it.
This is where Kubernetes governance and guardrails become essential to both the sanity of the DevOps team and developers. A service ownership culture — putting in place guardrails and actionable feedback — can help enable developers to self-service fixes in their deployments.
It starts first by understanding what policies need to be followed. Best practices for Kubernetes include considerations for security, reliability and cost, for example, not running containers as root, setting liveness probes, and adjusting CPU and memory based on actual usage. Turning these best practices into policy-as-code (or guardrails) and then enforcing them from dev through to runtime alleviates the time DevOps teams need to spend hand-holding.
By incorporating a service ownership model, you empower developers without them needing to know the ins and outs of Kubernetes. By giving them guardrails within the tooling they already use (i.e., incorporating policy-as-code into CI/CD checks) with recommendations on how to fix problems, developers won’t get stuck, won’t have weeklong delays waiting for DevOps resources and will ship applications faster.
It’s time to free up DevOps teams to do the projects and sprints they want with advanced, automated Kubernetes governance tooling. It’s time to get them out of being the help desk for development.
