Manage Kubernetes Spend with Cultural Buy-in, Granular Insights
Kubernetes is the fuel accelerating digital transformation initiatives across hybrid environments. One study found that 77% of organizations feel that Kubernetes is a central part of their pandemic-accelerated digital transformation strategy.
Despite the growing enthusiasm around Kubernetes, a range of stakeholders in large enterprises, including CFOs, CIOs, IT architects, DevOps professionals, and engineers all share a common pain point: it’s way too easy to overspend on Kubernetes. In 2021, the The Cloud Native Computing Foundation (CNCF) found that over two-thirds of respondents expected to spend at least 20% more on Kubernetes compared to the last 12 months, and 10% are already spending more than $1 million per month on Kubernetes deployments.
Small oversights and lack of visibility can quickly snowball into larger-than-necessary and unpredictable Kubernetes bills. In fact, the same 2021 CNCF survey found less than 25% of those surveyed said they could accurately predict how much they’d spend on Kubernetes to within 5% of actual cost.
If your organization is ready to pursue an organization-wide approach to containing Kubernetes costs, here are four things to consider:
Importance of Cultural Buy-in
The commitment to reduce Kubernetes spend needs to come from different stakeholders and must include a top-down and bottom-up approach. From a technical perspective, implementing cost controls for Kubernetes spending isn’t an effective strategy unless the organization’s culture is set up to support cost management — and every team trusts and evangelizes cost management methods. By ensuring the transparency, accuracy, and fairness of allocated costs, and by providing the monitoring and tooling to enable teams to make data-informed cost control decisions, you’ll get the buy-in you need to make the transition successful.
Assisting Your Developers’ Careers
Digital transformation continues to accelerate while the Great Resignation rages on. Organizations lose valuable time and resources if engineers burn out, get bored, or pursue more appealing job opportunities. One trend is consistent: developers want to stay relevant and will seek out positions where they’re able to work with modern technology stacks. Today, that means the best engineers want to work with Kubernetes, and organizations that leverage this technology to inspire and challenge their teams will be able to reduce attrition among technical staff.
Recognition of a Hybrid World
If you went to a tire center, would you expect that they check and service all four tires, or would it be adequate if they only evaluated two or three? Organizations looking to reduce the exploding costs of Kubernetes face a similar situation without a comprehensive view of both cloud and on-prem costs. CNCF found 5.4 million developers are using Kubernetes, and most (90%) of these are using cloud-managed services — and many are also using Kubernetes on-premise. Enterprises require visibility into both on-premise and cloud deployments in order to have a full understanding of Kubernetes spend.
Granularity Matters
Developers and engineers impact Kubernetes spend, but they need insights and data. They require visibility beyond namespace, controller, or custom label, and they need quick visibility down to the cluster level and to break down cost by resource, including CPU, RAM, GPU, storage, and network.
To foster cultural buy-in and implement accurate Kubernetes cost monitoring, consider these best practices:
1) Phase in chargeback strategies
Organizations realize greater savings through robust showback or chargeback methods that emphasize team accountability. The chargeback method goes beyond showbacks to actually require teams and business units to pay the Kubernetes and cloud costs they’re responsible for out of their budgets. To introduce chargebacks effectively, an organization must first achieve broad cultural buy-in to the importance of controlling these costs. Start with limited monitoring or using a showback method. It is not unusual for teams’ initial spending reviews to include a series of surprises. Going straight to a chargeback strategy can foster resentment from team leaders who need time to understand the costs they’re now more acutely responsible for (and how to control them). Giving teams access to cost and allocation data first offers them time to vet the fairness of the costs allocated to their team, and to introduce cost controls thoughtfully, before the bills begin arriving.
2) Transparency demonstrates the fairness of cost allocations
By its nature as a distributed system, Kubernetes makes cost allocation complicated. Ensuring fairness and gathering buy-in across teams requires completely transparent and reproducible allocation models. Be sure to audit these findings as well, to verify that all costs assigned to teams are correct. Importantly, allocation data must also be actionable — enabling teams to clearly and directly address their sources of overspending.
To ensure fair and actionable data, look at the following criteria:
- Consider how idle resources are allocated (often this relates to whoever makes provisioning decisions at the cluster level).
- Look at allocation of system-wide or shared resources.
- Make sure that team or business unit resource allocations are carefully delineated; for example, allocating by namespace offers a clear-cut approach.
- Decide if allocations are based on resource requests or usage (using the maximum of requests and usage is recommended, but only if teams are able to control those settings).
- Lastly, decide how to fairly bill teams responsible for costly but non-recurring jobs, such as research projects.
3) Assign each resource a clear owner
Define an owner for each resource, labeling the owner at the deployment, namespace, or cluster level. This enables a clear escalation path in case issues occur. An Open Policy Agent or admission controller webhook can enforce these labels.
4) Review spending data regularly
Proactive and frequent data review enables teams to rapidly identify overspend and avoid expensive future waste. Set up automated alerts to warn teams about excessive or unusual resource usage that could result in cost overages if left unchecked.
Organizations require a holistic approach to Kubernetes cost management. Once the transformational benefits are understood across the company, a collaborative cultural shift can begin. By arming developers and engineers with the right processes and data, teams can not only reduce costs by 30% or more but can ensure a successful, effective digital transformation to meet modern enterprise needs.
