Living with Kubernetes: API Lifecycles and You

Welcome to Living with Kubernetes. In this series, we’ll talk about what it takes to use Kubernetes — beyond creating a cluster and deploying applications. We’ll look at what it takes to safely upgrade your clusters, explore tools that can help you manage multiple clusters, and discuss other considerations for using Kubernetes long term.

This article is all about the Kubernetes API: what makes it great, how it can be extended with Custom Resources (CR), and what it means when parts of it are deprecated.

The Kubernetes API is the most powerful part of Kubernetes. It provides a predictable, extensible API for your infrastructure and applications. The predictability comes from well-designed usage patterns and strong contracts for stability. With those things in place, the API was always intended to be easily extensible and to change often.

The end result is something you can rely on to drive complex control loops with minimal, declarative data. These control loops are the key to everything, from the Kubernetes scheduler to GitOps. They’re at the heart and soul of having reliable and scalable applications with Kubernetes.

This pattern is so reliable that the industry has been adopting it for more uses than were originally imagined. It doesn’t matter if you’re using Kubernetes to deploy via the Cluster-API, running databases at scale via Vitess, or ordering a pizza with cruster-api. You can use the same Kubernetes primitives to solve business needs — or hunger needs.

What makes the Kubernetes API so powerful, and what’s the best way to be a consumer as the API changes? Let’s look at how the API is changing and how that affects you as Kubernetes becomes a critical part of your infrastructure.

API Groups and Extensions

The Kubernetes API is made up of different groups. These groups allow for

• Predictable API patterns
• Progressive adoption of features by users
• Independently defining resource scope and maturity
• Unique API paths for specific use cases (e.g. pod/logs)

Originally, the Kubernetes API didn’t have groups. This functionality was later named the core group (sometimes called “legacy”). Resources in the core group proved to be hard to mature, because of the tight coupling between resources and version number. With all resources behind a single /apis/$VERSION/ path, it was difficult for users to use resources from different versions and maintain compatibility between controllers. Developing the API required more versions to mature resources, and extending the API was not possible until the addition of Third-Party Resources (TPR). As customers and vendors started adopting TPRs, various shortcomings were addressed, and TPRs were replaced with CustomResourceDefinitions (CRD).

Since its inception, Amazon Web Services (AWS) has been the best place for customers to build and run open source software in the cloud. AWS is proud to support open source projects, foundations, and partners.

Learn More

The latest from AWS

CRDs are so successful that resources in the core group are slowly being moved into more specific API groups. Some API groups include apps, extensions and scheduling.k8s.io. You’ll see these groups in your spec files as part of the apiVersion field.

Other Kubernetes resources can be separated into namespaces. This shows up in the API as namespaces/$NAMESPACE in the following example:

This approach means we can perform predictable actions on a resource, such as reading all deployments in the “todo-list” namespace with the GET HTTP verb:

Or, we can read a specific deployment named “frontend” in the todo-list namespace with:

This pattern makes it incredibly easy to extend the API with new resource types and groups.

You can use the apiextensions.k8s.io group to create your own Custom Resources (CRs), which will create brand new groups to use. Here’s a minimal example to create a group mine.k8s.io with a version v1alpha1 and a resource type sock.

By submitting that small amount of data to the Kubernetes API, you will get the following API:

We can immediately check which sock resources we have with kubectl get socks --all-namespaces.

These constructs keep the API flexible enough to grow for unknown use cases and simple enough to understand and be approachable.

CRD vs. Aggregation

One thing to point out is that Custom Resources allow the Kubernetes API to recognize custom resources, and the API path is part of the main kubernetes-api process. You can also extend the Kubernetes API with an aggregation API server to claim a specific path (e.g., /apis/socks.mine.co/v1).

The main difference is that an aggregated API will proxy requests to a Kubernetes service endpoint. This means you will need to run a service inside your cluster that is responsible for state storage and version lifecycles. We won’t specifically talk about aggregated APIs, but it is important to know the difference when extending the API.

It’s important to become familiar with these concepts. In this article, we’re focusing not on how to use these things, but how to live with them. How Kubernetes adopts and grows the API is through strict contracts about how groups are versioned.

Versions and Lifecycles

Versions follow the group name as part of the URL path — or follow /apis/ in the case of the core group — and have defined guidelines about how things change from one version to another. These versions will show up in your yaml files after the group name in apiVersion.

Kubernetes follows a maturity progression of alpha → beta → stable with some additional versioning, so that a resource can iterate without needing to progress to the next level of maturity.

An alpha resource can start at v1alpha1 and iterate with v1alpha2 or, if there are breaking changes, maybe v2alpha1. A beta API may be the same spec as an alpha API, but the maturity and contract with the user will be different.

• Alpha APIs are experimental. They can have bugs and backward-incompatible changes. They are not enabled by default, and you should use them sparingly.
• Beta APIs are well tested and enabled by default. They can be relied upon for future functionality, but their implementation may change based on user feedback or constraints such as scalability.
• Stable APIs don’t have “beta” or “alpha” names. They are represented with a version (e.g., v1) and their implementation should not have breaking changes without changing the version number.

When a Kubernetes API is deprecated, it usually means that one of its versions is no longer available. The vast majority of deprecations have happened because:

• Resource scheme changes (e.g., v1beta1 → v1beta2).
• APIs have become more stable (e.g., v1alpha2 → v1beta1).
• Group names have changed (e.g., ingress moved from extensions/v1beta1 to networking.k8s.io/v1beta1).

A deprecation means a version of the API has been removed, and you need to verify in your manifests and resources that you’re using the correct version of the API. In some cases, you may need to change your resource fields to update the new scheme for the resource.

If an API is available in multiple versions at the same time, the Kubernetes API can silently upgrade some of them for you. However, you should still make sure you have the correct resource scheme — especially since as alpha APIs mature, the scheme may change between versions.

These versions matter, because when you want to upgrade your Kubernetes API server, you need to make sure your resources — stored in etcd and in static manifests — match what is available in the server. This process gets very tricky when you have version skew in multiple clusters or environments.

This step is also critical for your custom resources. Even if you don’t update your Kubernetes version, you can still have breaking changes between controllers and group versions if you don’t make sure to match or migrate them together.

Upgrading Kubernetes and Validating Manifests

You can get a list of API groups and versions from a running cluster with the command kubectl api-versions. The best part about doing this against a running cluster is that you can also see the custom resources and aggregated API groups you have.

If you don’t have a running cluster, you can see the default groups and versions available in the API reference documentation. Change the API version to your target version in the URL.

If you want to verify your static manifest files against a specific Kubernetes API version, check out pluto. You can back up static manifests from a running cluster with a tool like velero, or you can run pluto wherever you store your deployment manifests.

Pluto can read a directory of files and let you know if there are deprecations against a specific Kubernetes version.

Pluto can also help with default resource types but not custom API groups from CRDs or aggregated APIs. Running Pluto is a good smoke test to validate configs before updating your API server, or before deploying existing manifests to a new cluster.

Pluto can help with default groups and versions, but how do you upgrade your custom resources? If you’re using or creating CRDs via helm, you can check out its documentation on how to best use them. If you’re using CRDs from third-party vendors, you should check their documentation on how to handle upgrades.

If you want to compare CRDs in an open source project from one release to another, check out docs.crds.dev. It will help you quickly see which CRDs are included in the project — including what API group and version.

If you need to validate more than just API version differences, you should also look at kubeval and conftest. These tools can help validate manifest files based on scheme documents or open policy agent.

Custom Resources

For Custom Resources that you create, here’s how you can handle upgrades and deprecations. Testing your CR upgrades is important, in order to make sure your controllers are able to function properly with the custom resource and the Kubernetes API groups and versions. CR definitions may or may not need to be updated when you update your controllers for scheme and API version changes.

CustomResourceDefinitions can have multiple versions defined inside their specification. This approach allows the Kubernetes API to serve multiple versions at the same time.

Using our socks example from earlier with version v1alpha1, if we wanted to also serve a v1beta1 version of the API, we can define that with:

This specification means both /apsi/mine.k8s.io/v1alpha1/ and /apis/mine.k8s.io/v1beta1/ will be served from the API. When a new “sock” object is created, it will be stored in etcd as v1beta1 because of the storage: true under the v1beta1 version. Only one version can be stored in etcd.

If your versions have scheme changes, you’ll need to modify the resources when they’re submitted to the API. This step is handled via a conversion webhook. The webhook is responsible for reading the resource and converting the scheme to a different version and sending it back to the API server.

You can add the conversion webhook spec to your CR like this:

Any time a sock resource is created in the Kubernetes API server, its specification will be sent to the URL specified for conversion. The conversion webhook should do whatever is necessary to the resource and send it back to the API server as a ConversionReview object.

Using webhooks for conversion allows a lot of flexibility to manage CR lifecycles and lets you slowly migrate from one version to another as needed.

A CR with a conversion webhook requires the same amount of components as an aggregation API, but with CRs you can take advantage of etcd to store your objects. Plus the default API patterns can simplify how much you need to maintain.

Once all of your resources have been updated, you can deprecate an old version by using deprecated: true in the CR definition version. Deprecated versions will still be served by the API, but they will print a warning when resources are submitted to the API server using the deprecated version.

Conclusion

Kubernetes API was designed to change. One of its core strengths is to be flexible in any environment. Being aware of which groups and versions your resources are using is a responsibility users have in order to make sure their resources are compatible with the current Kubernetes API.

In many cases, resources can be transparently modified and stored as newer resources without any user action. This capability allows users to be more confident in API upgrades and allows for gradual scheme changes.

No matter whether you verify your resources statically with a tool like pluto or automatically convert your resources with a conversion webhook, it’s important to make sure you are able to safely migrate your resources from one version to another. Adding these tests early will help give you confidence as you use Kubernetes long term.

Justin is a Senior Developer Advocate at Amazon Web Services (AWS).

Read more from Justin Garrison