The Convergence of Object Storage and Cloud Native Technologies
While Kubernetes is an ideal container orchestrator, its lack of natively built storage features could be considered an Achilles’ heel for application developers. Persistent and stateful storage is essential for many containerized applications, as well as infrastructure constructs within the Kubernetes platform such as the registry, logging, metrics, and more. Unfortunately, Kubernetes’ lack of native storage capabilities also hampers scalability, as well as other factors that are critical to application development, including automation, visibility, and security.
Fortunately, there are efforts underway in the open source community to address these Kubernetes storage blind spots. Initiatives like the recently announced Ceph Foundation, as well as open source projects and technologies like Rook and API interfaces, are filling in the container storage holes that Kubernetes has yet to fill. Let’s take a look at a few of these holes and explore what developers need to consider as they invest in Kubernetes and containerized application development.
Ability to Scale as Needed
Traditional storage appliances are not designed to handle petabytes (let alone exabytes) of application data across on-premise and cloud deployments. Conversely, object storage, which breaks down files into smaller and more digestible pieces, is massively scalable, making it ideal for large quantities of data. It’s ideal for applications that depend on a lot of data to function, such as those designed to deliver streaming services.
The Ceph project is dedicated to developing solutions that allow application developers and storage administrators to provide storage to big data repositories. The Ceph Foundation, a neutral consortium of industry leaders with a vested interest in the development of Ceph, was formed to ensure that the work being done on the project is well funded. The Foundation is also committed to making sure that Ceph continues to deliver on its promise to be a distributed object, block, and file storage platform that addresses the growth and demand in data storage and services.
The Ceph Foundation is especially important today. Enterprises are seeking modern storage solutions that can be used for both large amounts of data and containerized development. The Foundation will allow the project to retain the support it needs to be able to continue to develop innovative solutions that can scale and address both of these needs.
Automation for Easier Management of Large Datasets
Very few developers have a desire to become storage administrators. They want to focus on building their applications without having to learn the intricacies of storage management. If they’re developing in containers–which are all about speed and agility–they definitely don’t want to have to wait the usual few days for the necessary storage to be provisioned to them by the IT team. Instead, developers should be able to automate their own storage provisioning. With just a few bits of code, developers can automatically order just what they need, when they need it.
Rook and API interfaces provide this capability. Developers can use Rook to self-manage their own storage needs without manual intervention. They can classify the type of storage they need on the fly–some faster data here, more resilient data there, etc. Software-defined storage API interfaces can be used to automatically move and access data. These solutions represent a win/win for administrators and developers; developers can get what they need without waiting, while administrators maintain control over the storage without having to respond to thousands of requests.
Automation of storage provisioning and other common tasks will also become more important as data continues to grow. It’s simply not possible for developers to personally manage the massive datasets they need for their containerized applications. They need something that takes that job out of their hands. Kubernetes does it to an extent. So does the Operator Framework, which provides a toolkit to manage Kubernetes native applications in a more automated way. But the other options mentioned above can take things even further.
Better Visibility and Security
Visibility and security go hand-in-hand. In addition to helping developers understand how much storage is available for their applications, better visibility into containers may help identify any potential security issues.
While Kubernetes doesn’t offer great options for these important pieces to the storage puzzle, the open source project Prometheus does. Prometheus provides deep and advanced visibility and monitoring. Developers can use the tool to monitor massively scalable datasets, be alerted to storage problems, and more. It’s designed to work with Kubernetes and most enterprise Kubernetes platforms should have Prometheus enabled by default. Developers can also use the Prometheus Operator to help make running Prometheus on top of Kubernetes as easy as possible.
Greater visibility into containers may also help developers identify potential security issues. It’s important to know that a container running a certain device or application has access to the data it needs, but only access to that data. Ensuring that data is encrypted, whether it’s in flight or at rest, is also critical. Developers should also consider implementing additional layers of security for their containers.
Developers are facing an interesting dichotomy. On one hand, they’re developing an increasing number of applications with large-scale data demands. On the other, they’re working with cloud-native technologies that are designed to break applications down into smaller, stateless chunks.
On the surface, these two things don’t seem to go together, but they can actually be great together. The Ceph Foundation, Rook, and other open source initiatives could bring big data storage solutions into today’s cloud-native development world.
The Cloud Native Computing Foundation (Kubernetes, Prometheus) and Red Hat are sponsors of The New Stack.
Feature image via Pixabay.