Data Persistence and Storage over Pancakes at KubeCon EU 2021
At this year’s KubeCon + CloudNativeCon EU 2021, some things were the same — it was still virtual which meant it attracted a huge turnout of a more broadly international audience — and some things were different — like that almost everyone’s bought into Kubernetes and cloud native architecture, it’s now just how they use it. In another KubeCon tradition, The New Stack hosted a live pancake breakfast to reflect on the maturity of Kubernetes particularly around data persistence and storage.
The New Stack publisher Alex Williams hosted this year’s discussion with Itzik Reich, Dell Technologies vice president, and Nivas Iyer, Dell senior principal product manager, along with pancakes regular Cheryl Hung, vice president of ecosystem at the Cloud Native Computing Foundation.
Williams kicked off this discussion reflecting on how in the last 18 months Kubernetes has become the de facto container orchestrator and now the current challenges of cloud native computing are about the tooling in that ecosystem, particularly around persistent data.
Reich jumped in to echo that, remarking how the enterprise customer base has really matured. They aren’t uncertain about containers and cloud, native but more the conversations with customers know what they want to use. He’s witnessing a particular impatience from developers who now want to leverage tooling to simplify storage, integrating the storage array with the upper-level orchestrator.
“They want to treat storage just like we all treat electricity in our houses. We want to flip a button and, within magic that’s going to occur, you know storage will basically become persistent to those containers. And of course, in order for this to come, storage needs to be super smart but it’s also super simple to use,” he said.
Iyer added that how traditional architecture is usually three-tiered, with security out on the third tier managed by a storage or database administrator. What’s evolved is now that work is decentralized and across several microservices and developers want — and expect — to have more control.
Data Persistence and Storage over Pancakes at KubeCon EU 2021
He said, “Now you’re seeing that each individual small development teams are managing their own databases inside the microservice, and so there isn’t a centralized data hygiene that’s being done. So now in Kubernetes I see a huge opportunity… they’re looking at storage on demand, but, at the same time it also needs to be something that is available, resilient, reliable and recoverable.”
Much of the discussion was around data persistence. Persistent data is the core data that exists no matter what systems created it, what instances are using it, and it can be retrieved again and again — even if there’s a power outage.
“Deployments are getting bigger. They’re getting more complex. People are expecting more out of their cloud native infrastructure.”
— Cheryl Hung, CNCF
Hung has been looking at persistent storage challenges since 2017, just now organizations aren’t so trepidatious about attempting it.
“I do see now that organizations are getting to the point where it’s not scary or not as scary anymore to consider running databases and running some parts as persistence using persistent volumes and persistent workloads,” She explained. She does note that persistent storage remains an issue for organizations moving to multicloud operations. “Moving stateless applications across clouds is not that difficult, but moving data and moving the analytics that is very, very difficult still,” she said.
We can replicate data across clouds, but that of course raises costs and makes data compliance harder. Iyer compares it to “drinking data out of a straw,” where the demand is for tooling that allows you to move data in a highly compressed format while ensuring data consistency across all the different clusters and clouds.
