With REX-Ray, Dell Backs the Container Storage Interface Spec
“There is no such thing as a stateless architecture,” declared a presentation slide seen, no doubt, by a fair plurality of proponents of stateless architecture. It was the backdrop for a Dell EMC World session last May, and it set the theme for how Dell Technologies would build a bridge from the open source world of containerization to the proprietary world of its Isilon Scale-Out network attached storage and ScaleIO software-defined block storage.
Now, the new {code} open source division of Dell (complete with curly brackets) finished a critical stage of work on what may become a vitally important project to the continued success of containerization in the enterprise, taking the containers out of the realm of the stateless.
Entitled REX-Ray, it has grown to become the first stable, working model of the open source Container Storage Interface (CSI), an effort to define an industry storage standard that will enable storage vendors to develop a plugin once and have it work across multiple container orchestration systems. In effect, CSI would be “a vendor agnostic storage orchestration engine.” And although its goal is to pair Kubernetes, Mesos, and Docker Swarm with the world’s principal storage platforms, achieving that goal in full — according to its own architects — will mean REX-Ray will step out of the picture altogether and cease to exist.
It’s the matchmaker for containerized storage. And if the match ends up being made in heaven, then will enterprises care whether it violates one or more of the notorious 12 factors?
This Persistent Need
The question that enterprise IT managers have been asking for the last four years, and continue to ask today, is this: If containerization is supposed to be this spectacular new architecture that replaces self-contained, hypervisor-driven virtual machines, then how come they haven’t resolved the problem of universal support for persistent storage?
There is an answer, and it’s an architectural one: Virtual machines had little or no problem addressing storage volumes because their installed applications were being executed by full-scale operating systems. There, the same drivers that made storage volumes addressable by physical servers worked perfectly well in virtual environments. Many of the organizational problems that skeptics cite as indicative of the immaturity of containerization, were only “solved” in the VM realm because VMs were snapshots of physical servers.
Distributed applications cannot be, by design, snapshots of programs running on single processors. That’s one huge reason containerization came into being in the first place: VMs aren’t scalable. But the design ethic credited with containers’ rise in popularity — the stateless, portable, 12-factor model — has taken a backseat or has perhaps crawled into the trunk, as mainstream enterprise adopters demand compatibility and a permanent link to their databases. They often don’t know it’s called “statefulness” now, but that’s what they’re demanding nonetheless.
“I think this a big area of the market that has prevented broader adoption of containers,” said Joshua Bernstein, a vice president of technology for Dell Technologies [pictured top], who spoke with TNS at the Open Source Summit in L.A. earlier this year.
“Remember when we were like, ‘Oh, 12-factor, microservices, we’re going to do all this! Storage, we’re going to do stateless applications?’ The reality is, the vast majority of real applications need to store state,” Bernstein continued. “I want to run MySQL in a container, or Cassandra in a container, and I want to do these things. So I have to solve this problem.”
During the last MesosCon, held in conjunction with the Open Source Summit in L.A., Bernstein moderated a panel of Mesos users that included Michael Aguiling, a chief technology officer at financial services giant JPMorgan Chase. Aguiling explained the problem from his firm’s perspective, describing how its current applications ingest and process data from Spark and messaging from Kafka — two of the major components in the emerging “SMACK Stack.”
Dell Technologies VP Josh Bernstein [left] moderates a panel with Michael Aguiling, CTO, JPMorgan Chase [speaking]; Stefan Bauer, head of analytics for Audi; and Cathy Daw, director of engineering at Mesosphere.
“For the longest time, the lifecycle of data was really unknown,” the CTO continued. “Data was always attached to a database; it was always attached to an application. But what if I can get that data in its own, rightful place in the center, and the applications come and go? Sometimes streaming, sometimes real-time, sometimes batch; but now the data becomes the core focus, and the applications and the capabilities, well, they’re just features now.”
The Mounting Challenge
The New Stack has covered the development of the Container Network Interface (CNI) since it began: a way for containers to be universally perceived as components of a dynamic network. CSI is intended in the same vein, as a universal means of connecting containers with storage platforms.
Yet there’s a very big difference between the mountains that CNI and CSI have had to climb. Both physical and virtual storage are markets unto themselves. Back when desktop operating systems mattered, Windows had the unenviable task of managing and maintaining a myriad of storage drivers — usually, proprietary tools, sometimes supplied by the manufacturer of the storage appliances, but often assembled with duct tape and baling wire by Microsoft itself.
Minimalized Linux kernels included within container images don’t have this daunting responsibility. And container orchestrators (which Dell calls “COs”) are too busy maintaining their own clusters to be concerned with big brand-name appliances.
“In the virtual machine days, the VM encapsulated the data or the file system,” explained Bernstein. “And when a VM would start up, maybe it would perform an NFS [Network File System] mount. With a container, that might not necessarily be the case, because of namespacing, security, isolation, and other types of things. Also, if the namespace or the endpoint changes, or is stored across different types of providers — Amazon, on-prem, NFS, or whatever — it has to be all abstracted. You don’t want the container to say, ‘Now I’m running on Google, so now I have to call the Google API.’ That’s what REX-Ray does.”
An original paper collage work by artist Rex Ray appears at the Turner Carroll Gallery in Santa Fe, New Mexico. [Photo by Michael Carroll, released under Creative Commons 3.0]
“We have a number of container orchestrators… that have evolved independently, with their own independent implementation of a storage interface. When they’ve implemented this, they typically even integrated into their own versions of storage plugins that live in the source tree of that container orchestrator. That is kind of bad too, because with many of these orchestrators, they’re on fairly time-consuming release cycles, and the underlying storage provider… is on its own release cycle.
“If you come across a scenario,” Wong continued, “where you’ve got a security patch or a bug fix that originates in the provider of the underlying storage, you might have to wait to get that into the source tree of the container orchestrator. And that’s a bad thing.”
At least it’s a bad thing now. Back when the lifecycle of the operating system set the cadence of the entire system, users were resigned to having to wait a few years anyway. But continuous integration means, at the very least, something more granular than biennially.
The Disappearing Solution
“In an ideal world, the container orchestrators would have a common interface to talk to storage providers,” Bernstein said, “and storage providers would offer a common interface outward. We did this with NFS. An NFS server and an NFS client could talk to each other.
The CSI is an effort to design a specification that all of the container orchestrators can implement, and then all of the storage providers can also implement so that nobody needs REX-Ray anymore.”
As Bernstein perceives it, REX-Ray is a stopgap or a shim, providing the ubiquity of connectivity that Swarm, Kubernetes, and Mesos (and with it, DC/OS) lack natively. It implements CSI the way he believes the container orchestrators should do, eventually. But to bring that eventuality closer to the present time, there needs to be more use cases of fully interfaced, persistent storage devices with containers in production. That won’t happen until more enterprises adopt containerization as a guiding principle; and they won’t do that until, well, there are more workable use cases.
Until CSI, or something like it, is grafted onto COs, developers will need a reference implementation that they can test against. Directly implying that such reference implementations don’t magically sprout forth in the wake of open source community hoe-downs, Bernstein said, Dell took the initiative, effectively donating its own incubator to resolve the chicken-and-egg issue.
“REX-Ray’s not in the data path,” Bernstein told us. “It’s a stateless service. It scales with as many nodes as you would like.” As a result, it’s capable of providing a namespace with access to as many volumes as the storage provider itself allows — for example, if an Amazon AWS service has a native limit of 128 volumes per namespace, REX-Ray’s limit in that instance would be 128.
By way of drivers, REX-Ray now supports EBS, EFS, and S3 File System from Amazon, as well as Digital Ocean Block Storage, Google GCE Persistent Disk, and most recently, Microsoft’s Azure Unmanaged Disk. There’s also generic NFS support, of course, along with OpenStack Cinder and a growing list of other platforms.
Verizon and Citi are among the {code} group’s early implementers of REX-Ray, according to Bernstein. “The future to us,” he said, “looks like a native connection between the container orchestrators and the storage providers.”
{code} is a sponsor of The New Stack.
