Why Microservices Require a Storage Rethink
Developers are increasingly looking at how to migrate to a microservices platform at least in some way — and for good reason, given the well-documented jumps in scalability, application performance and other advantages on offer.
And yet — microservices, in many ways, represent a radically different computing environment compared to monolithic deployments. And since the computing structure of microservices represents something fundamentally different, many organizations are faced with trying and testing new and different approaches for data storage compared to monolithic alternatives.
“Microservices and containers were seen as stateless entities — until they were not. They quickly became, like most data sources, needed data sources for analytics, AI, machine learning and data science,” Bill Peterson, vice president, industry solutions, for MapR, said. “As such, they needed to become stateful and they have needed to become part of the enterprise data ecosystem.”
Suddenly, data is generated from numerous and different microservices, as opposed to data generated from a single monolithic computing structure. In this way, storage and database access must take into account every microservice, which can often number in the hundreds or possibly thousands of individual threads.
Schematically, Torsten Volk, an analyst for Enterprise Management Associates (EMA), outlined three issues to keep in mind:
- Performance and reliability problems: “Using drivers for persistent block storage that is dynamically mapped to containerized microservices has proven difficult, as block storage was never designed to be rapidly mapped to and detached from their targets,” Volk said. “In addition, customers struggle with data integrity of their block volumes when they have to be mapped to larger numbers of containers at the same time.”
- Compliance: “Enterprises are struggling to keep a unified audit log to prove the compliant storage, encryption and use of data,” Volk said. “While virtual machines often stay in place for months or years, this is much harder for containers that could spin up and create events and then spin down after only two to three seconds. Monitoring tools often do not pick up these rapid changes, which is causing a whole within the audit history.”
- Backup, data recovery and archiving: “While Kubernetes offers built-in redundancy, enterprises still need to comply with backup, data recovery and archiving policies,” Volk said. “Many of them are struggling to leverage their current tooling, typically used for vSphere virtual machines, to Kubernetes containers.”
Microservices, Not Containers
Microservices and containers are oftentimes, yet incorrectly, used interchangeably. However, according to an EMA study, 63 percent of enterprises surveyed — but not all organizations by any means — run microservices on containers. Microservices can just as easily run on a virtual server as they do in a container, for example.
For several reasons, organizations should look to decouple containers and microservices management, which also includes storage, Ben Sigelman, co-founder and CEO of LightStep, said.
“Ultimately, microservices are just an ‘idea’ or a notion, whereas containers are a mechanism by which microservices are often — but not always — packaged and deployed. Containers have many benefits, especially when trying to ease off of a VM-centric infrastructure, but in the final analysis, they will become an often-invisible implementation detail,” Sigelman said. “Durable storage has been difficult for containers since by default their storage is ephemeral. When it comes to durable storage solutions, many of the challenges stem from the desire to manage the storage both at the container layer and the microservice or application layer — the projects and technologies that win will be the ones that hide the implementation at the container layer once and for all.”
Security Rethink
Persistent storage benefits from microservices agility and power, but in many ways, data security risks are compounded compared to data monolithic applications create.
“From a security standpoint, microservices security, as well as microservices security in general, is a nightmare,” Gadi Naor, co-founder and Chief Technology Officer of Alcide, said. “Because the amount of moving parts that you need to keep in control and define policies, amplifies and multiplies. This is why edge elements are especially relevant.”
Locking down the data stored from the multitudes of microservices threads requires a three-pronged approach. This includes securing the data threats and the data in runtime, Naor says.
“Securing the data threats involves the obvious first step of encrypting the data that is transferred to the storage environments, whether on the cloud or in a locally attached storage environment. The encryption covers unauthorized access to that,” Naor said. “However, malicious code embedded in any one of thousands of microservices data threads within the protected perimeter is enough to allow an attacker to access data in the runtime unencrypted. In this case, the microservice might load data to the storage environment that resides inside a container or storage service on the cloud, which can be used to access and exfiltrate the data within the otherwise-protected perimeter.”
In the case of a runtime microservices storage attack, a runtime defense that reinforced the network perimeter can prevent the exfiltration of data. “If something goes wrong, third-party tools exist that let you know that something is off, such as something trying to access data in an unintended way,” Naor said. “And historically, things do go wrong like this. We keep hearing about those kinds of breaches.”
This runtime protection layer for microservices data storage covers the protection layer for what is the worst-case scenario for many organizations when an attack is orchestrated from inside the protected perimeter and firewall.
“You need to make then an assumption that if something goes bad or something did go bad, what’s your game plan?” Naor said. “And that is where building the runtime defenses help you deal with kind of the transition of the data from the storage layer into the runtime for microservices storage.”
Don’t Forget DevOps
Sadly, storage and databases often fail to benefit from the same level of agility that DevOps affords development pipelines for a number of reasons — while microservices data storage and databases are no exception to this rule. Still, DevOps play an equally critical role in both software and storage development for microservices.
“DevOps can build more applications with all the data possible available to them. For example, as new microservices are deployed, data volumes can be created and retained, even when microservices are deleted,” MapR’s Peterson said. “Giving DevOps access to this data independent of where the microservices reside.”
Developers can then ensure that applications can be synchronized and updated with a unified view, without disruption and with the latest/up-to-date data. With DevOps in at this level, developers can ensure that container data in their applications:
- Is most current
- Are high performing
- Secure
- Highly-available
- And protected
Thanks to DevOps, microservices storage can thus live up to its potential. “DevOps can build more applications with all the data possible available to them,” Peterson said.
Alcide is a sponsor of The New Stack.
Feature image via Pixabay.
