Effective Microservices Architecture with Event-Driven Design
There’s no doubt, in the IT world, microservices are sexy. But just because you find something cool and attractive doesn’t mean it’s good for you. And it doesn’t mean you know how to use it properly
.
In fact, microservices, in particular, aren’t easy. Yes, microservices allow for different team members to work on different parts of the code at different speeds. But this wonderful autonomy often finds folks even more siloed, not knowing what the other is doing before throwing it over the wall to the testing and documentation teams, according to Red Hat’s Principle Middleware Architect Christian Posta.
In order to gain true microservices independence, Posta says that we need to shed our dependencies, such as the assumption that microservices mean each service controls its own database.
“The idea is sound: don’t share a single database across services because then you run into conflicts like competing read/write patterns, data-model conflicts, coordination challenges, etc.” wrote Posta in a recent blog post. Posta is Red Hat’s product specialist for integration messaging and the cloud platform product, traveling across North America, working with devs on what it means to be cloud-native.
If you have totally isolated roles then you need isolated data, except there are benefits to working with a single database. With a single database, you’re more able to get an overview of how everything comes together, as well as the ability to manage it all via one view. A single database also allows for ACID transactions — like when a withdraw from one back account and a transfer to another is viewed as one transaction.
Posta wants us to address how we take these conveniences of single databases and apply them to microservices. If each service has its own database, how can we maintain this independence while still reconciling changes and security across them all? According to Posta, the answer comes when you adopt event-driven design.
“By modeling the data within a set domain and context, microservices can shed these dependencies while still maintaining data consistency throughout the system,” Posta noted in a presentation at the most recent Red Hat Enterprise Summit:
The Problem: Where Microservices Are Still Missing
In his work, Posta sees all kinds of companies — from Amazon to Goldman-Sachs — calling themselves tech companies. He says that’s awesome but when traditional enterprises like insurance or risk management are used to outsourcing all things IT, and now want to go the other end of investing to in-house it all, but without considering how it can be a differentiator.
“Some of those companies — from the low level to upper management that endorsed these things — are getting really excited to do what Google or Netflix or Amazon did, and they end up investing a lot of money to do what they think those companies did to solve the problems they think these companies have and they aren’t getting the differentiator part of it,” he said.
These companies are investing in the currently sexy, like operative high networking and containers and, of course, microservices.
“And they end up sinking so many resources into some sort of platform, and they are on the hook to support, maintain, do bug fixes, security — working on things that aren’t super material to their bottom line or to delivering business value.”
For the last couple decades, these companies have relied on enormous strategic partners like IBM and Oracle to be their workforce and to use their software. This has left them with an ability of a more across-the-board view into their software, but, now that they are locked in, these companies are unable to achieve the levels of flexibility needed to increase this automation and autonomy.
“Now they are just going to try to in-house everything — maybe use open source — but sink a ton of money into something that’s not a differentiator,” Posta said.
The problem increases because each one of these open source tools have different communities and different release cycles.
In the end, these companies lack a differentiator and lack a way to make the most of the technology they are investing in.
The Solution: Event-Driven Design
Posta told The New Stack that since Martin Fowler wrote down his microservices observations, we have been working to build things exactly the way it was described, coming up with hard-fast rules to what microservices are, and “if you aren’t doing these five things, then you aren’t doing microservices.”
But one thing that is consistent: “Every microservice has its own data and they communicate on APIs, and the emphasis is on decoupling and all this stuff we know we need.”
Posta says this makes sense and compels many to want to then “build stuff so every microservice has its own database.”
However, he argues that without fully understanding what databases do today and the power that they bring and what relies on them — like transactions, concurrency, isolation, durability, materialized views, indexes — “We’re not fully understanding the principles of the decisions that were made of relational databases, [and] we can very quickly get into trouble reimplementing databases over API calls and implementing them much worse.”
This is why Post has developed this set of four questions we have to ask ourselves before beginning any microservices architectural transformation:
• What is the domain? What is your current reality?
• Where are the transactional boundaries?
• How should microservices communicate across boundaries?
• What if we just turn the database inside out?
“At least acknowledge that yes we will be building something that looks like a database and yes that’s very difficult to get right and a third thing that using an event-driven architecture lends itself to this problem than using an API-driven architecture,” he said.
So how is event-driven architecture different from asynchronous API-driven architecture? Event-driven architecture “makes you deal with up front the fact that data is not current. You have to tackle the idea of time and what data means and when is it current and when is it not current,” Post said.
“In older databases, everything was current, but, once you start making calls to the network, the data that you’re dealing with is, by inherent nature, stale,” he said.
The event-driven architecture makes that fact a central part of the architecture, understanding that when you deal with events, they are things that happen in the past, instead of assuming they are the current state, like with a bank account status that can change rapidly.
This event-driven architecture is the foundation of what LinkedIn, Netflix and Amazon all did.
“Enterprise architects, chief architects, the developers, they’ll be the ones spearheading something that implements resiliency, pulling in technology, architecture, management tools,” Posta said. “The most important part of microservices or any cloud-native architecture is the ability to look at the system and to understand what’s going on.”
Posta admits this is not a new problem, but perhaps a bigger one that modern tools and architecture can both draw more attention to and could complicate if not aware of. Tools in the observability space become more and more necessary to create metrics, logging and tracing of how these microservices interact, where things are slow and where problems are arising. These tools move toward a dynamic cloud-native environment and event-driven architecture where services could be changing all the time.
Posta pointed out how “another problem is having a deeper understanding of: What is the impact on our business? What does a slow transaction for customers mean to our core customers?”
