The New Stack @ Scale, Show 2: Microservices Complexity and Battlestar Galactica
Welcome to the second episode of The New Stack @ Scale podcast, our new, monthly program sponsored by New Relic that explores the issues that come with dynamic services and systems.
Recorded at AWS re:Invent, The New Stack founder Alex Williams was joined by co-anchor Fredric Paul, a long-time journalist himself and New Relic’s Editor-in-Chief. They welcomed two guests, neither of whom are strangers to the developer world and the complexities of distributed systems:
- Dave McCrory, CTO of Basho Technologies. Previously, Dave served as SVP of Engineering at Warner Music Group; as well as Senior Architect, Cloud Foundry at VMware; and as a Cloud Solutions Architect at Dell. He is also an inventor, an author, and a founder/CTO of several companies in the virtualization, systems management, and cloud computing spaces.
- James Governor, co-founder of RedMonk, a developer-focused industry analyst firm. James is also founder of Shoreditch Works, a coworking space in London with a developer-centric focus.
Later in the show, we hear from CEO John Sheehan about the infrastructure of Runscope, and also from Loris Degioanni, founder of sysdig, who speaks to the complexities of the new stack.
Also, make sure to keep your earbuds in place for the bonus section at the end of the podcast. You’ll be treated to a wildly entertaining deconstruction of “Battlestar Galactica” as an allegory about microservices, along with an equally valid analysis using “Star Wars,” the premiere of the soon-to-be-iconic “Parable of The Prize Bull,” and much more.
For other podcast episodes, check out the podcast section of The New Stack.
Alex sets the tone with the idea that there are two varieties of complexity: necessary and unnecessary complexities. One is necessary to solve a problem. One is an organic outgrowth of just systems and software built over time, cobbled together — basically a mess.
“Most of the systems that I see today are complex out of mistake or ignorance,” says Dave. “Someone was too lazy or didn’t want to invest in simplifying it, so you pay a price for that, and that price is complexity.”
“Not everything should, needs to be, or will be, a microservice,” he continues. “I think microservices are the hot term right now for distributed systems at scale.”
“Microservices can be effective, but they have lots and lots and lots of problems. Having (in a previous life) implemented a system that had lots of microservices, it creates nightmares. At the same time, if you have a simple problem, you should try and use a simple solution. If there’s already an existing simple solution, why go and make a complex one instead?”
Fred asks if we’re at a stage of evolution that just seems complex, because we’re not so far down this road yet, and he wonders whether, over time, complexity will diminish.
“No, they are complex,” replies James. “Distributed systems is always hard. You can make some abstractions that simplify some of the constructs within it. But, yeah, distributed systems is hard. There’s no way around it.”
“It’s kind of a mindset,” James adds. “That’s not to say that we don’t have an increasing number of people that can get the mindset.”
“Implement, re-implement, rinse and repeat,” James volunteers.
“We are constantly trying to do stuff that, frankly, we already did. Virtualization — yeah, VMware had a good run at that. But, they didn’t invent that. That was on the mainframe and, in fact, a lot of the stuff we’re talking about now is mainframe kinds of thoughts, frankly. I don’t think there’s anything new in IT. We just keep re-implementing stuff. You only need to look at the JavaScript ecosystem to see that.”
“I see some innovative things happen,” counters Dave. “I think the danger is confusing something that is innovative with something that is almost fad-like. We see that with programming languages. We see that with hardware solutions. We see it with services and businesses and products. That happens across all industries.”
“So, to under-cut what I just said,” says James, “certainly you can simplify stuff. Containers is not new, but Docker did a very good job of making it a simple experience on the Mac, and therefore enabled a lot of developers to feel really productive. Whether that actually solves the challenges of managing production environments and distributed systems is a very different question.”
“The idea behind microservices,” says Dave, “is to have each service be very discrete and specific about what it does, and not to mix or aggregate many different capabilities into a single service — so, very much a purpose-built service. The problem with that is, instead of being able to easily deal with the integration of things, you’ve loosely coupled everything, and loosely coupling everything leads to complexity.”
“Not only does it lead to complexity — and you now have to manage this network of things, which are dependent on one another — you can have circular dependencies, you can have unidirectional dependencies — but then, in addition to that, each one of those discrete services is versioned. And if you change it, it could affect five other things downstream. So you have this huge dependency graph now that you have to manage. Its really easy to manage the microservice code, because it only does this one little thing, which is great. But now, how do you deal with the ramifications of all the behaviors and all of those dependencies upon it? And that’s one of the real challenges once you actually start implementing microservices.”
Basho and New Relic are sponsors of The New Stack.
Feature Image: Post it notes by Jacqui Brown, licensed under CC GA 2.0.