Buoyant CEO on How Linkerd Sprang from Twitter’s Heady Early Days
This podcast was sponsored by Buoyant.
Buoyant CEO on Linkerd’s Origins During Twitter’s Heady Early Days
It wasn’t that long ago when deploying massive-scale projects to the cloud meant largely creating the underlying infrastructure to make the migration from scratch. This reinventing the wheel-like gargantuan task is what Twitter software engineers certainly had to do. But as they figured out how to scale the social media network from Twitter’s days as a fledgling startup to handle today’s 300 million regular users today, they also, almost by accident it seems, created the basis for the first service mesh Linkerd. As William Morgan, now CEO of Buoyant, the creator and primary sponsor of Linkerd, told Alex Williams, founder and editor-in-chief of The New Stack, during a recent podcast:
“And the most amazing thing about this migration was it actually worked,” Morgan said, referring to his Twitter days. “So, it was there I think, for a company to be like, ‘okay we’re rebuilding everything from scratch for that actually to succeed.’ But somehow, we managed to succeed.”
Linkerd, a Cloud Native Computing Foundation (CNCF) project, “introduced the notion of service mesh and that term to the world,” Morgan said. “It’s become very, very popular and now there is a lot of other projects that are kind of following the same path. But the origins of Linkerd are way back in the olden days of 2015, many, many eons ago in a kind of cloud-native computing [context], which is like when the dinosaurs were first scheduling containers by hand.”
The scope of what they had accomplished, as well as what it could offer the open source community, quickly dawned on Morgan and other Twitter engineers after creating the underlying cloud native infrastructure for Twitter as a platform. “Being on the inside, it was totally crazy because I had started to work with startups before but never one where the sheer volume of traffic was so high,” Morgan said. “And then there were the spikes, where you have these insane spikes in traffic from something that would happen after a world event that was totally unpredictable.”
Twitter’s infrastructure thus had to be “built out to handle what was really kind of cutting edge at that time and I think it still is,” Morgan said. “We combined that with this massive migration off of a monolith (we call it SOA because we didn’t know about the word microservices then), but now, we would recognize it as a big microservice deployment,” Morgan said.
After leaving Twitter, Morgan said he had “a moment of clarity.” “We were like, ‘gosh, that was kind of crazy.’ [We didn’t] think other companies should have to reinvent all of that stuff,” Morgan said. “So, that was kind of the genesis of Linkerd. And it was specifically around this pattern that Twitter had used to make the way that their microservices communicate, be reliable and be measurable and visible and be secure.”
Risk taking, of course, was also involved, such as the decision to opt for Rust as the underlying programming language. “Rust has gotten a lot of attention and the ecosystem was starting to grow, but you know, compared to something like Go and Java ecosystems certainly; it was very, very early. But what we liked about Rust.”
What has made Rust “such an interesting and kind of a compelling choice for proxies” is how it “allowed us to build the proxy layer of Linkerd in an incredibly fast native code way but with guaranteed memory safety,” Morgan said.
As an open source project, Morgan and his team, of course, were not expecting Linkerd to explode in popularity as it has done. “You never know what’s going to happen with an open-source project. We put it out there and all of a sudden, people are starting to adopt it and were like ‘holy crap, this thing actually makes sense,’” Morgan said. “And the thing that I think really helped us was the emergence of things like Docker and Kubernetes, which run at the same time, where all of a sudden this proxy model that the service mesh uses became operationally like a manageable concept.”
In this Edition:
1:54: The Linkerd story.
5:10: Did that become a load balancer question to some extent?
10:29: What came with Rust? What did you learn?
13:11: The internal developer experience at Linkerd.
17:26: The intersection between networking issues and Kubernetes acting as the orchestrator.
21:03: Exploring the freedom from dependencies that service mesh offers.
Feature image via Pixabay.
