The New Stack @ Scale Podcast, Show 7: ClusterHQ and the Science of Scaling
Sandeepan Banerjee grew up in a small town in India that was visited once a week by a single plane. “We all gathered around to watch it and it seemed to be a very placid process,” he recalled. He recalls today how quaint that small town airport is compared to the gigantic operation of Chicago’s O’Hare, with its multiple terminals and thousands of daily departures and arrivals.
For Banerjee, who signed on to container volume storage manager ClusterHQ in January as senior vice president of engineering and operations, O’Hare represents the challenges in managing large-scale automated IT operations, and how they are fundamentally different than the single server in the wiring closet.
“How do you spin up and down thousands and hundreds of thousands of nodes? How do you attach and detach storage to that? How do you stay a good steward of your data if it is being spread across multiple disks in a cluster?” he asked.
In this new episode of The New Stack @ Scale podcast, Banerjee was grilled by The New Stack’s Alex Williams, and New Relic’s editor-in-chief Fredric Paul about scaling up and scaling out.
Banerjee knows all about scalability. “ClusterHQ are the people who do volume management for the new stack,” he explained. Before joining ClusterHQ, Banerjee was data head at Google, managing exabyte-class Google-wide storage systems, the company’s web search infrastructure and YouTube video processing. Prior to his time at Google, Banerjee spent 10 years at Oracle, including overseeing product management for Oracle’s core database server technology group.
“A lot of people fetishize scale-out where they think that scaling out is sort of a one-stop goal that you reach at the end of any journey and then that’s not the case. You could do it from very many different standpoints and each of these is a different workflow that might need a different kind of message,” Banerjee said. This interview provides a fascinating glimpse into some of the considerations IT architects must make when designing systems at scale.
These are lessons that every IT shop, not just the Googles of the world, should take to heart.
“I remember prior to Google, I spent 10 years at Oracle and for many years, we had a Terabyte Club and that had, like, five customers and most of them were huge telcos,” Banerjee said. “Now, it is possible for every consumer to sell out a terabyte disk on their home PC with videos.”
New Relic is a sponsor of The New Stack.