In Pursuit of a Superior Server: Oxide Computer Ships Its First Rack
In Emeryville, California, on a quiet street half a mile from Ikea, Oxide Computer Company just achieved a major milestone.
The website of the four-year-old company promises the company is working toward “The power of the cloud in your data center… bringing hyperscaler agility to the mainstream enterprise.”
And on June 30, Oxide finally shipped its very first server rack.
In an email interview Friday, Oxide Chief Technology Officer Bryan Cantrill confirmed that “it arrived at its destination (no small feat when you’re 9 feet tall and 3000 pounds!) and has been successfully installed!”
And what was the reaction of lucky rack customer #1? “I think it’s fair to say that the customer has appreciated the transformationally different (and exponentially faster!) process of going from new rack to provisioned VMs.”
The same day Cantrill appeared as a guest on the Software Engineering Daily podcast. (“The crate, by the way? Its own engineering marvel! Because to ship a rack with the sleds, it’s been a huge amount of work from a huge number of folks…”) Cantrill wouldn’t identify the customer but said that “Fortunately when you solve a hard problem like this and you really broadcast that you intend to solve it… Customers present themselves and say, ‘Hey, we’ve been looking for — thank God someone is finally solving this problem’.”
On the podcast, Cantrill also admitted loving the “aesthetics” of its rack. “The rack, is — it’s beautiful… Servers are kind of like sooty and loud, and our server is beautifully quiet… ” Like Facebook, Oxide discovered that 80-millimeter fans move a lot of air — and “do it with a lot less energy.” Oxide even worked with their fan vendor Sanyo Denki to allow lower-power settings for just enough rotations per minute.
On the podcast, Cantrill remembered when the company’s engineers did their compliance testing, “The folks at the compliance lab — they see a lot of servers — and they’re like, ‘Are you sure it’s on?’ Because it’s so quiet!”
Cantrill argued that the acoustics of today’s data centers are “almost like an odor. It is this visceral reminder that this domain has suffered for lack of real systemic holistic thinking…”
Finally got my photo op with the @oxidecomputer rack. Even an empty EVT rack was super impressive, can’t wait to travel and see the full demo rack in CA soon! pic.twitter.com/S8xUIWDULj
— Kate Hicks (@mrskatehicks) September 14, 2022
Victory Lap
Launched in 2019, Oxide has had a fervent following rare in the world of server vendors. A tweet announcing the milestone soon attracted 2,302 likes. And congratulations poured in on other social media sites (including an acknowledgement from Rust creator Graydon Hoare).
Tough to express how proud I am of this team and what an achievement this shipping milestone is. The battles along the way were numerous and they make this victory that much sweeter. They’re just getting started.
Safe to say the computing world won’t be the same. @bcantrill… https://t.co/aGqZsPbwt7
— Seth Winterroth 🤖 (@Sethwinterroth) July 1, 2023
A July 3 episode of the “Oxide and Friends” podcast found the team feasting on a 7-pound package of mixed nuts sent in congratulations from one of their supporters. Co-founder and CEO Steve Tuck said that finally shipping the product felt a little surreal.
Cantrill added that was “in part because we’ve been working so hard for so long.”
And Oxide management noted triumphantly that one detractor who’d said they would eat their shoe if Oxide ever shipped anything had returned with a humbler counter-offer — to buy the team a case of beer.
It’s the culmination of years of work — to fulfill a long-standing dream. In December of 2019, Oxide co-founder Jess Frazelle had written a blog post remembering conversations over the year with people who’d been running their own workloads on-premises. “Everyone setting up infrastructure themselves is in a great deal of pain and they have been largely neglected by any existing vendor…
“Hyperscalers like Facebook, Google, and Microsoft have what I like to call ‘infrastructure privilege’ since they long ago decided they could build their own hardware and software to fulfill their needs better than commodity vendors. We are working to bring that same infrastructure privilege to everyone else!”
The former: basically all the hardware (above the ASIC level) is custom designed by Oxide, including a ton of high-spec PCB layout.
See https://t.co/SGeXvL1aIo
for a cool picture— Matt Keeter (@impraxical) July 1, 2023
Frazelle had seen a chance to make an impact with “better integration between the hardware and software stacks, better power distribution, and better density. It’s even better for the environment due to the energy consumption wins!”
And that same month, looking back over 15 years as a customer of server vendors, Oxide founding engineer Joshua Clulow pinpointed a key problem. “The software and the hardware are generally not co‑designed — the flexibility inherent in the off-the-shelf approach leads to a lot of incidental complexity…”
“I’m excited to be here at the start, with a mixture of new and familiar faces, as we take a swing at building an integrated software and hardware system to bring the benefits of hyperscale computing to server customers everywhere!”
9 Startups in 1
Software Engineering Daily asked Cantrill to name the three things he was most proud of. But Cantrill insisted it’s more complex than that. “The extant server ecosystem infrastructure is so ossified that you can’t just take a little bit of it. You kind of have to take the whole thing, which is what we’ve done. But when you take the whole thing, it’s not one innovation. There are so many different ones.”
Cantrill added that in the end, shipping a new kind of rack also involved developing things like their own networking switch and their own power shelf controller. “We joke that we’re nine startups within one startup.” But this also meant “we can truly integrate this stuff together, and solve some of these really thorny problems, and deliver to the end user a turnkey experience.”
There does seem to be one thing Cantrill is especially proud of. In 2019 Cantrill gave a talk titled “I have come to bury the BIOS, not to open it,” and on the podcast noted proudly that Oxide’s system isn’t using the traditional BIOS firmware from American Megatrends Incorporated.
“This is a BIOS manufacturer from the 1980s that has somehow remained at the brainstem of server-side computing,” Cantrill said later on the Changelog podcast. “And right now x86 parts, be it Intel or AMD, have got AMI-authored code — proprietary AMI code — that you can’t see, change or operate, that is part of that machine bring-up, that platform enablement. And it’s a huge problem…”
“First of all, it’s just bad. I mean, it’s just poorly written; it is at the very lowest layer of the stack. It has got no idea what’s running above it, and so it will kind of hijack the machine to its own purposes… And that’s not the way you build a reliable system… Again, that is antithetical to this idea of unifying, of taking this hardware/software co-design approach.” But there’s not even a UEFI in Oxide’s system, Cantrill says on Changelog. “We do not need to have these layers that allow arbitrary other layers of software to run on top of them.”
And on the podcast for Software Engineering Daily, Cantrill pointed out that even the hardware-managing baseboard management controller (or BMC) was replaced with “a proper service processor. It runs an operating system that we developed, an all-Rust operating system called Hubris, appropriately enough.” Cantrill emphasized that Oxide eliminates the BIOS altogether. “The first instruction, after the AMD Platform Security Processor executes, is our operating system as it pulls up the rest of the system.”
Cantrill admits it was a “steeper” path to eliminate the BIOS — but it was ultimately also faster, because “we controlled our own fate… That has been the decision that I have been grateful for many, many, many times over.”
And then Cantrill described the result. “These things boot like a rocket.”
An Evolution
In December of 2019, the Oxide blog acknowledged its financiers “led by Eclipse Ventures and joined by an incredible group of institutional and angel investors…”
“I guess when you’re doing something bold, you will self-select for VCs that themselves are bold,” Cantrill said on the Changelog podcast. Cantrill added proudly that they were funded by Eclipse’s Pierre Lamond, “who hired Andy Grove at Fairchild Semiconductor.” Now 92 years old, “Pierre is certainly the oldest venture capitalist in Silicon Valley, and he is absolutely the boldest.”
On the Software Engineering Daily podcast, Cantrill confirmed that the company also took investment from their first customers who trusted their vision. Cantrill succinctly summarized one value proposition. “If you’re going to use a lot of compute, you actually don’t want to rent it — you want to own it.”
And it’s all open source. “Everything we do is out there for people to see and understand.”
On the Changelog podcast, the conversation turned to whether Oxide’s computer itself is a revolution on the scale of open source development and distributed version control. Cantrill’s response? “What we are doing, I would say, is bringing those revolutions to the computer itself. I mean, it’s kind of embarrassing that if you look at the server-side computer that is commercially available, it hasn’t really evolved since the 1990s. It is basically a personal computer! It is a racked personal computer… And the approach that we’re taking is true rack-scale design.”
Cantrill sees real problems now in the proprietary firmware that sits between hardware and systems software. “I don’t necessarily view it as a revolution in its own right, so much as it is bringing the open source revolution to firmware.”
“There is much that is now possible,” Cantrill said in our email interview Friday, “given the holistic foundation that we have created for ourselves.” And those possibilities even start to create their own momentum. “A blessing about creating a product like this is the excitement that it generates in one’s own customers: they have many ideas for the kinds of things that they’d like to see us build — and we’re excited to build it for them!”
So what’s the mood now over at Oxide Computer Company? Cantrill confirms that, as expected, “the team is excited.
“It’s very rewarding to be able to ship the result of many years of hard work!”