Specs: Dropbox’s Sixth Generation of Custom-Built Servers
File hosting service Dropbox released the details on its sixth generation of servers, a fleet optimized for a higher capacity, and lower latency.
Dropbox began running its own hardware infrastructure nearly a decade ago with the introduction of Magic Pocket, an in-house multi-exabyte storage system. Being one of the few exabyte-scale storage systems in the world, Dropbox found it to be of paramount importance to take on the responsibility of keeping the engineering in-house thus preventing any outside entity to prevent them from proving the best service they believe they can to their customers.
Architecting, shipping, and maintaining all the physical hardware in the Dropbox Data Centers is no small feat and an exciting milestone was just met when the sixth generation of hardware was shipped as described in great detail by Eric Shobe and Jared Mednik in their recent article.
The One vs. the Many
One approach available to designing server configurations is what the hardware industry calls snowflakes. A snowflake is a piece of hardware designed for each individual service. Shobe and Mednick describe this approach as a fancy sports car, optimized for speed, but doesn’t handle very well for trips such as dropping the kids off at school or picking up the dry cleaning. This car could work if one never has to make those trips. But since Dropbox has over 100 services, that would be a lot of fancy cars optimized for individual trips.
Keeping with the car analogy, the approach Dropbox took for these server configurations was more of an all-purpose sedan. It gets everyone where they need to go no matter how far or how close. This sedan though is a 2022 sedan and it’s still a little fancy. The solutions Dropbox came up with are general-purpose designs best suited for the increasing size and breadth of their service diversity.
There are three classifications for the server tiers for core applications and one tier for serving their Edge network.
- Storage: runs Magic Pocket (Dropbox’s immutable block storage system) and hosts all customer data
- Database: powers their persistent storage layer (think MySQL and metadata)
- Compute: provides general-purpose compute for applications
- Edge: connects customers to data centers through distributed POP sites
Meet the Sixth Generation Servers
There was much time spent discussing the server code names. The Dropbox team tries to stick to cartoon names, each with the first letter matching the first letter as its server tier. Edge is the exception as it’s considered a branch of the compute tier rather than its complete own tier. The sixth-generation hardware names are Scooby, Diego, Cartman, and Coco.
Scooby the Storage Tier
This layer was co-designed with the Magic-Pocket team. They offer a range of high-capacity 3.5” HDDs, allowing for scaling up to more than 2PB per server, and more than 20PB per rack. Each chassis contains more than 100 drives, a compute enclosure to run Magic Pocket software and a 100 Gb NIC for connectivity. Eight chassis fit in a rack offering the highest amount of performance per rack when compared to previous generations.
Diego the Database Tier
Rack density is effectively doubled in the persistent storage layer now. Each chassis has more than 60% CPU cores and 16 TB NVME SSDs which is 2x the amount of flash storage as generation five. Flash storage is key to providing very fast input/output and low latency access to metadata. Edgestore, File Journal, and Magic Pocket are a few of the services supported by this layer.
Cartman the Compute Tier
Each node is a low-end server, housed in a 1U enclosure, stacked 46 per rack, and interconnected with a top-of-rack switch. Cartman gives the compute tier a 3x improvement in speed and power — far beyond any prior generation. The leap in compute was mostly driven by multitenancy — meaning multiple applications sharing the same space — and CPU industry trends that gave Dropbox more performance across their fleet with fewer CPUs. Inside the box is a 48-core processor, 256 GB DDR4 RAM, a 1 TB SSD for boot, and a 25Gb NIC.
Coco the Edge Tier
Coco’s servers are responsible for accepting, encrypting, and decrying traffic in and out of Dropbox’s data centers and are the closest tier to customers. This sixth generation allowed Dropbox to scale up the rack density of POPs by 17%, increase the core count by 50%, increase network speeds by 25%, and triple their total memory. The end result is lower latency and improved user experience when connecting to a data center.
What’s Next?
Hardware engineers work closely with software teams to monitor how the hardware is performing. Co-designing hardware with software teams allows them to take advantage of the latest technologies and taking a custom in-house approach allows them to create a solution that is uniquely theirs.
Dropbox revisits server configurations at regular intervals to ensure they are making the best decisions and continuing to best service for the customer’s needs. As mentioned in the article, Dropbox prides itself on making innovative bets with new technologies such as partnering with AMD to launch their Naples and Rome processors and deploying SMR drives for Magic Pocket.
The seventh-generation is already taking shape though no release date was mentioned at this time. Two new technologies on deck for the seventh are machine learning and heat-assisted magnetic recording (HAMR) hard drives.
Hardware accelerators, new to the Dropbox fleet, offer huge performance gains when running inference and regression-type workloads and machine learning will allow Dropbox to fully take advantage of them. HAMR uses a superheated laser-focused on the platter while writing new bits into place. This will unlock new densities in data storage.
