DevPod: Uber’s MonoRepo-Based Remote Development Platform
By 2017, Uber’s codebase was incredibly fragmented to the point where problems would bleed into library versions, build tools, dependency management and collaboration, and code sharing was deeply affected. So the company moved developers to a remote development environment, based on a monorepos run on a Kubernetes cluster.
Uber’s build-strategy team recommended migrating to a monorepo architecture to host Uber’s code. Their plans encompassed over 10 programming languages, over 4,000 services, over 500 web apps, more than nine build tools and more than six configuration tools distributed across thousands of repositories at the time, with trunk-based development and one single version per third-party library at that time.
Uber’s recent blog post details the company’s custom in-house remote development environments, Devpods, which run on Kubernetes clusters. These secure environments need zero set up, and they run in the cloud rather than on laptops. Build times have improved, and over 60% of engineers are using Devpods as of November 2022.
The new monorepo, a single centralized build platform (built on Google’s Bazel), brought better dependency management, consistent universal production library versions, easier support for a standard set of tools and processes, and improved visibility, collaboration and code sharing.
But it also brought challenges to the daily code edit-build-run development loop — development was taking longer, much longer, with cloning taking several hours in some cases.
Remote development challenges plus monorepo challenges led to the need for innovation.
Platform Engineering
The idea of Devpod is to allow developers to build in the cloud, using a faster machine, with all the tooling kept in a secure, controlled environment.
A Devpod is tailored specifically for an Uber developer’s needs and what Uber considers the “best possible developer experience.”
Devpods run as a container on a Kubernetes cluster, with the necessary tooling and compute resources to enable seamless development on Uber’s large-scale monorepo. Productivity is enhanced by Devpod’s improved performance and nearly zero setup. Security was improved by running the latest stable versions of each dependency.
Devpods are persistent, meaning engineers won’t lose their setup, files or code changes while working on different devices or collaborating with other engineers in a single environment. They use cloud resources — up to 48 cores and 96GB of RAM over laptop resources, which allows for faster Git, Build, and responsiveness. Devpods come with the latest version of IDEs configured with cached IDE index. Git configuration is optimized, and the Linux file system is used rather than the laptop file system, which provides a better performance environment.
Since Devpods run in the cloud computing environment, there’s less freezing/overheating, better battery life, and developers are able to freely switch between environments.
Other advantages include:
- Zero Setup and Maintenance: Devpod has all the tools and configurations needed to start working in any of the main monorepos right away. Tools are ready to go. Environments are provisioned on demand, IDEs are set configured, repositories are pre-cloned.
- Secure: Devpods are updated automatically with the latest tooling and security upgrades overnight. Since Kubernetes pods run in the cloud, and don’t have laptop battery constraints, it’s “trivial” to scan disk for malicious artifacts or activity during off hours. Vulnerable applications can be patched during non-work hours as well.
- Prioritizing Low Latency: Devpods are deployed globally with the goal of reducing latencies for developers and avoiding the “sluggish” feeling for UI operations.
User Experience
Developers are presented with a “flavor,” a Docker image preset with the tools and default settings and configurations for a specific group of engineers. The following flavors are supported:
Putting the tooling and configuration into the docker image “made perfect sense,” according to Uber engineers. They built the Devpod image base on top of the production image base to reuse their critical production software configuration. Uber uses Debian for production workloads. The consistent base allows for the current infrastructure to resume for the building and distributing of internal packages.
This web-based interface was created to provide a better first-time experience with Devpods and improve troubleshooting. The future goal is to extend Devpod Central and bring visibility to resource usage and consumption.
The following IDEs are available in Devpods: VS Code remote, VS Code over web, JetBrains IDE via ssh using JetBrains Gateway Vim and Emacs via terminal.
Kubernetes as a Platform
Moving from laptops to the cloud was a great start, and Kubernetes provided additional necessary building blocks. The ability to host containers on powerful hardware, networking to connect containers, and persistent volumes to store engineering work between restarts was also a necessary step.
Uber used custom resource definitions (CRD), a way of extending Kubernetes by providing a data structure stored inside the control plate, and removed the need for an internal database for Devpod metadata. It also allowed for the creation of and access to Devpods using kubectl.
Kubernetes’ CRDs also have the ability to write custom software that reacts to resource changes, called the operator pattern. Uber combined custom resources and small custom controller to translate a high-level Devpod description into standard Kubernetes primitives that can be handled by standard Kubernetes controllers.
The custom controller was later extended to react to VolumeSnapshotContents and PersistentColumeClaim in order to cut costs when Devpod isn’t used.
Progress So Far
Devpods are effective in reducing the local Git command performance. The chart below shows that git status times have been consistently below four seconds.
Devpods provides a 2.5x improvement over laptops on average for longer, more complex builds and 1.8x improvement over laptops for average builds.
As of November 2022, over 60% of Uber software engineers have adopted Devpods.
Devpods have a bright future at Uber. The goal now is to make Devpods completely seamless for Uber engineers. Some areas of exploration are reduction in the time it takes to set up Devpods, ephemeral Devpods and seamless file transfer between laptops and Devpods.