EngFlow Harnesses Google Bazel for Faster Monorepo Builds
After Google open sourced Bazel, a version of its internal build system Blaze in 2015, it started taking off and has grown to a project with 20,000 starts on GitHub. Yet some of its creators realized that not every company is like Google and the project needed shepherding for the other 99%.
Thus the creation of EngFlow, a company focused on making builds faster, making no apologies to developers whose runs for coffee might have to be cut short.
“The more that we heard from different companies, the more it became clear that there needs to be this strong focus on building, leveraging these concepts, but outside of Google, because not every company looks like Google, not every company has the same challenges.
“And so while Google open sourced Bazel into the world, adopting Bazel and an building ecosystem of tools on top of Bazel needs to be done differently than the way Google has approached it internally,” said EngFlow CEO Helen Altshuler, who was working with Google customers adopting Bazel at the time. Her co-founder, Ulf Adams, was the lead developer of Bazel at Google.
Multilingual Build System Bazel
Bazel, as explained in this early post on The New Stack, speeds up builds using a massive shared-code repository or monorepo, though it also handles a microservices architecture. It optimizes building using both caching and parallelism. Version 6.0 was released in December.
Rather than requiring a different build system for each language — Maven for Java or CMake if you use C++, for instance — Bazel is multilingual. It speaks Java, C++, Go, Android, iOS, Ocaml and many other languages and runs on Windows, macOS and Linux. The tool itself uses Starlark, a declarative programming language derived from Python.
As codebases grow, build times don’t have to, according to Altschuler. With Bazel’s approach to dependency analysis and dependency graph, it figures out as you go along what has changed, so you only need to recompile and rerun tests on the parts of the code that changed. That leads more to efficient processing and faster builds. It offers consistent reproducibility, integration with Docker and Kubernetes, and scalability.
It has been adopted at the likes of Uber, Spotify and Dropbox. Twitter was in the midst of migrating its Pants build system to Bazel when Elon Musk took over, one of a number of open source projects at the social network apparently no longer supported.
Designed for large, engineering-heavy organizations, however, Bazel has been criticized for requiring extensive configuration, hence the need to provide more help.
Paralleled Builds
EngFlow wrote its own product based on Bazel, which will parallelize actions and distribute them on a cluster of compute nodes.
Using remote execution, EngFlow distributes builds and tests across a cluster of machines and remotely caches the results to make them faster. It can be fully managed or self-managed, in the cloud or on-premises.
“So instead of running your build locally, on a 32-core machine, or whatever a number of cores, you can submit the build to the cloud to hundreds. Many of our customers are running tens of thousands of cores in parallel to distribute a significant amount of build compilation times of source files and test execution,” Altshuler said.
A Remote Execution cluster consists of schedulers and workers. The scheduler instances are the “brain” of the cluster that coordinate and distribute work and perform maintenance. The worker instances are the storage and execution units of the cluster.
On Amazon Web Services (AWS) for instance, EngFlow’s Remote Execution software distributes build and test actions across hundreds or thousands of worker instances, which run on EC2 instances with mounted EBS volumes. It uses AWS Auto Scaling to scale the required compute capacity. Remote Execution combines with EngFlow’s Remote Caching solution to prevent work duplication. With all build/test artifacts cached on an organization’s local storage, teams can download previous artifacts rather than re-execute the actions themselves. The build/test artifacts are then synced to Amazon Simple Storage Service (Amazon S3) for persistent access.
The company has built out analytics and observability tools. It recently released its Invocation Analyzer, an open source tool to help users identify the cause of problems such as slowness. It provides recommendations such as where parallelization can provide greater efficiency to improve performance.
“Bazel profiles are a powerful resource for improving performance, but understanding them and extracting action items from them is hard. The Bazel Invocation Analyzer automates the process and gives concrete suggestions based on the data found in the profile,” said Sara Adams, EngFlow founding engineer and Bazel Invocation Analyzer lead.
And the company’s support extends beyond Bazel. It supports all clients that use the open source Remote Execution API, including BuildStream, Goma Server (Chromium), Pants, Please, Recc and Soong (Android Open Source Project.)
Heavy on Bazel Talent
Launched in early 2020, just before the COVID-19 shutdowns, EngFlow is an all-remote company, though team members recently met up in Iceland.
It recently announced an $18 million Series A, following a $3.7 million seed round in October 2021. Investor Martin Casado of Andreessen Horowitz noted at the time of the seed round that 70% of EngFlow employees came from the original Bazel project at Google or were contributors. It since has touted hiring more talent in the Bazel ecosystem including Jay Conrod, Corbin McNeely-Smith, Benjamin Peterson and Ola Rozenfeld.
Its customers include private search engine Brave, agricultural software vendor Blue River Technology and network proxy Envoy Mobile.
‘Build- Acceleration Company’
There are a range of alternatives to Bazel, and thus EngFlow, though many are language specific, such as GNU Make, Maven, Ninja Build and SCons.
For instance, Turborepo is a Bazel-inspired open source monorepo build tool for JavaScript. Angular team members and former Googlers built Narwhal, the company behind the open source Nx build system for JavaScript.
Mobile DevOps company Bitrise recently acquired Flare Build Systems in a bid to provide Bazel backend services to the mobile DevOps ecosystem.
While EngFlow continues to build out the Bazel ecosystem, it’s also looking beyond it. It’s describing itself as a “build-acceleration company.”
“We start with Bazel because that’s our team’s deepest expertise. We have multiple core engineers who built Bazel, and who are continuously active contributors to the open source project today. We also have contributors to Chromium and Goma project as well, which is the open source for Chromium,” Altshuler said.
“And we are starting to look at how we can expand these capabilities, and really continue to change the developer mindset that you don’t have to wait for your builds. Your build system should be horizontally scalable, just like your CI system is and just like many other parts of the full developer lifecycle.”
