Catch Performance Regressions in eBPF with Rust: Intro
This is the first of a five-part series. Read Part 2 here.
Extended Berkeley Packet Filter (eBPF) is useful for extending the functionality of the Linux kernel without the hassle of having to maintain a kernel module. At a high level, eBPF is a virtual machine within the Linux kernel that runs a special kind of bytecode. To create an eBPF program, source code in languages such as C, C++ and Rust are compiled to eBPF bytecode. This bytecode is then loaded into the kernel. The kernel then checks the bytecode with the eBPF verifier.
The eBPF verifier evaluates eBPF bytecode; it basically has to solve the halting problem with it. This is a very important step. Verification ensures that an eBPF program does not brick the kernel. As a result, eBPF has some limitations on the operations it is allowed to perform. All programs are limited to one million instructions; there are no unbounded loops, and there’s no way to wait for a user-space event inside of eBPF.
Once eBPF bytecode has been verified, it can finally be loaded into the eBPF virtual machine and run within the kernel. The eBPF program can perform one of several different tasks within the kernel: trace syscalls, probe user space, probe kernel functions, instrument Linux Security Modules (LSM) and filter packets, the last of which was the original use case. At the time it was simply Berkeley Packet Filtering (BPF). As new use cases were added over time, it became known as extended BPF. Now with so many possible applications, the initialism has been dropped in favor of eBPF, now just a jumble of letters meaning eBPF.
There are several different languages and toolsets to work with eBPF. A foundational tool for doing so is libbpf, which is written in C and is developed within the Linux kernel source tree under tools/lib/bpf. It is the standard-bearer for how to work with eBPF. However, libbpf is rather low level, so additional tooling has been added to help make writing both eBPF programs and their corresponding user-space programs easier.
A tool called bcc allows one to write eBPF programs in C and user-space programs in Python and lua. There is also ebpf-go, which allows one to write eBPF programs in C and user-space programs in Go. Finally, there is the Rust eBPF ecosystem. libbpf-rs is the official Rust wrapper for libbpf. However, libbpf-rs still requires the eBPF programs to be written in C. In order to write the eBPF program in Rust, a tool called RedBPF was created. This has since been superseded by Aya. Aya completely removes the dependency on libbpf in exchange for a pure, native Rust implementation.
| Library | Userspace | eBPF | Syscalls |
| libbpf | 🪤C | 🪤C | 🪤C |
| bcc | 🐍Python + lua | 🪤C | 🪤C |
| ebpf-go | 🕳️Go | 🪤C | 🪤C |
| libbpf-rs | 🦀Rust | 🪤C | 🪤C |
| RedBPF | 🦀Rust | 🦀Rust | 🪤C |
| Aya | 🦀Rust | 🦀Rust | 🦀Rust |
We’re going to be working in Rust, a modern programming language with a focus on performance, reliability and productivity. This makes it a great language to do systems programming, which caused it to be added recently as the first new language in the Linux kernel next to C. To write both eBPF and userspace programs in Rust, we will be using the Aya toolset throughout the rest of this series.
When writing eBPF programs, performance is of paramount importance. Since eBPF programs are run in the kernel, if they are slow, it can bog down the entire system. A single invocation of an eBPF program can add up to 100 milliseconds of latency to a call. This level of performance regression could be detected in development. However, this rarely happens unless developers are already on the lookout for them. Most development teams don’t have the infrastructure in place to detect performance regression in CI, like they do for feature regressions. This leaves performance bugs to be detected in production, where they are already affecting users, and they are the costliest to fix.
Performance bugs are bugs, and development teams should try to shift the detection of performance regressions as far left in the development cycle as possible. Relying on developers to manually run benchmarks on every change is a non-starter. For the same reasons that unit tests are run in CI to prevent feature regressions, benchmarks should also be run in CI to prevent performance regressions. This will require a continuous benchmarking tool, such as Bencher to track benchmarks and catch performance regression.
In this series of blog posts, we will cover:
- Writing a basic eBPF program in Rust
- Evolving an eBPF program in Rust
- Benchmarking an eBPF program in Rust
- Continuous benchmarking an eBPF program in Rust
All of the source code for the project is open source and is available on GitHub.
