Favorite Social Media Timesink
When you take a break from work, where are you going?
Video clips on TikTok/YouTube
X, Bluesky, Mastodon et al...
Web surfing
I do not get distracted by petty amusements
eBPF / Observability / WebAssembly

eBPF: Meaner Hooks, More WebAssembly and Observability Due

While most enterprises lack the expertise to directly utilize eBPF and should opt for tools configured with eBPF and extended layers of functionality, help is on the way this year.
Feb 9th, 2024 3:00am by
Featued image for: eBPF: Meaner Hooks, More WebAssembly and Observability Due

eBPF has indeed lived up to its hype in 2023, and in 2024, we will see more interesting developments. This is because eBPF has demonstrated its applicability as it sees continued adoption and thanks largely to the hard work of the open source community for monitoring, observability, networking and security.

It all started with the Berkeley Packet Filter merging with the Linux kernel in 2010 before it became “extended” and called eBPF in 2014. Today, eBPF has shown how the information it can provide in the form of logs, metrics, traces and other information through its hooks. These hooks originate from the Linux kernel that underlies applications, infrastructure tools, CI/CD and deployment for the developer, the operations team, and the SRE.

That said, challenges remain for it to reach its full potential.

Let’s look at what’s in store for 2024.

More Hard Work

There were numerous developments last year, or in 2023, concerning open source work. The highlights included the graduation of Kubernetes and cloud native-oriented eBPF projects from CNCF, such as Cilium. Furthermore, there has been ongoing applicability and adoption of eBPF through tools like Kubescape, Inspektor Gadget, Hubble, Tetragon and Falco. Additionally, as indicated by Gartner, it is noteworthy that these open source developments leverage eBPF, and these tools are instrumentalized to meet business and security observability needs for organizations.

Gartner advises that most enterprises lack the expertise to directly utilize eBPF and should opt for tools configured with eBPF and extended layers of functionality.

“While it is realistic for technology vendors and hyperscalers, most enterprises lack the expertise and skills necessary to build and integrate eBPF-based functions,” Gartner analysts Tony Harvey and Jason Donham write in “Hype Cycle for Compute, 2023.”

In other words, playing around with eBPF can be fun in a sandbox environment but don’t bet your organization’s security policy without trusting an established and proven tool and process for eBPF.

Meanwhile, challenges will arise, as solutions will depend on how open source projects that primarily support the evolution of eBPF are refined, and delivered.

“Currently, eBPF is mostly a ‘passive’ technology that listens to relevant systems data coming directly from the OS Kernel. This passive character, for example, limits how much auto-instrumentation eBPF-based observability platforms can offer,” Torsten Volk, an analyst for Enterprise Management Associates (EMA), said. “If eBPF allowed these observability platforms to pass context data through system calls and network requests auto-instrumentation could become a lot more turnkey, as this data could be used to connect the dots between the Kernel and the application.”

This auto-instrumentation could be automatically compiled into the application code, without requiring any code changes by the application developer team, Volk said. “However, while I would describe this level of auto-instrumentation as the ‘holy grail of observability’, allowing applications to make changes to systems data at the kernel level could become a very sticky issue due to the potential impact of these changes on system security, stability and overall performance,” Volk said. “On the flipside, auto-instrumentation at the Kernel-level could even enhance app performance, as eBPF code is compiled and runs faster than interpreted code.’

Organizations Will Wise up about Security ‘Relevancy’

eBPF largely owes its unique ability not only to providing observability for vulnerability and attack detection but also to identifying and fixing vulnerabilities. Additionally, it plays a crucial role in distinguishing between and providing context for vulnerabilities that can be discerned from attacks requiring immediate remediation or minor misconfigurations in code occurring during CI/CD, for example.

As Liz Rice, chief open source officer for Isovalent, writes in “Learning eBPF: Programming the Linux Kernel for Enhanced Observability, Networking, and Security”: “The difference between a security tool and an observability tool that reports events lies in the security tool’s need to differentiate between events expected under normal circumstances and events suggesting potential malicious activity.” Rice emphasizes that eBPF serves as the core technology enabling the creation of tools to “build on the concept of event detection, resulting in eBPF-based security tools capable of detecting or even preventing malicious activity.” In this way, Liz concludes that eBPF makes security distinct from other forms of observability.

More Integration

We anticipate more integration with eBPF tools and projects. For instance, Cilium, created by Isovalent, will undergo extensive work integrated with various tools and processes. Examples include the extension of Border Gateway Protocol (BGP) features in Cilium, allowing Kubernetes workloads to connect seamlessly with traditional services/workloads and integration of security event reporting from Tetragon with SIEMs, to give security teams detailed forensics to investigate malicious events, Rice told The New Stack.

Additionally, there will be increased integration with other technologies, such as WebAssembly, which, while not as steadily adopted as eBPF, holds promise. This integration will leverage WebAssembly’s capabilities to distribute applications in a tunnel connecting endpoints, with eBPF aiding in maintaining closed loops.

“We can think of WebAssembly as a userspace sandbox and eBPF as a kernel sandbox, each allowing custom processing in their respective spheres,” Rice said. “So, it would be reasonable to anticipate infrastructure tools that might combine functionality in both arenas to complement each other.  An example of this integration includes custom advanced L7 processing in Wasm Envoy plugins, combined with Cilium networking implemented in eBPF, “to create advanced and dynamic network functionality to meet an organization’s bespoke needs,” Rice said.

It could eventually be possible to compile eBPF code to WebAssembly, automatically injecting application containers with observability, and possibly auto-instrumentation, “no matter where they run,” Volk said. “The integration between eBPF and WebAssembly could be exciting indeed,” Volk said. “From a security perspective, as eBPF runs in the kernel, while WebAssembly runs in the user space of the Linux OS, combining the two could provide enhanced isolation for the complete app stack.”

The AI Connection

Certainly, AI will undoubtedly have a profound effect not only on society in the coming months and years. It will be intriguing to observe how AI is applied or used in conjunction with eBPF. This remains a very general prediction because the actual applications are not known. Speculating on how eBPF will manifest or how AI will integrate with eBPF would be purely speculative.

Simultaneously, it will be interesting to see how networking, in particular, can be utilized in conjunction with eBPF, rather than relying solely on LLM for networking security with say ChatGPT, which is not recommended. Observing how this dynamic unfolds will be interesting in 2024. “One example I would anticipate is network policies created by AI, and enforced by Cilium,” Rice said. “We’ve seen a little bit of experimentation in this area which, like a lot of ChatGPT applications, isn’t yet reliable enough to depend on, but I expect this will improve.”

Also, using eBPF to add Kernel-level data to the current stream of telemetry data coming from the user space of the OS will provide important context for LLMs to make better decisions and recommend more specific remediative action to human security engineers, Volk said. “Taking this a little further, the integration of eBPF and LLMs could allow the LLM to implement and evaluate security policies based on their impact at the system level,” Volk said. “This is where it gets really technical.”

Hacked in eBPF

eBPF is designed to offer customization across the network, extending from the kernel or across runtimes, especially for Kubernetes. However, since eBPF is integrated with the Linux Kernel, this might raise security concerns for some. After all, having malicious code with direct access to the operating system and the CPU is something no one wants except for attackers.

To address this eBPF security concern, the eBPF Verifier checks the code and only grants eBPF write privileges while also verifying that the program is licensed under GPL. Of course, nothing is entirely foolproof. As Rice notes, the verifier checks that the program is safe to run, but it can’t guarantee that the program isn’t malicious. “For example, I might write an eBPF program that drops packets from a particular address because it’s a source of malicious traffic or I might write a program to drop packets from a particular address because I’m a hacker who for some nefarious reason wants to block that address,” Rice said. “The verifier can’t tell the difference.”

Not only is it a good idea to not customize eBPF tools in-house as mentioned above, but it is also essential to rely on properly vetted providers. “For this reason, it’s important to only load and run eBPF programs from suppliers you trust,” Rice said. There is ongoing work in the kernel to help users verify the provenance of eBPF programs (much like supply chain security checks for applications).

I’m in this for the fun actually when it leads to the furtherance of good. I thus predict and hope that a hack will be revealed in 2024, not just for fun but in furtherance of locking down eBPF for networking encryption.

But, as Riced noted: “eBPF can do lots to customize networking functions but it’s more typical to lean on other in-kernel encryption implementations (such as Wireguard) rather than doing encryption in a custom eBPF program.”

Also, by definition, when you increase the ability of the application layer to interact with the kernel layer, the system’s attack surface is also increased, Volk noted. “This makes intuitive sense, as it was also the reason for keeping the kernel separate from the application when creating Linux in the first place,” Volk said. “However, it is likely that after so many years have passed since the initial creation of Linux, we now have ways of allowing conditional basic Kernel access to the application without risking the farm.”

Group Created with Sketch.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.