Don’t Rely on eBPF Alone for Kubernetes
eBPF (Extended Berkeley Packet Filter) has garnered significant attention for security, monitoring, observability and other use cases. Indeed, its capability to extend its hooks for security purposes and observability arguably makes it the best cornerstone option for these purposes for Kubernetes (more about that below). This is especially the case when using BPF with a CNI or single API.
But like a Formula 1 car that requires a lot of know-how to drive and the proper tools to get it race-ready, it is not recommended to use eBPF without a proper service provider and third-party tool set. In other words, it is not advisable to use eBPF alone for production environments that you might build into your infrastructure.
Indeed, most enterprises lack the expertise and skills necessary to build and integrate eBPF-based functions, according to Gartner. However, Gartner also recommends organizations: “Seek eBPF-based Kubernetes CNI [container network interface] solutions when scale, performance, visibility and security are top of mind.”
“Gartner has a point here,” CTO and co-founder Benyamin Hirschberg of Kubernetes security provider ARMO said. In practice, eBPF-based packet routing in a Kubernetes cluster proved to be more effective than the standard Linux packet routing which is very convoluted in Kubernetes and containerized environments. For example, Cilium adds Kubernetes CNI-specific packet routing logic via eBPF that bypasses routing layers which makes it efficient, Hirschberg said.
“eBPF is not an end-user-facing technology and it was never designed as such. Its interface was meant for kernel developers,” Thomas Graf, CTO and co-founder at Isovalent and founder of Cilium, said. “While it does lower the entry bar somewhat and while it does add verification of programs which make it a safer option compared to writing raw kernel source code, the technology remains ultimately targeting kernel developers and highly advanced systems engineers. eBPF is like an Android SDK: it gives the ability to develop and run sandboxed apps on a wide variety of devices but most end-users will never develop their own apps even though they technically could.”
eBPF’s core feature is its ability to run within the kernel and extend across various stacks, including runtimes inside Kubernetes nodes and clusters, in a closed or sandbox environment. eBPF’s extensibility across highly distributed Kubernetes environments makes it especially well-suited for cloud native security tools and platforms. But properly using eBPF for security — as well as for monitoring and observability — requires proper tools and platforms. In other words, it can be fun testing eBPF’s capabilities in a lab environment but is highly advisable not to bet your organization’s security on your in-house creation or an untested tool.
However, like Kubernetes, eBPF is a foundational technology layer, Torsten Volk, an analyst for Enterprise Management Associates, said. eBPF comes with the potential to secure rapidly changing distributed environments by operating at the Linux kernel level. “But with great power comes great responsibility, meaning that if you roll your own eBPF platforms and make a configuration mistake or don’t respond promptly to freshly discovered security vulnerabilities, you may have just granted malicious actors free admission to not just one VM, but to all your distributed application clusters. Or maybe one of your own, non-malicious developers, runs new code via eBPF that slows down your Kubernetes clusters to a crawl,” Volk said. “Therefore, I would encourage the use of existing and well-maintained and supported eBPF-based tools, instead of adopting a do-it-yourself approach.”
A number of eBPF security tools exist. Many, if not most, of these can overwhelm Security and DevOps teams with too many “false positives” or with little guidance of how severe or attention-worthy vulnerabilities might or might not be. This can be the case when relying on a number of public images from cloud native resources such as Nginx, Elasticsearch, Kafka or others, Volk said.
There are a large number of commercial tools for Kubernetes security posture management, including Google’s GKE security posture dashboard, and Tigera’s multicluster multicloud container networking platform, Volk said. However, there is still space in the market for an open source project that bakes security straight into the Kubernetes cluster API, he said.
While scanners of some tools can certainly identify most known vulnerabilities, the information is not worth much if you have hundreds if not thousands of alerts in a CVE file since a vulnerability in an image does not mean that it can be exploited.
@armosec's @slashben81 at @KubeCon_ 2023 Chicago: "Forget Everything You Know About Image Vulnerability and Prioritization" Don't waste resources fixing non-critical vulnerabilities so how about knowing which ones instead you must fix now? @CNCF @thenewstack pic.twitter.com/dKuHqn9irH
— BC Gain (@bcamerongain) November 8, 2023
During a talk at KubeCon + CloudNativeCon in Chicago this week, ARMO’s Hirschberg described how images are the source of many vulnerabilities in Kubernetes installations but how they vary in severity. He shared research from ARMO showing that a massive number of vulnerabilities just aren’t loaded into memory, and do not pose an immediate risk to the installation. He then described the different scoring systems in place that open source KubeScape provides that characterize vulnerabilities in terms of their severity and threat level. This allows the security team to address those vulnerabilities that pose immediate threats to the environment and to de-prioritize those more benign and often “false-positive” vulnerabilities that can otherwise serve as resource-draining distractions. Hirschberg explained how ARMO’s reachability feature provides results that are consistent with the Exploit Prediction Scoring System (EPSS). He also showed how ARMO relies on eBPF to inform the “reachability” feature so users can prioritize and rank vulnerabilities based on the degree of threat they pose and thus allow them to prioritize which ones are the most urgent to fix.
During his talk, Hirschberg showed how it is possible to automate the prioritization of vulnerabilities so instead of not knowing which ones are the most urgent to fix out of say tens of thousands you might only have to fix a few dozen in the immediate term. Reachability is a “way to describe the effects of vulnerabilities and give them a score for so you know which actions to take,” Hirschberg said.
Indeed, eBPF should improve the problem of “noise vs. signal,” Graf said. “Security tools tend to demo well in noise-free environments but the real world is full of nuances leading to false positives. eBPF is incredible at providing security observability with an amount of context that was hardly imaginable just a few years ago,” Graf, said. “While that doesn’t magically remove the false positives, it does improve the signal quality with more context and as we leverage the additional context with better pattern accuracy, the signal-to-noise ratio will improve.”
A reduction in security noise levels and false positives is a natural benefit of eBPF’s initial use case for performance benchmarking and troubleshooting with tools like bcc and bpftrace, Graf said. The use of these tools poses the same challenge of requiring more context to derive accurate performance bottleneck pinpointing, he said.
A proper platform that uses eBPF should be able to allow DevOps teams to monitor what should be running in the Kubernetes cluster and offer actionable results when policies are violated or security threats are detected.
A number of drawbacks have hindered the adoption of using scanners for Kubernetes security, Hirschberg said. “They include an almost endless list of misconfigurations and CVEs, that have led to alert fatigue due to an often exorbitant number of false positives such tools can create,” Hirschberg said.
However, by applying insights eBPF provides in runtime and then applying that context back to the scanners more precision in the prioritization of security scanner results, Hirschberg said. With it, DevOps teams can know what needs to be fixed in order to more positively impact their security posture. “In addition, risk acceptance is based on more facts, rather than gut feelings,” Hirschberg said. “This in itself cultivates a culture of better cooperation between DevOps teams and security teams.”
The evolution of eBPF for Kubernetes security is also manifested in how tools and platforms are better aligned with cloud native needs. “The Kubernetes and cloud native market has evolved so rapidly, that the initial tools developed, primarily focused on early adopters looking to ‘build, customize and end’ are not well suited for the new market reality of ‘install and use.’ It’s early days with dozens of Kubernetes distributions, installers and tons of overlapping projects very much reminded me of the days of countless Linux distributions and everybody compiling their own Linux kernel,” Graf said. “We have transitioned to a community that is looking to consume open source solutions instead of just open source code. eBPF doesn’t magically solve this — it provides amazing capabilities to build solutions that can but is up to the cloud native vendors and community to evolve the technology and its potential into capable solutions.”