Can eBPF Keep Your Kubernetes Cluster from Getting Hacked?
eBPF (Extended Berkeley Packet Filter) is receiving significant attention for security, monitoring, observability and other use cases. Its core feature is its ability to run within the kernel and extend across various stacks, including Kubernetes nodes and clusters, in a closed or sandbox environment. eBPF’s rapid speed and its extensibility across highly distributed Kubernetes environments make it especially well-suited for cloud native security tools and platforms.
However, a key thing to keep in mind is that eBPF itself is just the starting point. Tool and platform providers must develop tools that successfully take advantage of eBPF for Kubernetes. In short, there is a lot of work to be done before eBPF can be used to its full potential.
As Gartner notes, most enterprises lack the expertise and skills necessary to build and integrate eBPF-based functions. However, Gartner also recommends organizations: “Seek eBBF-based Kubernetes CNI [container network interface] solutions when scale, performance, visibility and security are top of mind.”
Specific eBPF use cases for security on Kubernetes (and other environments) include malware and intrusion detection, network packet routing and policy enforcement, resource usage monitoring, and other cluster-related uses. Associated tools and platforms can leverage their monitoring and tracing aspects to enhance security as a starting point and offer additional security features. With eBPF, they can provide actionable insights to detect threats and enforce security policies that security solutions can apply, especially in Kubernetes environments, which are gaining popularity. These actionable insights originate from hooks within the kernel but can extend to the entire stack across a network.
An additional layer of capabilities that security tools that rely on eBPF should provide are:
- Lowering the number of false positives.
- Lowering the volume of false positives.
- Better analysis and rankings of threats based on severity.
- Reducing the risk of breaking applications when security vulnerabilities and attacks are detected and the code is automatically changed to resolve the issue.
- Improving eBPF’s resource-hungry nature.
The open source community and commercial software tool providers are actively working on solving these issues. Meanwhile, let’s drill down on what is lacking as eBPF for Kubernetes security continues to evolve as a work in progress.
False Positives, False Negatives
eBPF’s security functionality lies in the way it runs within the kernel without altering the kernel code. It extends a program’s capabilities across numerous distributed environments, much like the way WebAssembly operates within a closed or sandboxed environment where the code remains inaccessible. Tools built on these programs can provide the ability originating from the kernel to detect or prevent attacks or vulnerable code anywhere within the network or computing environments.
However, this power comes with a drawback: It can generate numerous false positives. False positives have been a common issue in security tools and platforms that aim to offer all-encompassing security solutions. Security experts, even those with years of deep experience, will squander their time and energies if they are required to adopt a “whack-a-mole” approach to fixing vulnerabilities, improving the security of infrastructure and policy management and mitigating without an automated analysis and ranking of threats by severity.
The challenge is to put vulnerabilities, attacks and security issues into context and determine, in an automated manner, whether they require immediate remediation. Otherwise, an influx of false positives can lead to alert fatigue, which has caused organizations to overlook critical and damaging attacks and vulnerabilities.
At the same time, malicious hackers are continually exploring ways to configure malware to bypass eBPF security alerts. Their creation of scripts and orchestration of attacks designed to circumvent eBPF-based security alerts is a real concern. The security tools used to enhance monitoring and observability are not flawless. They may suffer from false negatives, often due to misconfigurations or limited visibility into various program layers within the kernel and different runtime environments. It’s important to understand that eBPF cannot be used to monitor all code. It is unable to access and open certain file types and libraries, an obvious source of false negatives.
To avoid such scenarios, comprehensive and well-documented guidelines will be essential to prevent misconfigurations and misunderstandings. Documentation and vendor support for eBPF-based solutions are as critical to robust security as eBPF’s technology itself.
More Precise Authorization and Access
DevOps or security engineers still often spend a significant amount of time manually investigating configurations when applying security tools with eBPF for Kubernetes. For instance, when a system indicates that a workload is configured as privileged, questions arise about its actual impact. Researchers and investigators often manually inspect these configurations to understand whether the use of privilege is proper or problematic.
eBPF lays a foundation to automate inspection and mitigation, but in some eBPF security tools, identifying a vulnerability and providing a fix after a scan can render an entire application unusable. Consider a straightforward misconfiguration: A container with an exposed ingress point offers an open door for attackers. The automated response is likely to shut down the ingress point entirely. This could disable a container-hosted application that relies on interconnectivity with microservices or system calls from outside a Kubernetes pod or clusters. A more precise approach would define permissions and implement authorization and authentication measures, but not all automation tools that run in eBPF are there yet.
Emerging Improvements
ARMO platform
The good news is that the shortcomings of eBPF have been well-recognized within the open source community. They are frequently discussed during conferences and similar events, leading to the emergence of solutions and workarounds.
One notable open source solution is CO-RE, “Compile Once, Run Everywhere,” which can minimize CPU and memory use by eBPF tools in distributed environments. CO-RE allows eBPF programs to run simultaneously across different kernels, so security tools can inspect and interact with various kernels without the need to access each kernel directly, a significant saving in resources. CO-RE is supported by multiple vendors and open source tools, including BumbleBee and Kubescape, which Kubernetes security provider ARMO has introduced. Meanwhile, while Kubescape and BumbleBee creators use CO-RE to distribute their tools, eBPF compilers and frameworks like eBPF-GO and eBPF bcc support vendors implementing CO-RE.
There are also emerging security solutions that use eBPF but provide remedies or workarounds for its weak points. A proper platform that uses eBPF should be able to allow DevOps teams to monitor what should be running in the Kubernetes cluster and offer actionable results when policies are violated or security threats are detected. Relying on CO-RE to optimize resources, ARMO’s enterprise version of Kubescape has been shown effective in a DevOps environment on Kubernetes. There it is used to make smarter automated decisions about security access to pods and clusters, based on its ability to both assess the potential impact of changes to an application and make more nuanced security fixes to code and policy changes. In our earlier ingress point example, Kubescape could pinpoint the specific folders that require write access and remove it from others, shrinking the attack surface rather than breaking the application.
In the case of Kubescape, the Cloud Native Computing Foundation sandbox project covers the life cycle of applications and their updates for Kubernetes applications. This includes IDEs, CI/CD pipelines and clusters for risk analysis, security, compliance, misconfiguration scanning and image scanning. Hardening recommendations such as network policies and security policies are also available as part of open source offerings.
Kubescape is used to integrate with a checklist of the necessary tools your DevOps teams use with the platform, such as software bill of materials (SBOM), signature scanning and policy controls. It begins running its scans at the very beginning of the development cycle and extends across CI/CD and throughout the deployment and cluster management process.
It is used to find and fix misconfigurations and vulnerabilities with frameworks such as NSA-CISA, MITRE ATT&CK and the CIS Benchmark. Kubescape scans YAML files, Helm charts and clusters upon deployment. Kubescape can also be integrated with Jenkins, CircleCI, GitHub Actions, GitLab , IDEs such as Visual Studio Code), Prometheus, Lens and Docker.
ARMO platform
A recent capability with eBPF that ARMO offers applies to vulnerabilities relevancy and prioritization. Relevancy and prioritization allow ARMO Platform and Kubescape users to deprioritize vulnerabilities that belong to unused software packages and components. By first deprioritizing vulnerabilities that are less critical, users can focus on addressing the ones that pose a greater threat to their running cluster.
According to ReveCom’s preliminary tests of ARMO Platform, the security vendor makes a strong case for its ability to address these challenges described above, and will provide more insights as its tests progress. Ideally, one day, organizations should be able to automate their security scans and protection for Kubernetes and other environments so they don’t even have to know that eBPF is running everywhere and can just rely on it to keep their organization out of trouble.
