Falco: Lock Down Kubernetes from the Kernel on Up
Want to have a really bad day? Make a security blunder that the whole world can see — Yes, we’re looking at you SolarWinds. No one wants that. That’s why even though Kubernetes isn’t the easiest system to lock down securely, you must do just that. Fortunately, programs like Falco can help.
The 2020 North America KubeCon+CloudNativeCon featured several lectures on Falco. Here are the essentials you need to know about this useful Kubernetes security program.
First, it may be new to you, but Falco actually dates back to 2016. It was created by security DevOps company Sysdig. Falco started life as an open source behavioral activity monitoring agent. Described as a mix between snort, ossec, and strace, Rather than using a signature-based security monitoring approach, which looks for specific exploits and vulnerabilities, Falco looks for suspicious system activities, which show an attacker may be at work.
It does this at the Linux kernel level. Specifically, Falco works by taking Linux system call information at runtime and rebuilding your kernel’s state in memory.
Armed with this data, it uses a simple filter language to create rules that can catch bad behavior in the act. Once detected Falco sends out the appropriate error message. For example, if mysqld is spawning a new process after startup, you get a report that there’s a SQL injection attack underway.
Want to quickly get a handle on these rules? Sysdig’s Cloud Native Security Hub provides best practices rules for many popular programs such as etcd, Kubernetes clusters, and Nginx.
That’s it. Falco doesn’t do collection, alerting, reporting, or remediation. Following the “do one thing well” Unix philosophy, Falco is an edge agent. It’s meant to work hand-in-glove with whichever systems you already use for reporting and monitoring.
To make this easier Falco is transitioning to an “API first” architecture. With this, you’ll be able to more easily use Falco with other programs such as the Prometheus monitoring system and the Envoy proxy service.
This technique has remained the same, but how it’s implemented has changed over time. When it began the Falco engine relied on the Falco Kernel module driver to deal with the kernel’s raw stream of system call information.
Since then a Falco extended Berkeley Packet Filter (eBPF) probe has been introduced for environments, such as Google Kubernetes Engine (GKE) where kernel modules aren’t trusted or allowed but eBPF programs are. The newest probe, pdig, runs in user space and is built on top of Linux’s process trace (ptrace). It is, as you might imagine since it runs in userspace, slow, but if you’re not permitted to use kernel modules or eBPF, it’s better than nothing.
Regardless of the probe, Falco serves as a last line of defense against new, unpatched vulnerabilities, insecure configurations, leaked or weak credentials, and insider attacks. Combined with the open source Clair, a container scanner that uses static analysis to seek out application container vulnerabilities, the pair makes a strong team.
Organizationally, Falco has changed too. In October 2018, Falco became the first Cloud Native Computing Foundation (CNCF) security Sandbox Project. In January 2020, it became the first CNCF incubation-level hosted project.
By opening Falco up to the community, Falco has become quite popular in development circles. It’s seen a 100% increase in commits year-over-year and now has more than 55 contributors, including many engineers from outside Sysdig.
Users like it too. Falco has been downloaded more than 20-million times since becoming a CNCF project. It’s also finding use in other companies’ security packages such as AWS Firelens‘s Elastic Container Service (ECS) log router; AWS CloudTrail logging tool; and Google Anthos, a Kubernetes cloud framework. Sysdig’s own commercial Falco monitoring system, Enterprise Falco, can deploy Falco on pretty much any platform that supports Kubernetes.
Sysdig believes that open-source security tools such as Falco are the way forward for Kubernetes security. It looks like they’re right. If you’re not using Falco on your Kubernetes clusters yet, it’s time to take a long, hard look at it. I think you’ll like what you see.
