Supply Chain Attacks and Cloud Native: What You Need to Know
The past couple of years have seen a rise in software supply chain attacks, with the most salient example being the Solarwinds attack. As production environments have gained multiple layers of protection and much of the attention of security teams, malicious actors have set their sights on “poisoning the well,” that is, targeting where applications are developed or their building block components. This is done under the (mostly correct) assumptions that dev environments are not scrutinized as closely as product environments, and that targeting the right components, for example, those that are very widely used can ultimately give the attack access to many production environments once it is deployed undetected.
What can attackers do once they’ve successfully penetrated a supply chain? Many things. The final objective of many observed attacks in the cloud in recent years has been cryptocurrency mining, which is harmful, if not dangerous. However, we’ve also seen attacks aimed at data theft, credential theft, denial-of-service attack infrastructure (bots), as well as attacks that can branch out to multiple targets stage by stage depending on what they find.
But what does this mean for cloud native environments? Are cloud native applications more susceptible to supply chain attacks? I’ll try to answer these questions, illustrating my points using examples from Aqua’s cyber research findings from attacks observed in the wild.
Threats to the Cloud Native Supply Chain
There are several characteristics of cloud native application development environments that make them a lucrative target for attackers looking to embed malicious code into the supply chain.
First, cloud native application development is characterized by the widespread use of open source components, often obtained from public registries. While many open source projects are well-governed and scrutinized under many eyes, some are not, and this allows malicious actors to masquerade as contributors. More sophisticated attackers can also embed malware in a way that’s extremely hard to detect using static code scanning. The higher up attackers can go in the supply chain, the better their chances of achieving widespread dissemination of their code, which is why the more advanced attackers look for widely used packages that might reappear in many different applications.
Additionally, container images, functions and packages are updated frequently using CI/CD (continuous integration/continuous delivery) pipelines, creating multiple opportunities for attackers to embed themselves into the process. Team Nautilus, Aqua’s cyber research team, has detected and analyzed attacks on CI SaaS environments that abused the CI process itself to gain access to cloud CPU time. From there, it’s a relatively short hop into the artifacts being built in those CI pipelines.
Observed Attack Vectors
Our cyber research on attacks in the wild that target containers and Kubernetes environments has shown varying levels of sophistication and evasion techniques. Starting from the relatively simple to the more advanced, we’re seeing:
- Placing malicious images in public registries, either for pulling them into compromised environments or in the hope someone will use them.
- Hiding malicious code inside benign, “vanilla” images in the hope that developers will use them not knowing what lurks inside them. We’ve also seen “typo-squatting” used in this regard, where a popular image name is deliberately misspelled — for example, “tesnorflow” instead of “tensorflow”.
- Evading static analysis of malware by using polymorphic, evolving malware that isn’t recognizable based on its signature, and calls in additional components after an image has been deployed and run as a container.
- Circumventing the controls in the CI/CD pipeline by building images directly on hosts.
- Abusing cloud-based CI tools, in which the attackers take advantage of free SaaS offerings to run cryptocurrency mining under the guise of building an application.
Figure 1: AVScan attack tree showing gradual branching and malware downloads (Source: Team Nautilus, Aqua)
At various stages of these attacks, the malicious actors would perform gradual, increasingly severe actions to attain their objectives and establish persistence in the environment — i.e., enable them to continue an attack beyond the specific container, host or even cluster.
Any Good News in This Scenario?
Thankfully, yes. There are inherent characteristics of cloud native applications that make them more resilient to attack and enable them to limit the damage of an attack:
- The high degree of automation in the pipeline allows developers to embed controls that would potentially detect or at least make it harder for supply chain attacks to succeed. (More about that in the next section).
- Due to the orchestrated and often ephemeral nature of cloud native workloads, it is extremely hard for the attacker to persist. When an attack like the Solarwinds attack happens and infects a Windows server somewhere, that Windows server might be running with the same permissions, same IP address and no downtime or upgrades for many months — sometimes even years. By contrast, a serverless function typically runs for seconds and a maximum of 15 minutes in most cloud environments, which would limit the potential damage.
- Observability in cloud native is vastly superior. Thanks to open source tools like Tracee, we can get very detailed real-time information about every process, every network connection, every event. It’s easier to detect anomalies and easier to isolate them at a process level, function level or container level.
Eliminating the risk of supply chain attacks is virtually impossible, but there are measures that DevOps and security teams can take to reduce that risk:
- Control access to public registries and open source components: Reduce the risk by limiting the number of people who can access public images and packages, and create a curated internal registry for base images. Implement a process to ensure that only trusted images are used.
- Digitally sign images or use other methods of maintaining dev-to-prod image integrity: Ensure that the images you’re deploying are the same ones you vetted and prevent the deployment of images outside the predefined pipeline — for example using Kubernetes admission controllers.
- Isolate your dev/staging environments: If your supply chain is successfully penetrated, attackers might try to jump into adjacent environments even without targeting production environments for things like admin credentials, sensitive IP or source code. Be sure to authenticate access and limit it to specific IP addresses (or VPN).
- Scan images for malware, using both static and dynamic analysis: While there’s some value in scanning images and functions for malware using static, signature- or pattern-based tools, the more sophisticated attacks now evade such techniques by phasing their attacks — embedding small, innocent-looking components in the image, then downloading and running additional malware only when the image is run as a container. To detect such malware, you need to use dynamic analysis and tracing methods to observe a container’s behavior in a secure sandbox.
- Use ingress/egress network controls and allow-listing: Many types of malware will require a network connection to the outside world, either for command and control, for downloading additional malware components or for exfiltrating sensitive data or cryptocurrency hashes.
- Monitor your runtime environment: If your supply chain is penetrated despite all of the above, this will ultimately show up as abnormal activity in your production environment.
Supply chain attacks are here to stay and will evolve, and organizations should adjust their security practices to detect, identify and mitigate them.