Containers Can Be Both Leaky and Risky — But That’s Always Been True
Synopsys sponsored this post.
Cue the leaky container jokes — security startup Prevasio posted a report earlier this month that should give you plenty of material. The company analyzed about 4 million public Docker container images hosted on Docker Hub and found that more than half (51%) had critical vulnerabilities. It also found 6,432 images that included malicious or potentially harmful elements.
That’s a lot of holes in a lot of containers.
Which isn’t funny, of course. It sounds serious. Prevasio said in a press release that its findings show that “Docker containers present a potentially serious risk to enterprise customers implementing container technology without adequate security protocols in place.”
“Docker Hub is not as resilient to attacks as previously thought,” said Prevasio CEO Rony Moshkovich.
Well, yes and no. Other container experts say it’s neither as serious nor as shocking as it sounds. They agree that there are many vulnerabilities in Docker container images, but that it’s been common knowledge for years and that the results of this report need more context.
David Belcher, associate principal consultant at Synopsys’ Software Integrity Group, called the report “a sensationalist view of a well-known problem.”
“It is a very real and valid problem, so maybe selling it in a sensational way isn’t all that bad,” he said, but then noted the obvious — that Docker Hub is an open platform. “Anyone can submit projects there, and old projects no longer maintained can sit and fester there for years. Using container images from public sources has always been, and will always be, risky.”
Adrian Mouat, chief scientist at cloud native service provider Container Solutions and author of the 2015 book “Using Docker,” agrees. He said the lack of context makes the raw findings “alarmist.”
“If I build and push an image today using completely up-to-date software, it will hopefully have no vulnerabilities,” he said. “But new vulnerabilities are being found all the time, so in a few weeks or months, this is likely to change. In the meantime, I may have stopped using the image because I’ve created a newer one or because I no longer need it. But it’s still there on the Hub, contributing to the 51%.”
Mouat also said that other analysts have found similar results, even though they used different analysis tooling. “This report focuses on ‘dynamic’ analysis, but it doesn’t look like the outcome was vastly different,” he said.
The Prevasio report said its method is indeed different, and more complete. It said it conducted the “first and only comprehensive security scan of the entire Docker Hub” with its own proprietary analyzer that “‘detonated’ each container inside of a dedicated virtual environment, capturing its behavior.” The system, Prevasio continued, “was executed nonstop for one month on 800 machines running in parallel.”
This kind of scan, done in a “dynamic analysis sandbox,” allowed the company to capture the behavior of a container image in a way that’s “out of reach for any static image scanner,” according to the Prevasio report.
Chandu Ketkar, principal security consultant at chip design firm Synopsys’ Software Integrity Group, agrees that a dynamic scan can be superior to a static scan. “It can detect images with dynamic payloads, where the original image on the face of it appears to be malware-free. But it is scripted, at runtime, to download malware and execute it. Such issues are more difficult to detect statically,” he said.
He added that “a malware-infected container image can proliferate a lot faster since the transmission mechanism, via trusted registries, is already baked in.”
The report is obviously intended to raise a cautionary flag about one of the most popular container platforms in the industry. Prevasio’s report notes that Docker Hub, first released in 2013, has made building and running containers much easier “by offering one common industry standard.”
“From the developer’s perspective, the biggest benefit that a container provides is a concept of ‘build once, deploy anywhere.’ A Docker container is a standard form of packaging software when all the software along with its dependencies is packed into a lightweight, standalone, executable form,” the report continues.
Docker Hub repositories, it adds, “allow the development community to push (upload) and pull (download) container images.”
But as Prevasio reported, many of those images have vulnerabilities and may contain malware. The specifics are:
- 51% of all containers had critical vulnerabilities, 13% of vulnerabilities were classified as high, and 4% as moderate.
- 6,432 containers were infected with cryptominers, hacking tools/pen-testing frameworks, and backdoor trojans. “While many cryptominers and hacking tools may not be malicious per se, they present a potentially unwanted issue to an enterprise.”
- There were more than 400 examples (with nearly 600,000 pulls) of “weaponized Windows malware crossing over into the world of Linux. This crossover is directly due to the proliferation of cross-platform code (e.g., GoLang, .NET Core, and PowerShell Core).”
Moshkovich said that Prevasio “had several calls with Docker in early September about our planned scan. We also disclosed to Docker the research results for comments 24 hours before we went public, but we didn’t receive feedback on the findings.”
Michelle Lazzar, vice president of communications at Docker, said the company “recommends that users only use known/trusted content,” and added that Docker “has built vulnerability scanning into our tools to make sure developers can detect issues early in the application development process.”
But Sergei Shevchenko, co-founder and Chief Technolgy Officer at Prevasio, said the vulnerability scans Docker is offering are only available to developers who pay, with a subscription to a Pro or Team plan. He also said that the service won’t scan for malware, and argued that Docker should do “integrated security scans” for both free and paid users.
Mouat said some of the infected images are low-risk. “They’re probably labeled as what they are, like ‘cryptominer,’ so users aren’t at risk of running them accidentally — although it’s still fair to question if they should be distributed by the Hub at all,” he said.
“But a percentage are masquerading as something else and trying to trick people into running them. This is dangerous — ideally these images would be removed from the Hub.”
Mouat also said it would have been more helpful to “focus on ‘official images’ — those that are vetted in some sense by Docker or partners — and find out how many had vulnerabilities in the most up-to-date versions, since old versions would be expected to have vulnerabilities. Also, look at how many of the vulnerable images are being actively used and if they are the latest version.”
Belcher agrees. “The ease of literally anyone uploading images to Docker Hub and the lack of a requirement to clean up older images reduces the significance of this issue significantly,” he said. “If their assessment was limited to specific, trusted hubs — images curated by Docker itself or other trusted parties — the findings would be much more significant.”
All of which raises a fundamental question: Whose responsibility is it to find and fix those vulnerabilities?
Shevchenko said it lies with both Docker and its users, noting that a container “is like a mini OS with all the dependencies prepackaged in it. If it’s left as-is for a substantial period of time, it will become vulnerable and therefore exploitable. Container images need to be continuously rebuilt and retested to make sure they don’t rely on vulnerable dependencies.”
There’s plenty of advice available about how to build and maintain the security of container images, including this webinar: “Vulnerabilities in Containerized Production Environments,” with Tim Mackey, principal security strategist within the Synopsys Cybersecurity Research Center (CyRC).
Ketkar said that, in general “you need to worry about the security of your build infrastructure, image security, registry security, and runtime security.”
Also, verify before you trust. “We always advise mistrust of images retrieved from any public repository,” Belcher said. “You should always test and verify containers created by a third party, prior to use. That an easily accessible, public image repository has images that contain older vulnerable libraries and even malware is common knowledge in the container security field.”
Mouat said the main takeaway for organizations from the report is that “they need to keep software up to date. This is a big task — it means not just looking at container images, but also the libraries used by developers, the version of Kubernetes and Linux on the hosts that run the images, etc.”
But he said it’s also important to remember that vulnerabilities and malware “are not specific to containers. The same code could also be deployed in other ways, such as via a package manager or in a VM.”
Feature image via Pixabay.
