Citi Shows Continuous Secure Ingestion for Software Packages
AppSec developers at Citi have pledged to open source a platform they have developed to protect software supply chains by automating continuous security checks on the software and libraries requested by developers.
The proliferation of open source software has provided another way for malicious or simply malformed software components to find their way into corporate software stacks and beyond. The latest Sonatype State of the Software Supply Chain report said there had been a 742% average annual increase in attacks over the past three years.
Horrendous
James Holland, director of application security at Citi, told the Kubernetes Community Days UK conference in London that the bank manages three million external packages. “It’s horrendous,” he said.
He highlighted the danger of packages with pre- and post-install scripts as a route into organizations for bad actors. “As soon as the developer requests that library, that will get triggered within the package manager, on that developers’ application, on their workspace. As soon as that happens, you’re in trouble.”
“So, as a group do we think we should allow a package in, that has a pre- or post-install script?”. But, he continued, “How do you check? There’s nothing really there in the world to check that.”
He asked, “Can we have a secure supply chain if we do not understand the maturity of our libraries? It’s not possible?”
Solutions
Signing and verification schemes were some help he said, but malicious developers could still work around these systems. Intelligence feeds, such as Google Scorecard and the OpenSSF’s Alpha Omega Project were part of the answer to identifying mature packages, he said. Other tools and frameworks were starting to emerge, he said, such as the SBOM, the SLSA framework, and KEV.
Some components were easy to exclude, he said, because they came from sanctioned countries, relied on a single maintainer or had been identified as coming from a bad actor, for example. The challenge was how to approach components in the “gray area” that might not immediately meet the standards of more formal checks or could raise red flags with intel services but were not clearly in the “red zone” either.
“We’re trying to automate all these checks. So, if it fails that check, what do we do to get a good smell about it? How do we make sure that we can actually start using this library?”
For example, he said, React is signed, but one of its dependencies is not, which could fall foul of a strict “no unsigned libraries” policy. So the answer is to carry out additional checks on the dependencies “To make sure that we can bring them in and say, ‘it’s got no signature’, that’s fine we’ve done other things assuring it’s mature enough for us.”
Continuous Secure Software Ingestion
This has been brought together in a project called Continuous Secure Software Ingestion. “We want to try and automate the ingestion and grooming of the libraries.”
Holland said not everyone would have fixed, hardcoded policies, so “We need a nice flexible way for people to set the acceptable risk of that organization.” And, he said, offer a way of re-evaluating policy if intel or testing changes.
Citi is building the software with its partner, ControlPlane, to allow developers to request a resource through their normal tooling. The initial workflow will see the system run checks, and if requested components pass, they are moved to a binary/image store, and the developer is notified. Future workflows could involve Static or Dynamic Application Security Testing, or the use of assured providers.
The project is currently in “early alpha”, Holland said. “It is in the bank. We are going to open source it.”
This was a way of helping the whole community, he said. “Because at the moment there’s not a lot out there that does help developers make a good choice on what libraries they should be using.”
Other Types of Attacks
The dangers of supply chain attacks were also flagged up at the conference by Luke Hinds, senior principal engineer in the emerging technologies group at Red Hat’s CTO office, and the originator of Sigstore, which focuses on signing and verifying open source components.
“We see some prolific ones,” he told the conference. “Leaked tokens happen a lot, typosquatting happens within the package managers, key compromise.”
He also cited “Protestware, where developers become burned out and effectively hit out using the projects that they maintain. And then being a supply chain, these attacks proliferate to other systems. So it’s not an isolated attack.”
He said it was essential that supply chain security efforts were easy for developers to use within their existing workflows. “If you introduce complexity, and risks, such as key management and setting up a YubiKey and all these sorts of things, for some developers it’s something that’s just too much labor, then your adoption is generally quite weak,” he noted.
