Not Your Usual Supply Chain Hack: The Codecov Bash Uploader Blunder
We all know about the SolarWinds supply chain hack. But, while smaller in scope, Codecov‘s Bash Uploader Security supply chain failure is also a record-setter. And, this is not a record anyone wants to break.
Months after their code was busted Codecov only discovered the foul-up, thanks to a security-conscious user. He checked the Secure Hash Algorithm 1 (SHA-1) checksum for the Github version of Codecov Bash Uploader and the SHA-1 checksum for the downloaded Bash Uploader version with shasum — a Linux program that calculates and verifies SHA-1 hashes — and found they didn’t match. In other words, they were not the same program.
Whoops!
Codecov is a reporting tool that inserts coverage metrics directly into continuous integration (CI) workflows. Its job is to watch for coding problems while running test suites. It especially looks in pull requests where new features and bug fixes are usually found and new bugs and problems often pop up.
Bash Uploader’s task is to export users’ CI environmental data. This includes any credentials, tokens, or keys users were working within their CI runner when the Bash Uploader script was executed. That’s already dangerous enough because its name is perfectly descriptive. Bash Uploader uses the Bash shell and curl to upload unencrypted environmental data to Codecov. And, oh yes, to the attacker’s server as well.
This method of sending data was just asking for trouble. As a Slashdot reader said, “Piping a URL you don’t control directly into Bash… what could possibly go wrong?” I mean, there is literally a Hall of Shame webpage for people who make this blunder.
Adding insult to injury, the Bash Uploader is also used in three related uploaders: Codecov-actions uploader for GitHub, the Codecov CircleCl Orb, and the Codecov Bitrise Step, so if you used Codecov’s uploaders with any of these you were tagged by this problem.
So what could happen to you? Glad you asked. According to the company, the altered Bash Uploader script might affect:
- Any credentials, tokens, or keys that our customers were passing through their CI runner that would be accessible when the Bash Uploader script was executed.
- Any services, datastores, and application code that could be accessed with these credentials, tokens, or keys.
- The git remote information (URL of the origin repository) of repositories using the Bash Uploaders to upload coverage to Codecov in CI.
If you think it through, as one Ycombinator reader did, this means “So the hackers stole every environment variable for the context in which the Codecov script was run. It means that if you use CI to deploy your code, all of your credentials have been leaked.”
So, how long did this go on? The company reported there was periodic, unauthorized access to the company’s Google Cloud Storage (GCS) key beginning January 31, 2021. This enabled a malicious third-party to alter its bash uploader script to potentially export CI to a third-party server until April 1, 2021. No, that’s not an April Fool’s joke.
This is what I like to call a “Shoot me now” moment. As in, shoot me now rather than have to go into the executive suite and tell the CIO, CTO, and CISO that our CI data’s been hijacked for three months.
How did this happen? Somehow — we don’t know exactly how — the attacker gained access to the code because of an error in Codecov’s Docker image creation process. This enabled the hacker to extract the credential required to modify the Bash Uploader script. With that unauthorized access, they modified the script to export the CI data to a third-party server.
The only “good” news here, according to a Ycombinator comment ascribed to Eli Hooten, Codecov’s chief technology officer, “Based on the nature of this attack I do not believe malicious actions were executed directly against the CI pipeline, nor do we have any evidence of it.” If true, this break-in may have been a true hacker just poking at a system to see what they could find out rather than a cracker eager to monetize your data.
Not that matters for as far as what you need to do now. As Codecov states:
We strongly recommend affected users immediately re-roll all of their credentials, tokens, or keys located in the environment variables in their CI processes that used one of Codecov’s Bash Uploaders.
If you’ve been auditing how these tokens are used in your system, now would be a good time to start scanning your logs for unusual activity from the end of January to the beginning of April.
Codecov is saying all the right things now about securing their systems and auditing their code. But, with the horse already out of the barn, if I were one of their approximately 29,000 customers using its programs to check my code quality, I’d have a long, hard talk with them before I’d trust them with my CI data train.
I’d also take a deep look into their scripts to see if there are other problems lurking about. Finally, few people bother to check SHA-1 and other checksums for problems between what a program says it is and what it really is. It’s time to start. After all, checking these sums with programs such as shasum and sha256sum can be automated by pipelining them in a Bash script. This would make using curl in a script like Bash Uploader safer.
