How DevOps Sites Are Battling Cryptocurrency Miners
This week GitLab announced a subtle but significant change in policy for its web-based DevOps lifecycle tool. Going forward, all new trial accounts wanting to use continuous integration jobs will be required to provide a valid credit or debit card number. “No charge will be made and no money will transfer,” the announcement stressed. But GitLab is hoping the change will make it more difficult to abuse its platform after what it’s calling a “massive uptick” in cryptocurrency mining.
It’s part of a larger war that’s now heating up between cryptocurrency miners and some of the biggest continuous integration/continuous delivery platforms.
Elsewhere, Dutch security engineer Justin Perdok described a tactic he’d been seeing since 2020 involving GitHub Actions. In April, Perdok told The Record he’d experienced pull requests on his repositories that triggered automated testing, where the tested code would then temporarily create a virtual machine on GitHub’s infrastructure that ran cryptocurrency-mining software — sometimes, 100 cryptominers at one time. Perdok had spotted at least one account creating hundreds of such pull requests, and The Record reported that those attacks “appear to be happening at random and at scale.”
TIL that if a miner targets a specific user/organization with these Github Actions shenanigans they can (apparently) effectively prevent other actions from running on unaffected repo’s, if the miner spawn enough jobs at his target. pic.twitter.com/lQS4UD6Hkv
— Justin Perdok (@JustinPerdok) April 7, 2021
In late April a GitHub blog post acknowledged that “cryptomining on Actions is not new,” adding that “we’ve been fighting abusers since the beginning.” (GitHub Actions launched in 2018.) “However, as the price of coins has gone up, the number of abusers has escalated. We’ve spent thousands of hours combating abuse and implemented dozens of different mitigations to detect and prevent it.”
So now GitHub, too, has made some new changes, updating its policies last month to require pull requests from first-time contributors to first be approved — manually —by a repository collaborator with write access. The company’s April blog post lauded the way users of its CI/CD service were boosting productivity, but added that “we’ve unfortunately also seen a wide variety of bad actors abusing Actions, affecting service performance, and causing denial of service to open source projects.”
Running Amok
But then, in the wake of The Record’s article, more services came forward last week to report similar attacks. In a follow-up article, the Record also cited GitLab, Microsoft Azure, TravisCI, LayerCI, CircleCI, Render, CloudBees CodeShip, Sourcehut, and Okteto. (Their article’s headline? “Crypto-mining gangs are running amok on free cloud computing platforms.”)
The site also reported on a new tactic: Creating free trial accounts on the platforms to run cryptocurrency mining apps — again and again, “keeping the provider’s servers at their upper usage limit and slowing down their normal operations.” (The list they provide of affected companies is nearly identical: Microsoft Azure, LayerCI, TravisCI, Sourcehut, CloudBees CodeShip, and CircleCI.)
In February, Microsoft’s Azure Pipelines even ended its free pipelines for new CI/CD projects, complaining that abuse, especially cryptocurrency miners, “has gotten substantially worse,” accounting for “a high percentage of new public projects in Azure DevOps … In addition to taking an increasing amount of energy from the team, this puts our hosted agent pools under stress and degrades the experience of all our users — both open source and paid.” (Microsoft Azure has since updated the policy to allow this capability after receiving an emailed request with a description of the intended use and links to the repositories for the build.)
“We are sorry for the inconvenience this will introduce for open source customers wishing to use Azure Pipelines for CI/CD,” the announcement stated. “Unfortunately, we believe that this is necessary for us to continue providing a high level of service to all our customers.”
And GitLab’s blog post had also emphasized that the problem of high-load cryptocurrency miners is bedeviling several more platforms that also offer free CI/CD pipelines, citing an April blog post by Colin Chartier, CEO and co-founder of hosted DevOps platform LayerCI. Warning that “Crypto miners are killing free CI,” Chartier shared the story of more affected CI providers, including TravisCI, and Shippable (now owned by JFrog), which are also “all worsening or shutting down their free tiers due to cryptocurrency mining attacks.”
“We love our open-source software teams … and we fully want to support that community,” stressed a November announcement from TravisCI, a hosted continuous integration service. But the company added that “in recent months we have encountered significant abuse of the intention of this offering,” citing both increased activity of cryptocurrency miners, as well as “TOR nodes operators, etc. … Abusers have been tying up our build queues and causing performance reductions for everyone.”
Triggering a Discussion
Chartier’s April blog post argues that the problem is exacerbated by the rise in the value of cryptocurrencies. “It’s become profitable for bad actors to make a full-time job of attacking the free tiers of platform-as-a-service providers.” The company provides specific case studies: One user was apparently pulling in $77 a month, “a considerable sum in many countries, especially given that the only tools required are a laptop and an internet connection.”
The post continues, “Providers can do their best to enforce terms of service, but as long as it’s profitable and untraceable to make such attacks, they will continue to become more sophisticated and circumvent measures.”
GitLab’s post had enumerated the real day-to-day impact of the cryptocurrency miners. “In addition to the cost increases, the abuse creates intermittent performance issues for GitLab.com users and requires our teams to work 24/7 to maintain optimal services for our customers and users.”
But this triggered a larger discussion. Docker CTO Justin Cormack shared LayerCI’s blog post with his 11,500 followers on Twitter, drawing some interesting reactions. “We have to redefine ‘free tier’ as ‘The price to make crypto mining unprofitable’,” quipped DevOps consultant Ismail Baskin.
And there was more discussion when LayerCI’s blog post turned up on Hacker News. In a comment, Philadelphia-based coder Drew DeVault, the maintainer/sys-admin behind Sourcehut, warned the problem was spreading. “I’ve been in touch with many other people working in the CI industry and this has become a massive problem for all of us over the past few months. Entire industry working groups have been set up for knowledge sharing to combat the crypto mining epidemic.”
Sourcehut also recently announced plans to, going forward, require all project maintainers to keep a paid account for their continuous integration service builds.sr.ht, complaining that malicious users “have been deliberately submitting huge numbers of jobs under dozens of frequently registered accounts and deliberately circumventing our abuse detection to use as much of our resources as possible to mine cryptocurrencies.
“This exhausts our resources and leads to long build queues for normal users.”
DeVault’s comment in Hacker News went on to argue against cryptocurrency itself — claiming, among other things, that it’s “introduced perverse incentives into the entire technology sphere.” It’s a point LayerCI’s Chartier made in his blog post: that free tiers on cloud platforms are endangered by the way many cryptocurrencies reward “proof of work.”
“Yup, we’re dealing with it on CodeShip,” added another comment, on Hacker News, adding “I’m pretty sure all CI services are swamped by mining on free accounts or fraudulent paid accounts.”
Speaking to The Record, a CodeShip engineer added that “it’s not just the free accounts. Sometimes they pay the small fees for our accounts, which are way cheaper than renting on AWS directly, and mine cryptocurrency at maximum capacity.”
During the Hacker News discussion, Anurag Goel, the founder and CEO at Render, a unified cloud platform hosting apps and websites, also agreed that “It’s not just CI providers: we’re seeing the same thing on Render.”
Or, as The Record describes it, “If it is a web service that provides free access to a high-computing system, cryptomining gangs have most likely tried to abuse it by now.” The site reports that some cryptocurrency forums have even included tutorials that sharing how to mine with a free trial account on Oracle Cloud or on the cheaper tiers of Alibaba Cloud.
In the end, it’s become a battle between the technical fixes of the infrastructure providers — and the technical tricks of the legion of miners trying to bypass them.