In the distributed world of the Internet, a security threat can have ripple effects out over an ecosystem, creating challenges in keeping everyone up to date.
Infrastructure service providers need to address the breach in their own systems, then alert their customers, partners and resellers. These partners, in turn, need to check and fix their own breaches and must then repeat the process of alerting their customers. And so on, and so forth.
As Tal Klein wrote on The New Stack earlier this week:
“It is staggering to consider the scalability and agility needed both by IT and business units to adopt the cloud, migrate, rapidly scale and harness the myriad of new technologies that are emerging from this movement to create new stack infrastructures.”
The potential of the new platforms that new stack infrastructures are enabling suddenly becomes a burden when a security risk pervades the system. In this world, the service providers arguably face the greatest risks, even if they take every precaution possible. Realistically, these companies are far more able to isolate threats than their customers can manage to do it themselves but the unknowns are always the biggest threat.
Shellshock adds another dynamic, making it all seem like we face infinite possible attack vectors.
Case in point of this eerie chaos is illustrated in a post by Ericka Chikowski on Dark Reading about research from Incapsula that in “just 24 hours, it recorded 17,400 attacks against its WAF installed base, lashing out at approximately 1,800 domains. The attacks are originating from 400 unique IP addresses, with more than half of them in China or the US. Researchers have noted not only an increasing volume of attacks, but also a growing variety of ways attackers are leveraging the Bash bug to commandeer web servers.”
Kaspersky Lab’s Stefan Ortloff detailed two attacks. One, known as a reverse-connect-shell, will “just create a new instance of bash and redirect it to a remote server listening on a specific TCP port.” The other uses specially crafted HTTP-requests to start installing Linux backdoors on victims’ servers.
Shellshock and Bash
Security threats like Shellshock — identified on September 24 — are particularly difficult to deal with. The Shellshock bug affects all Bash instances through to version 4.3. Bash is a command processor used as the default shell application for Unix, Linux and Mac OS X. This is particularly worrisome because Unix and Linux are the operating systems often used to run server architectures and power cloud infrastructure web services. What’s more, security problems in Unix and Linux systems require manual security fixes: system administrators need to implement bug patches, whereas consumer-oriented operating systems usually install updates automatically.
Already, Amazon Web Services has alerted their customer base of the threat, noting that in many EC2 instances, their APIs and backends were not affected, while providing links to specific instructions for Amazon Elastic MapReduce, AWS OpsWorks, AWS CloudFormation and other services that require updating. Interestingly, in Amazon’s communique, they do not refer to the security bug as “Shellshock” at all, instead using the official codes of CVE-2014-6271 and CVE-2014-7169. Nor do they mention which version of Bash users should be checking to make sure theirs is up to date.
Meanwhile, cloud hosting provider Digital Ocean wrote to all of their customers on September 26, alerting them to the issue (referring to it as Shellshock in the main missive, with the technical CVE names listed on an accompanying tutorial).
The main email alerted customers that “the problem is serious” and provided a link to the tutorial page that helps guide users through checking their system vulnerability and how to update Bash on various Linux distributions.
For service providers like Digital Ocean, there is little risk that customers will confuse the widespread nature of the security risk with thinking it is unique to Digital Ocean’s architecture, says Mitch Wainer, CMO at Digital Ocean:
The word was out before we made our announcement. There’d be no reason for someone to believe it was a Digital Ocean specific problem, unless they were severely misinformed. That said, we do our best to respond quickly once we have a grasp of the issue at hand, remaining transparent and open to our users at all times.
We wrote the tutorial and promoted it via all of our social channels. Then we made sure to send an email out to all of our users with the necessary information and a link back to the tutorial. We also have a very active community where questions can be asked and answered in real time.
Media archive storage provider, StorageDNA, took a similar route. They believed widespread publicity around Shellshock’s existence meant they could leap straight into a communications focus on what to do about it. Director of Marketing Rebecca Adler Greenwell confirmed their systems run off Linux CentOS, which includes Bash, making customer’s vulnerable without the patch:
Since this incident was widely publicized, StorageDNA’s approach was to provide concise factual communications about the security risk including the company’s recommendations, with directions for how customers can apply the patch to their systems.
Communications have been sent as a preemptive measure through all of our outlets (web, support site, reseller portal, email, and social media). In addition, we offer customers the option to open a ticket via our support site if they require further assistance with applying the patch. Because the notice and fix guide has been posted to all of our communication outlets, the vast majority of our customer and reseller community will see the information. It’s possible that some will not see the notice and will proceed to contact the company directly. In these cases, the StorageDNA team can manage these inquiries on a case by case basis.
But in the stack infrastructure ecosystems, it is not just the cloud hosting providers that must alert their customers. Many of these customers are resellers or value-add partners who offer another layer of service architecture. Tutum, for example, is a docker platform built in part on Digital Ocean’s infrastructure. CTO and Cofounder of Tutum, Fernando Mayo Fernandez explains:
For our service, because we are another provider, we had to fix it in two places: first, we had to fix our own service. Our service runs on top of AWS, so we fixed the nodes and our containers. The other side is the user nodes. We run our user’s containers.
The nodes in our fully managed service were in AWS, so we had to fix those for Shellshock. But we cannot update our user’s containers, and there is nothing we can do about that.
For our second service, our user’s containers run on Digital Ocean. In this version, our users must update their containers. They have to look inside their containers and make sure they are running the correct version of Bash.
While Tutum have not emailed their clients directly as yet, or published a blog to explain how to update systems, they have plans to do so. In the meantime, they are confident that their clients using Digital Ocean containers, will have received the email alert from Digital Ocean directly.
Apart from communicating the security concern through various channels, it is difficult for the hosting providers to ensure clients take the requisite action. Security researchers at SpiderLabs on October 1 said that the threats are real: they are seeing a “huge spike in attack traffic” since the bug was identified on September 25.
The ongoing ramifications of Shellshock may continue, as these hackers continue to seek out weaknesses in web service architectures.
The good news is that those at risk are responding to the threat. Digital Ocean shares that their email alert had an opening rate of 52.1 percent – 10 points higher than engagement levels of other emails sent by Digital Ocean. What’s more, 38.9 percent then clicked through to the tutorial, up more than 400 percent than usual click-through rates for emails, Digital Ocean told us.
Still, that leaves almost half of their customers perhaps not addressing the threat: communicating security threats is one thing, getting ecosystem partners to take action to prevent a breach is an even harder task.
Digital Ocean and Adallom are sponsors of The New Stack.
Feature image via Flickr Creative Commons