Here’s Why You Should Secure Your Etcd Deployment
Etcd, a key-value store and a core component of Kubernetes clusters, is used to store highly sensitive configuration data but is also easily left unprotected, as a developer recently found.
Puerto Rican software developer Giovanni Collazo was looking into etcd, first developed by CoreOS, and realized that before version 2.1, released in July 2015, it didn’t support any type of authentication. Even after it was added, this feature was kept off by default for backward compatibility reasons.
A similar approach was taken by MongoDB developers in the past and resulted in thousands of insecure deployments on the internet that were abused by hackers. So, Collazo set out to see if etcd’s design decisions had a similar effect.
A quick search on Shodan, a search engine for devices and services, revealed 2,284 etcd servers that were directly accessible from the internet through their RESTful APIs.
“I clicked a few and on the third try I saw what I was hoping not to see,” Collazo said in a blog post. “CREDENTIALS, a lot of CREDENTIALS. Credentials for things like cms_admin, mysql_root, Postgres, etc.”
The developer wrote a script to grab the exposed etcd key stores in JSON format and stopped after collecting around 750MB of data from 1,485 servers. He then searched the data for interesting strings and found 8,781 passwords, 650 instances of “aws_secret_access_key,” 23 of “secret_key” and 8 of “private_key.” He also came across a few certificates.
“I did not test any of the credentials but if I had to guess I would guess that at least a few of them should work and this is the scary part,” Collazo said. “Anyone with just a few minutes to spare could end up with a list of hundreds of database credentials which can be used to steal data, or perform ransomware attacks.”
Aside from being used for storing Kubernetes cluster data, etcd is also used with other tools such as locksmith, vulcand and Doorman. So, in addition to stealing valuable secrets, attackers could also potentially change configuration data for various services.
Administrators should avoid exposing etcd deployments directly to the internet and should secure them by enabling authentication. According to the official Kubernetes documentation, the user-based etcd authentication model is not currently supported, but TLS-based authentication, where clients identify themselves through certificates, is.
“After configuring secure communication, restrict the access of etcd cluster to only the Kubernetes API server,” the documentation advises.
There’s also a difference between the authentication model in etcd v2 and etcd v3, which uses gRPC as its transport instead of a RESTful interface.
Red Hat, which recently acquired CoreOS, is a sponsor of The New Stack.
Feature image by Jason Blackeye on Unsplash.
