Temper Kubernetes and Container FOMO Through Security

Sep 25th, 2018 9:26am by Chris Ford

Chris is the vice president of product at Threat Stack, where he is responsible for product management and user experience. He brings more than 20 years of experience leading product teams and building innovative security solutions, including data loss prevention, enterprise mobility management and secure cloud collaboration. Prior to joining Threat Stack, Chris was chief product officer at Blue Cedar Networks and senior vice president of product management at Intralinks, a secure cloud collaboration platform.

Gartner estimates that 50 percent of companies will use container technology by 2020, up from less than 20 percent in 2017. Many times, businesses rush adoption of “hot technologies” like containers without fully understanding how to properly secure the new tech they are deploying. Companies may be inadvertently doubling down on their insecure cloud security practices by leveraging orchestration tools like Kubernetes to speed up their container deployments and help them manage multi-container clusters at scale.

The operational benefits of containers are clear but letting Kubernetes FOMO (“fear of missing out“) blind businesses to the risks of implementing these technologies within their stack without fully understanding their security implications can have a significant detrimental impact. Like many aspects of cloud infrastructure security, containers and Kubernetes can have a significant, often unexpected impact, and organizations must align their business and security operations to effectively mitigate risk.

Kubernetes’ Blast Radius

The “blast radius” concept is central to many Kubernetes-related attack scenarios. Similar to the area affected by an explosion, blast radius in this context refers to how much access a malicious party can gain from the initial point of compromise. A compromised Kubernetes console can make it easy to escalate privileges and take control of other nodes in a cluster; therefore, any secure deployment will put a heavy focus not only on access control but also on limiting the potential blast radius.

“Write access to the etcd backend for the API is equivalent to gaining root on the entire cluster, and read access can be used to escalate fairly quickly.”

In particular, cybercriminals have been targeting Kubernetes’ backend database (etcd) by searching on web crawlers like Shodan because many businesses leave them inadequately protected as a result of their inexperience with the technology. According to Kubernetes ’ security documentation, “Write access to the etcd backend for the API is equivalent to gaining root on the entire cluster, and read access can be used to escalate fairly quickly.”

The good news is that Kubernetes has built-in role-based access controls (RBAC) that regulate container access and limit the impact of compromised containers. This isolation is further strengthened by leveraging Kubernetes’ isolation mechanisms like namespaces and network segmentation.

Most organizations should also minimize the number of containers that can run in privileged mode, keeping in mind that the blast radius gets exponentially larger with greater access to clusters.

Configuration Errors are Costly in Container Security

In the same way that the majority of data breaches can be prevented by basic cybersecurity hygiene (i.e., routine patches), container security starts with ensuring that each cluster has the proper security configuration. Our research indicates that 73 percent of companies have at least one critical AWS security misconfiguration, and this type of basic mistake also extends to containers. Dev and ops teams that don’t have a lot of experience with containers raise the risk of misconfigurations during deployment which could put critical data at risk.

Oftentimes, businesses don’t realize that secrets, including service passwords and API keys, are frequently hardcoded into containers. Third-party tools such as Docker secrets, CyberArk Conjur and HashiCorp Vault empower IT admins to encrypt this type of sensitive information, but many organizations are making the mistake of relying on Kubernetes’ own mechanism to perform this function. There’s nothing actually wrong with this practice. However, if combined with a configuration error, the results could be disastrous. Kubernetes has detailed documentation that IT admins can use to help them avoid these types of mistakes.

Another common configuration error that occurs when deploying Kubernetes is that, by default, many clusters are automatically configured with an authentication token provides access to the Kubernetes API. Configuring RBAC controls can help mitigate the risks, but if that token has cluster admin rights, an attacker who accesses a single container can escalate these privileges and take over the entire cluster.

The best way to ensure safety with Kubernetes is to read the “manual,” aka the documentation. The second-best thing that organizations can do is build security into their container and Kubernetes deployments right from the start. Securing your environment proactively through proper configuration is much simpler and less expensive than trying to respond to a data breach after it occurs.

Feature image via Pixabay.

TNS owner Insight Partners is an investor in: Docker.