Unfixable Kubernetes Security Hole Means Potential Man-in-the-Middle Attacks
Well, it had to happen eventually. Every now and again — I’m looking at you Intel with your Converged Security Management Engine (CSME) snafu — a security bug shows up that can’t be fixed. Now, it’s Kubernetes‘s turn. Tim Allclair, an Apple software engineer, who also helps oversee Kubernetes security, announced a “security issue was discovered with Kubernetes affecting multitenant clusters. If a potential attacker can already create or edit services and pods, then they may be able to intercept traffic from other pods (or nodes) in the cluster.”
To be exact, if a hostile user can create a ClusterIP service and set the spec.externalIPs field, they can intercept traffic to that IP. In addition, if a user can patch the status of a LoadBalancer service, they can also grab traffic. Now, the latter is a privileged operation and Joe and Jane User shouldn’t have that right, but, in practice, mistakes are made and it happens.
As Markus Jevring, a software developer asked on Twitter, “Is this even how Kubernetes is intended to be used? Was multitenancy in the same cluster a design-time concern?” No, no it wasn’t. But, as Allclair replied, “Certainly not. But regardless, it’s how it’s being used now, and we’re forced to adapt.”
Unfortunately, as Jerving pointed out in his follow-up, I “just feel like issues like this aren’t security issues. It’s like leaving the house key in the lock and being surprised when you get burglarized.” That’s pretty much right.
And, there’s worse news ahead. As the CVE-2020-8554: Man in the middle using LoadBalancer or ExternalIPs report states: “All Kubernetes versions are affected. Multitenant clusters that grant tenants the ability to create and update services and pods are most vulnerable.”
What’s the answer you ask for this problem. You’re not going to like this, but there’s No Patch for it. Instead, “it can currently only be mitigated by restricting access to the vulnerable features.” Why!? ” Because an in-tree fix would require a breaking change, we will open a conversation about a longer-term fix or built-in mitigation. There’s no word when a fix will be in. It won’t, it appears, be out in time for the next Kubernetes release. There’s not a word about it in the Kubernetes 1.20 changelog.
What you can do is restrict the use of external IPs with an admission webhook container: Here’s its source code and deployment instructions. This prevents services from using random external IPs. Or, you could lock external IPs by using OPA Gatekeeper, with the following sample Constraint Templatecan
Kubernetes isn’t providing a mitigation for the LoadBalancer IPs since granting users patch service/status permission was never a good idea. If, for some reason, you absolutely had to do this, the Kubernetes security team suggests that you lock down LoadBalancer IP using the same general approach for the external IP mitigations.
Since external IP services are rarely used, the Kubernetes crew recommends you manually audit your clusters’ external IP usage. Besides making sure users can’t patch service status, you should keep an eye out for user patch service status requests.
Put it all together and this has the potential to be a real security mess. But, it’s only being rated at medium severity, 6.3. That’s largely because there aren’t that many multitenancy clusters.
Still, I’d keep an eye on this one. I wouldn’t want to be the sysadmin to discover that my clusters are wide open for snooping between tenants. Would you?
