Red Hat: Why CVE2020-8554 Is No Kubernetes Apocalypse
Red Hat sponsored this post.
Last month, there was a kerfuffle around CVE 2020-8554, the External IP Services Routing vulnerability in Kubernetes. Rather than being a simple patch-and-fix problem, some argued that the issue was baked into the very core of Kubernetes’ design. That it was a fundamental flaw endemic to the architecture of Kubernetes. Red Hat would like to stand up for Kubernetes and the Cloud Native Computing Foundation to defend its work on fixing this issue. There is no single solution for all users, unfortunately.
As The New Stack security correspondent Steven J. Vaughan-Nichols described the flaw: “If a hostile user can create a ClusterIP service and set the spec.externalIPs field, they can intercept traffic to that IP. In addition, if a user can patch the status of a LoadBalancer service, they can also grab traffic. Now, the latter is a privileged operation and Joe and Jane User shouldn’t have that right, but, in practice, mistakes are made and it happens.”
At issue is the fact that Kubernetes users can get into the middle of traffic that is being routed to and from the Kubernetes cluster, by exploiting an unrestricted service property.
David Eads, Red Hat senior principal software engineer and also a leading contributor to the Kubernetes Project, explained the security flaw as follows:
“There is a field on services that allows you to configure iptables’ logical rules on every node which says a particular IP should get routed to my pod. For OpenShift, Red Hat’s enterprise Kubernetes offering, we had to find a solution to this flaw that would work for our customers, for our on-demand self-managed solutions, and for the OpenShift installations users were handling themselves.
Our solution restricted access to this feature, and said ‘No, normal users cannot set this. Cluster admins can set this and delegate and set an allowed list of IPs.’ Traffic comes in looking for destination 192.168.0.2 and this service can say, ‘Yeah, that’s me.’ In upstream Kubernetes there is no restriction on any user from putting in any IP.”
While this issue goes deeper than a simple one-size-fits-all patch, it’s been an ongoing focus of attention for Clayton Coleman, architect for containerized application infrastructure at Red Hat, and for Maciej Szulik, principal software engineer at Red Hat. Both are contributors to Kubernetes and Coleman was on the CNCF’s Kubernetes Project Steering Committee.
The issue at hand was recognized many years ago and fixed in extremely early versions of OpenShift, but the upstream fix was not so easy to apply. It turned out that restricting this functionality caused problems for some users. Every time the issue was brought up, a good generic solution was not found.
That’s always the precarious balance — trying to find the right solution for a community-driven project’s problem. Coleman said he opened the first issue on the problem for OpenShift in 2016, and even the folks quoted then had some in-depth discussions as to what the correct solution to the problem should be. Even Red Hat’s software developers had some differing opinions on how to solve the problem while causing the fewest compatibility issues for users.
In the end, said Clayton, the problem isn’t necessarily that this flaw is, as the previous The New Stack article implied, “unfixable.” The issue is that a generic solution for this problem is not easy to find. While Red Hat’s solution was to essentially control access to this feature, another upstream solution is a plug-in acting as an off switch for external IP services entirely. That’s a drastic step, but certainly a more secure one than allowing anyone to access those IPs.
“It’s not always easy to find the right solution for everyone,” said Coleman. “That doesn’t mean we should give up. We are committed to working with upstream communities, and always maintaining that level of security.”
An upstream solution for this particular problem is not entirely certain yet. That’s not to say there aren’t workarounds, like the aforementioned off switch.
Indeed, the issue of External IP Service Routing remains open on GitHub and debate continues as to what the best solution will be. And while the CVE may be new, the issue behind it has been mentioned in the Kubernetes GitHub issue stream in the past — but has been closed out.
So while this CVE may sound like an earth-shattering vulnerability that could shake the very foundations of Kubernetes, this has been a long-standing issue for the Kubernetes project for some time already. The only real change due to the filing of this CVE is that a solution will likely occur within the community sooner than it would have otherwise.
Behind the CVE is an extremely healthy community of very talented people who have to choose, every day, which bit of the project to focus on with their limited time. They’ve routinely addressed every major issue they have come up against, and the same will prove true here as well.
There will undoubtedly be future security issues for Kubernetes, but they’re not likely to leap from the depths suddenly and without warning. The CNCF has a great many extremely smart people working day and night around the globe to make sure Kubernetes is as reliable and secure as possible, and they generally know where the tricky, fiddly bits are located already.
The real trick for open source development isn’t necessarily identifying those complex parts that need tending to, but instead, it’s about finding the right people and finding the time for them to do the work.
Feature image via Pixabay.
The Cloud Native Computing Foundation (CNCF) is a sponsor of The New Stack.
