How Rancher Discovered the Kubernetes Vulnerability
The trouble with the recent Kubernetes vulnerability (CVE-2018-1002105) started a couple of years ago, long before it became public earlier this month. In 2016, Rancher users were complaining about mysterious error messages and setup failures they were experiencing with the release of Rancher 1.6 and Amazon’s ALB. More recently, the community began to experience similar problems in August with Rancher 2.1.
“It was a pretty low risk vulnerability in the sense of it was very likely that there were other protections in place that will protect you from getting to it, but overall, the experience for us was really amazing working with the rest of the Kubernetes community to identify it, get it pushed out, get patches pushed to everyone and things worked the way they’re supposed to,” Shannon Williams, co-founder and vice president, of sales, said. “We identified it. It was kind of kept quiet until the fixes that were pushed out, and then everyone had the ability to patch really quickly last week.”
Williams described Rancher’s course of action, and with the help of the community, how it had plugged the Kubernetes security holes, during a podcast hosted by Alex Williams, founder and editor-in-chief, of The New Stack at KubeCon + CloudNativeCon North America 2018.
The core of the risk was how by using the exploit “you could take your privilege and escalate it to admin,” Shannon Williams said. “Most people were protected because typically, load balancers kind of ensure that the kind of traffic that would go through to make this happen couldn’t get through, and the exploit couldn’t be consumed,” Williams said. “But it was there and for anyone who’s using without that kind of a load balancer in front of it, there was definitely a risk.”
As mentioned above, Rancher discovered some unusual activity a couple of years ago with the use of Rancher 1.6 and Amazon’s ALB. “It was kind of on the back-burner, something we weren’t paying attention to and then maybe two months ago, all of a sudden, it resurfaced,” Williams said. “And we saw it again with Rancher 2.0 on Kubernetes and once we saw it, we were like, ‘you know, that bug shouldn’t still be there with ALB. ALB is pretty stable now.’ And so, we looked into a little bit deeper and found there was actually a potential exploit where people could escalate their privileges.”
Without downplaying the security risks CVE CVE-2018-1002105 poses, Williams said the exploit was a “pretty low-risk vulnerability in the sense of it was very likely that there were other protections in place that would protect you from getting to it.”
“But overall, the experience for us was really amazing working with the rest of the Kubernetes community to identify it, get it pushed out, get patches pushed to everyone and things worked the way they’re supposed to,” Williams said. “We identified it. It was kind of kept quiet until the fixed that were pushed out, and then everyone had the ability to patch really quickly last week.”
The community followed the typical route of fixing the vulnerability by releasing the patch and getting the word out in a way that has not been “different from any other open source project where the community identifies the vulnerability and assuming that it’s identified by someone non-malicious, it’s going to be sort of escalated through the process,” Williams said,
“What we saw was everyone took it incredibly seriously. A number of engineers worked on resolving it and the notice was done very effectively to everyone once there’s a patch available to put it,” Williams said “When we discovered it, we put in place some stuff in Rancher to prevent it anyways to make sure that it was prevented. And so, our customers were already protected from it but as soon as the patch was out, we gave them a really simple mechanism to upgrade their deployments so you know, they just had to go roll and upgrade and they were on the latest version of Kubernetes.”
In this Edition:
1:23: Your take on the Kubernetes vulnerability we saw last week.
3:43: What are the first round of checks you make when you see packets being dropped?
8:41: What does it mean that this is the first Kubernetes CVE vulnerability for you?
14:08: The developer experience at Rancher and the built-in experience that Kubernetes offers.
17:56: The changing complexities and practices as Kubernetes evolves.
20:39: The thriving tool ecosystem and its continued maturity and consolidation.