During the CoreOS Fest in Berlin last month, leading Linux kernel developer Greg Kroah-Hartman shared some insights on security, noting “All bugs might be a security breach. We don’t know if it’s a security breach or not.”
You can’t get rid of bugs, of course, they are part of the software development process. What you can realistically do is take care of them, as soon as possible. “Kernel bugs are serious things. We want to fix those things,” said Kroah-Hartman.
For the past 15 years, the Linux kernel community has been following one rule: to fix the bugs as soon as possible. And the kernel community has been doing an incredible job at fixing those bugs, but there is a problem. Vendors that build their products on top of the Linux kernel don’t take those fixes. A majority of such vendors don’t bother with upgrading their devices. And that puts users of such devices at risk.
Kroah-Hartman gave a good example of how a benign looking bug can turn into something nasty. Some three years ago a TTY1 layer bug was fixed. Back then it looked like a normal, harmless bug. The kernel community fixed it, pushed it in the new release and was done with it. Three years later someone was found it to be a security bug. Now companies freaked out. They didn’t bother to use the patches back then, and now even companies like SUSE and Red Hat had to go back and fix all their old stuff.
“We have a very bad history of keeping bugs alive for a long time. Somebody did a check of it, most known bugs live for five years in systems. These are things that people know and know how to exploit. They’re not closed. That’s a problem in our infrastructure,” said Kroah-Hartman.
There are many reasons why companies don’t bother to update software. Companies sell them and move to the next version of the hardware because they make money by selling more hardware and not by maintaining the old gear. Since there is no monetary incentive, they don’t bother to update software on this hardware.
Lots of people run systems where they don’t accept they can’t update their kernel, or they think if they stick with the kernel and if nothing changes, it’s good. That’s not true,”– Greg Kroah-Hartman.
Google’s Android is a good example because it is using a long-term supported (LTS) model, but runs a very old kernel which has known exploits. Kroah-Hartman said, “There’re some well-known easy ways to get root on my phone … Which is great, because I like getting root on my phone, but that’s already been fixed. Fixes are pushed publicly, but they’re not being updated.”
However, running old kernel doesn’t mean it’s a bad thing. There are genuine reasons why people do run older kernels, and that is why Linux maintains LTS releases, updating them, largely thanks to Kroah-Hartman’s coordination work, with bug fixes long after the bulk of development work has moved on to newer versions of the kernel. But what good is fixing those older releases if companies are not pushing the patches to their Linux-dependent devices?
Over four years old, the 3.2 kernel is an LTS release and still is getting two fixes a day and being updated on a regular basis: Kernel developer Ben Hutchings is doing a release every other week. The Debian community is doing an excellent job at taking those patches and keeping it updated.
“A non-profit organization built of volunteer people is doing a better job than some of the largest Linux providers out there. That’s insane. That’s bad. Base yourself on Debian or update your kernel overtime,” Kroah-Hartman said.
New Kernel, Old Mindset
There is a common perception in the system admin community that once a system has been set-up and its stable, it shouldn’t be touched. Kroah-Hartman is not very happy with such a scenario.
“You have to be able to run a system that can upgrade itself. Lots of people run systems where they don’t accept they can’t update their kernel, or they think if they stick with the kernel, and if nothing changes, it’s good. That’s not true. We’re fixing about ten bugs in the kernel every day. Not all of them are security issues, but sometimes the big problem is we don’t know if an issue is a security issue or not,” Kroah-Hartman said in an interview.
— The New Stack (@thenewstack) May 9, 2016
There are a lot of nefarious parties that keep a close eye on unfixed bugs. They invest time and resources in finding out whether that bug can be exploited for security. If they find any such exploit, they may use it, or sell it to someone who could use as part of a larger attack.
“That is sad because we are doing the best we can. We’re fixing them, and we’re getting them updated,” Kroah-Hartman said.
The best approach is to use (or design) systems that update themselves, in Kroah-Hartman’s views. He gave examples of Chrome OS and CoreOS operating systems, both of which call in for updates and update themselves automatically.
“The Chrome OS and then the Core OS guys adopted the same mentality. You have two system images. You’re going to update one. Once you know it works, it can switch over to the other one. You have to be able to update it in a secure way. This technology’s been proven. The problem has been solved. People just need to use it and build it into their systems. The kernel is not going to go around updating itself on its own.”
CoreOS is a sponsor of The New Stack.