Linux Kernel Security in the Age of Spectre, Meltdown
The security landscape for Linux has been a different world since the hardware-based attacks of Spectre, Meltdown and others have proliferated, according to Greg Kroah-Hartman, speaking at the last KubeCon+CloudNativeCon event in Shanghai earlier this year. In a nutshell, they have forced Linux users to an unfortunate choose-your-own-adventure when it comes to running Linux: performance or security.
“The bad part of this is that you now must choose: performance or security. And that is not a good option anybody wants to make,” Kroah-Hartman told the crowd.
The most recent vulnerability found earlier this year was, was MDS (Multiarchitectural Data Sampling), and follows in the path of Spectre/Meltdown attacks in the year prior in that they took advantage of the speculative execution model found in Intel processors. Such processors try to anticipate the next action that is taken, in order to speed performance, which has been problematic in terms of inadvertently exposing data. This is particularly troublesome in cloud computing environments, where one customer’s data could be accessible by other users.
It turns out that the OpenBSD community, well ahead of Linux developers themselves, were the first to understand that the technology that speculative execution was based on — namely Intel’s SMT (simultaneous multithreading), where a CPU split itself into multiple virtual cores — was problematic. It called for disabling SMT in June 2018, well before Zombieload, RIDL (Rogue Inflight Data Load), Fallout, MDS and others subsequently appeared. They all are variations of the same basic themes, and disabling multithreading eliminated many of the problems they introduced.
Although these are hardware bugs, it is the role of the operating system kernel to fix them, Kroah-Hartman said. To mitigate against such attacks, users must disable hyper-threading, and update their kernels and their hardware BIOS. “If you just do one or another you will not be safe,” he said. RIDL and Zombieload, for instance, can steal data across applications, virtual machines, even secure enclaves.
Syscalls Are Now Expensive
The downside to patching for these vulnerabilities is the fact that servers ran slower, as the cloud service providers quickly found out. “Traditionally, system calls in and out have been fast. Now we have to flush these buffers. It slows things down,” he said.
A big part of Kroah-Hartman’s own job, maintaining the older Linux kernel releases, is, not surprisingly, building kernels, a very CPU-intensive process. “The more CPUs I have, the better it goes,” he said. Even with hyper-threading left on, he has witnessed about a 2% slowdown in CPU performance, with the new patches in place. When he disabled hyperthreading, that lag jumps up to 15%.
That said, each end-user should test their own workloads to see what the performance hit they get. Some workloads may not slow down at all, others may slow down a great deal. Users should also look at what choices their cloud providers made, Kroah-Hartman said. The site Make Linux Fast Again provides all the kernel flags to run raw Linux at full speed, giving users the ability to estimate the impact that patching will take place on their systems.
Lessons Learned
As a final topic, Kroah-Hartman delved into the alerting routine Intel set off by the discovery of the vulnerabilities. In summary: It could have been better, initially, and there is still room for improvement.
For last year’s first batch, Spectre and Meltdown — Intel only informed the Linux kernel community involved late in the process, he said. The distributions were caught unaware as news of the holes left Linux users vulnerable to attack. Intel disclosed the info to the distribution core maintainers through NDAs, not allowing different vendors to speak with each other about possible solutions, even after the news was made public. “It was a total nightmare on our part,” he said.
“Intel needs to bring Debian in earlier” — Linux kernel maintainer Kroah-Hartman.
The chip giant quickly worked out a more agile process for subsequent vulnerabilities however, alerting players in the community earlier in the process —before public disclosure. This allowed the Linux community to prepare patches to be ready when the researchers announced their findings. Though not everyone was included — Among the most notably missing was one of the largest distributions, Debian, which was only notified 48 hours prior to the release. To their credit, the Debian team scrambled to prepare patches.
“Intel needs to bring Debian in earlier,” Kroah-Hartman told the crowd.
Nonetheless, a lot of work had to come after the initial patches were sent out. The many ways in which Linux is used leads to many edge cases, ones that need corrective code. Something like Spectre doesn’t just require a single patch, but a whole stream of them in the weeks to follow. Kroah-Hartman issues a kernel update about once a week, with about 22 patches. At least one of those is related to security.
One interesting wrinkle is that users can not rely on CVEs to keep track of the kernel vulnerabilities. Typically the industry uses the Common Vulnerabilities and Exposures database to learn of security-compromising bugs. In the case of the Linux kernel, however, CVE reporting is only done after the fact — way after the fact. It shouldn’t be relied upon as the primary source for Linux vulns, Kroah-Hartman warned.
In fact, very few of the fixes in the kernel get CVEs. You can’t just “cherry-pick” the CVEs of Linux and expect to get it covered. You’ll miss all the follow-up patches. Spectre and Meltdown had dozens and dozens of follow-up patches, a slim margin of which were actually tagged for CVEs. And those that were tagged were identified, on average, about 100 days after the vulnerability was first found.”
Source: Greg Kroah-Hartman’s presentation.
CVEs “is not how the Linux security team works,” Kroah-Hartman said, pointing to a web page that explains how the Linux security team works.
In short, when it comes to kernel security, “The only way you can assure that you are running a secure machine is if you are using the latest LTS [long term support] kernel, or working with a distribution that does it themselves,” Kroah-Hartman said.
The Cloud Native Computing Foundation and the Linux Foundation are sponsors of The New Stack.
