Linux’s Rocky Road to Rolling Releases
The way the keepers of Linux increment the version numbers of their kernel releases, and what many of its users think these release numbers mean, are two different things. And now the Linux kernel community is trying to reconcile public perception with reality, according to a talk given by Google senior staff engineer and Linux kernel contributor Sasha Levin at the Linux Foundation Open Source Summit last week.
Currently, a new Linux kernel is released every 10 weeks, more or less. There are no “major” or “minor” versions any longer. The latest release, 5.14, was no more or less significant than the 5.0 release in 2019. Organizations are just as safe updating to what would appear to be a major version as much as they would be updating to an apparent minor release.
Now each release is just a collection of updates that made it into that particular 10 week-window. The kernel developers are confident that each new version of Linux, no matter how seemingly major, will not introduce backward incompatibilities, and most all of the performance regressions that each new version will invariably bring will be hammered out in the eight-week release candidate cycle preceding each new release.
Many enterprise Linux users, however, still plan their upgrade cycles around the mistaken notion of major/minor release, Levin noted, reading into the release schedule that assigns version numbers without regard to the importance of the release.
Even though it would appear that Linux follows standard semantic versioning rules, where software version numbers are based around the format of major.minor.patch, where a 5.0 represents a major update with potential application-breaking API changes, but where the minor (5.1) and patch (5.0.1) are OK to slip into production without major testing.
The kernel community also continues to support long-term support releases on behalf of cautious users (5.14 is the latest LTS), even though they feel such releases are redundant, even harmful. The team back-porting fixes to LTS versions for the six years following the release. But these tend to create “Frankenkernels,” Levin says, which are not really any more stable the latest versions.
Every Day Is a Winding Road
When Linus Torvalds first released his “Linux” hobby project in 1991, he took the usual route of applying the typical semantic version numbering of releases. The first was 0.01. Larger releases got bigger bumps in versions than smaller ones. This went one through version 1 when there was a sufficient amount of features “to pretend to be an operating system,” Levin said.
With version 1 came actual users — users who were not kernel developers. They wanted to build applications and run workloads on top of the kernel. This forced the kernel developers to crave off a development branch, which would be separate from what would hence become a “stable branch.” The developer branch is where new ideas get tested, the stable branch in effect was the last known working copy of the kernel that outside users could rely on. Version 2.4, for instance, was a development branch, and 2.3 was the stable branch, where only backward fixes are applied.
As more people contributed to the kernel, however, the release cycles between 2.0 and 2.6 of the kernel nearly ground to a halt. There was no central management to declare when the next release would be. Just going between 2.4 and 2.6 took two years.
Google’s Sasha Levin
Fearing the release cycle would slow even further, the kernel community decided on moving from feature-based releases to time-based cadence, a release every 10 weeks, no matter how small or how large the release would be.
“We started saying version numbers don’t matter because at this point they release don’t. We just release every 10 weeks,” Levin said.
With this change came the end of the stable branch. Now, every release is a “merge window that takes two weeks followed by stabilization period, over a list of weekly release candidates for about another seven, eight weeks. And at the end, we can say that the kernel release is stable enough for customers to run,” Levin said.
This is also when the Long Term Support (LTS) was created, for those who wanted support — bug fixes and whatnot — for a time beyond the usual support window. In the timed releases, all the changes were merged into the new release; with LTS and the fixes were backported to the LTS version.
So this approach worked really well, Levin felt. But where they had left off, at 2.6, they kept wrangling the old versioning format to the new releases. Some major feature additions got minor release numbers and vice versa.
Users got confused, at least those who still imparted meaning upon the version numbers. For them, a jump from 2.6 to 3.0 would require lots of testing and adjacent upgrading, they assumed.
An upgrade to such a major version, in the minds of many enterprise CTOs, requires a multi-year migration process with all the associated paperwork. They can only take on such a major upgrade project when they have the time, which was basically never. So they stuck with earlier tested versions, instead of migrating much more quickly and more often. So they tended not to upgrade to version 3.0, even though it was a relatively minor release (much to the shock of the trade press).
“Changing the major version is big and scary,” Levin said, even though in this case it was just a routine release that served to be the jumping point off of the 2.6.* series. The jump to 3.0 also came with another set of issues for its users. Maybe even some of their own configuration scripts had version 2.6.* baked in.
On the other hand, users seemed to be OK updating to minor versions, which in its own way, also added to the confusion. Moving from 2.6.70 to 2.6.71 didn’t seem to require as much fretting as moving from 3.0 to 4.0. Though such incremental numbers also (falsely) signaled the unimportance of upgrading to such a seemingly minor update.
Infinity and Beyond!
The 2.6 chain of releases also raised the question, when would they jump to the next major version number, beyond 2.6.x? Or would they just continue to add on to that chain off into infinity?
Torvalds himself has said he would like to keep releases to 20 before a major jump in version number because 20 is how many he can count on his fingers and toes. This scheme, too, is completely arbitrary and furthers the disconnect to how users view releases, Levin notes.
But he admitted that this fault lies at the hands of the kernel community, who haven’t done enough to inform users why sticking with stable or long-term releases is unnecessary.
Levin said that he himself would like to do away with the LTS releases, as they are redundant. And some users are finally getting the messages. He noted downloads of LTS release are starting to diminish. Some distributions, such as Debian and Arch Linux now replicate the 10-week release cycle within their own distributions. Android follows closely behind. “They understand that it’s just a time-based schedule,” Levin said.
Competitive organizations, eager to get the latest features and performance improvements, are letting go of the LTS releases. “They don’t want to be on an old kernel. They have products they want to sell to,” he said.
