How Red Hat’s License Change Is Reinvigorating Enterprise Linux Distros

At the Oregon Convention Center in Portland, mid-July saw the inaugural edition of the Software Freedom Conservancy‘s new conference — the Free and Open Source Software Yearly.

And its first-ever keynote panel turned out to be a lively discussion about Red Hat‘s recently announced plan to change its licensing for Red Hat Enterprise Linux (RHEL) source code —essentially hiding it behind a paywall for subscribers.

The question of the day: How should downstream Linux vendors respond?

The panelists clarified their respective plans for the future — and their philosophical positions on where principles of open source were being infringed.

But the discussion also captured some of the raw passion of the open source community — and the heat of this moment in time.

Clarifying the Situation

Leading the discussion was Bradley M. Kuhn, a policy fellow/hacker-in-residence for the Software Freedom Conservancy. The panel included representatives from groups affected by Red Hat’s decision:

• benny Vasquez, the Chair of the AlmaLinux OS Foundation.
• Jeremy Alison, Samba co-founder and a Rocky Linux-focused software engineer at CIQ.
• James (Jim) Wright, Oracle‘s chief architect for Open Source policy/strategy/compliance/alliances. Oracle has its own Linux distribution, based heavily on RHEL.

Kuhn first acknowledged there were some companies that weren’t represented. “Red Hat themselves did not reply to our repeated requests to join us on this panel,” Kuhn told the audience. (Although last week Red Hat reiterated its commitment to open source, telling Ars Technica “We would rather work together in CentOS Stream instead, where improvements are possible.”)

And SUSE — which recently invested over $10 million in its own fork of RHEL — “was also invited but let us know they were unable to send someone on short notice to Portland for the panel.” (Though SUSE’s announcement includes a statement from CIQ CEO/Rocky Linux founder Gregory Kurtzer that CIQ “is thrilled to collaborate with SUSE on advancing an open enterprise Linux standard.”)

But Kuhn’s first question seemed designed to clarify the essential situation. He asked the panel what exactly was wrong with just paying Red Hat’s subscription fee if you wanted to rebuild RHEL. AlmaLinux’s Vasquez explained that it wasn’t that simple. “The way that they have asked us to work with them is specifically not to do that — to not go sign up for an account and use that code and re-package it and ship it… ”

And Samba/CIQ’s Alison concurred. “Red Hat’s agreements make it very difficult to do that, because essentially what they’re asking you to do is to forego some of the rights that you’re granted under the GPL… What they’re asking you to do is to forego some of the freedoms that you as a user should have. And I don’t think that’s a reasonable ask, unfortunately.”

The SFC’s Kuhn joked later that Red Hat’s business model was “If you exercise your rights under GPL, your money is no good here.”

But Oracle’s Wright said there were two answers, and joked that he’d first answer “crassly, in the Oracle way.” Why not pay Red Hat to access the source code? “Because I don’t want to give you money!”

“But sort of beyond that, I feel like this is intrinsically and irreconcilably in conflict with the community norms, in which we are all stakeholders. The Free Software Foundation’s ‘free software’ definition requires that the software be freely re-distributable. The OSI’s ‘Open Source’ definition requires that the software be freely re-distributable and that the source be published.” There’s even a web page on Red Hat’s own site titled “What is open source?” And Wright said Red Hat wasn’t even following its own definition.

“The second sentence of the very first paragraph says ‘Open source software is code that is designed to be publicly accessible — anyone can see, modify, and distribute the code as they see fit’… When you close up the availability of that source, is that designed to be publicly accessible? When you provide it only under the terms of a contract which clearly provides that you cannot redistribute the code, can anyone see, modify, and distribute the code as they see fit? No.

“The conclusion is sort of unavoidable — that what they’re doing is not — it’s not consistent with anybody’s idea of what open source is, even their own. Until now…”

Alison praised Red Hat, “very much for creating what is considered the Enterprise Linux standard. And the reason that people want them — Alma and Rocky and Oracle — is because Red Hat has such a strong brand they’ve created around that, and they have to get a lot of credit for that.” But Alison describes Red Hat’s new stance as “come to Red Hat to get everything.”

“And that’s just not what the customers want. They want a range of options [for] purchasing enterprise Linux.”

Freedom and Standards

Alison reiterated this point later. “Customers want freedom… That’s the whole point of free software and open source. They want to be in charge of their own destinies, and they don’t want to have to go to one single source to get a product. I mean, If you wanted that, just buy Windows, right?”

Later AlmaLinux’s Vasquez made an even stronger point. “Red Hat has done a great job of establishing a fantastic target for all of us — but they don’t own the rights to enterprise Linux. We can make this happen, without forcing an uncomfortable conversation with Red Hat… We can get around this. We’re still going to build Enterprise Linux without violating or even having to fight with Red Hat. That’s how I feel about it. That’s what Alma’s doing. We’re just going to build it…”

Samba/CIQ’s Alison also believed the code would find its way out of Red Hat. “It doesn’t matter how hard Red Hat tries to make it for people to do this. People are going to do this. If they have to dig through CentOS Stream releases, figure out exactly what matches the RHEL bits — they’re going to do this. Never underestimate the tenacity of a bored programmer…”

And later AlmaLinux’s Vasquez reiterated: “We’re not afraid of digging around in source code. ”

When someone in the audience asked if the downstream distros might now add some new features not available in RHEL, Samba/CIQ’s Alison said “Yeah, sure,” and AlmaLinux’s Vasquez was also open to additions. “100% — yeah. One of the things that we’re kind of excited about is the opportunities that this opens for us… [W]e have all kinds of options.”

In fact, the panel was held just days after the AlmaLinux OS Foundation announced its decision to drop 1:1 compatibility with RHEL. (With the qualification that “We will continue to aim to produce an enterprise-grade, long-term distribution of Linux that is aligned and ABI compatible with RHEL in response to our community’s needs, to the extent it is possible to do, and such that software that runs on RHEL will run the same on AlmaLinux.”)

That blog post notes that AlmaLinux could now accept bug fixes outside of RHEL’s release schedule — an issue that surfaced later. And the blog post also points out that “we will include comments in our patches that include a link to where we got the patch that’s been applied.”

But on the SFC panel, Oracle’s Wright reminded the audience that the demand for RHEL compatibility was coming straight from customers — some of whom are targeting RHEL as a standard. “They only want to build and test on a single system.” Maybe their own customers are running RHEL.

Later the SFC’s Kuhn shared his own thoughts, that as a “software freedom zealot… my sympathy for people who really want compatibility so they can do proprietary software, is admittedly quite limited.”

But Oracle’s Wright pointed out the inconvenience also affects open source projects targeting the platform.

Customers and Competition

Perhaps the most interesting exchange came when someone in the audience asked what the odds were for creating a new open enterprise Linux standard that distributions could follow?

“Chances are real high…” answered AlmaLinux’s Vasquez. “This is a very new thing — we’re, what, three weeks into it? So I think everyone sees that as the obvious answer. I think that’s the obvious next step. I’ll leave it at that.”

Oracle’s Wright returned to his point that it’s customers who will ultimately decide. But Wright also said, “To the extent that the market evolves — answering your question — to the extent that the market asks us to standardize? We’re all responsive.”

Samba/CIQ’s Alison agreed that enterprise Linux “is what the customers say it is. And so if the customers say “Something that’s close to Red Hat but not exactly Red Hat is good enough,” then that’s what we will be. If the customers say, “No, it has to be a rebuild, bug-for-bug compatible,” then that’s what we’re going to try and be. We’re going to try and meet the market needs — we’re going to try and do what the users require.”

So if this discussion caught a snapshot of our moment in time, then what’s ultimately most telling is what each panelist chose to say in the final 10 seconds Kuhn allotted for their closing statements.

Oracle’s Wright re-visited a point he’d made earlier. (“So, we’re hiring a ton, right? We’re going to be hiring a lot, effectively, to have our own compatible distribution.”) He closed by reminding the audience, once again, that “If you want to work on open source Linux, we are hiring.”

AlmaLinux’s Vasquez said they were also looking for contributors. “Come hang out. We’re doing awesome stuff. I’m blown away by the support and excitement that we’ve seen. And it’s going to be a good time.”

And Samba/CIQ’s Alison took it one step further, sharing a joke he’d heard to describe how Red Hat’s move was ultimately self-sabotaging. RHEL is “the official OS of stepping on a rake.” So his closing thought was that Red Hat’s missteps would be good for all of its competition.

“They brought a great deal of interest and excitement around the Red Hat alternatives by doing this. So thank you! Thanks for that. I appreciate it!”