Apache Mesos Narrowly Avoids a Move to the Attic (for Now)

I remember the first time I set up a Linux cluster. I felt like I’d created something magical like I had acquired some otherworldly skills and earned the title “l33t.” I had, effectively, taken three Linux servers and merged them into one. It was a thing of beauty.

Or so I thought.

Truth be told, it was far more common than I could imagine at the time. Admins were clustering hundreds of machines together on a level I couldn’t comprehend.

But I still did it.

The cluster in question was a Beowulf, powered by MPICH messaging. I was incredibly proud of getting that up and running and, to be fair, I don’t ever remember doing anything with the cluster. Just knowing I could deploy it was victory enough. That was the early 2000s; since then, better clustering tools came into being. In fact, since that time I’ve come to realize just how simple that original setup was.

We’ve watched clustering tools come and go. And any Linux admin knows that the traditional cluster has taken a back seat to the likes of Kubernetes. After all, containers are far more flexible and scalable. And, let’s be honest, in today’s world of cloud native computing, contains are far more viable than the traditional clustering platforms.

One such “pre-Kubernetes” platform is Apache Mesos. Mesos began as a University of California Berkeley open source project for next-gen cluster management. The goal was to learn from compute platforms like Google’s Borg and Facebook’s Tupperware. Unlike Borg and Tupperware (which were both monolithic), Mesos took a decidedly modular approach to the task. Mesos provided isolation for CPU, memory, I/O, file systems, rock locality and even offered a two-tier scheduling mechanism.

The guiding purpose of Mesos was to:

• Abstract data center resources
• Colocate workloads
• Automate deployment, self-healing, scaling, and upgrades
• Provide the ability to run new applications without modifying the cluster manager
• Provide massive scaling

One might be so inclined to think the above list was attributed to Kubernetes. Mesos could work with microservices and containers. In 2013, the founders of the project formed a company, called Mesosphere, which built an enterprise-grade platform from the technology, the Data Center Center Operating System (DCOS).  Soon after launch, Mesos was adopted by Twitter, Apple, Yelp, Uber, and Netflix.

Even with such an impressive lineup of companies using Mesos, the platform was nearly sent to the attic earlier this year.

Why did this (almost) happen? Let’s see if we can find out.

The Decision

On the apache.org Mesos mailing list, several threads showed two differing camps on what to do with Mesos:

• Send Mesos to the Attic
• Re-activate the project

For those that don’t know, the Apache Attic is where projects that have reached their End Of Life are sent.

According to Vinod Kone, who is the Project Chair of Mesos (PMC), the two camps came down like this:

• Attic: “Most existing PMC members who have chimed in so far seemed to be in favor of moving the project to Attic.” Kone continues, “The main argument for this seems to be that it’ll be hard to re-activate the project at this juncture with new PMC members/committers. Also that it signals the current state of the project more accurately.”
• Re-activate: “There are some active users in the community who would like to see this project stay alive and are even willing to step up to become committers/contributors.” Kone then states, “Some of these users are working for companies who are using Mesos in production. They would like to know potential new roadmap (there is a separate thread going on for this) and manpower needed (my take is 6-8 [people] to cover different areas of the project).”

Like every project that has reached this stage, it boils down to those who do not see any use in continuing the project, vs. those who are still using the project (thereby still see value in its continuation).

In the end, however, it looked as though no one on the Apache Mesos private mailing list showed any interest in remaining active with the project. Everyone knows, without developers, a project must go… even if there is still interest in the project. Why? It’s quite simple: A project may still function without developers, but without engineers adding security patches and bug fixes, that project could quickly become a security threat.

To that end, Kone said, “Since the existing committers are unable or unwilling to mentor new contributors into new committers, I think moving the project to attic is the right move. If there is no objection to this, I’m happy to call a vote for this.”

And so it looked as though Apache Mesos was destined to be relegated to the attic.

But then something interesting happened. In a new post on the Apache Mesos mailing list, Kone made this announcement:

Thanks for further responses and strong interest from a few folks in
keeping the project going!

I also had a chance to talk to some ASF members and it sounds like the ASF
preferred option forward in this situation is to keep the project going in
ASF.

Given all the above, I’m canceling this vote thread.

I will start another thread to elect a new PMC chair and let them handle
adding new PMC members/committers.

Saved at the eleventh hour.

Why Did This Happen?

In a word, Kubernetes captured the imagination of those interested in orchestrating distributed workloads. Bowing to the pressures of the marketplace, Mesosphere replatformed DCOS on Kubernetes in 2019, rebranding itself as D2IQ.

Nonetheless, Mesos remains a viable platform, with many production users. That being said, why was Mesos so near to being sent off into the sunset? Turns out, according to Vinod, the barrier to granting committer rights has led to the slowdown of the project. In fact, Vinod said of this, “The current guidelines we have for adding new committers is a pretty high bar and I don’t think any of the current contributors would be immediately eligible to be voted in as committers.”

So the Apache Mesos team had two choices:

• Change the guidelines.
• Allow some existing committers to mentor some of the contributors into committers.

In the end, it turns out the ASF recanted, in favor of a third option: Finding a new PCM chair so new members can be added to the project.

Mesos Is Dead. Long Live Mesos!

One of the great things about open-source software is clearly illustrated here. Apache Mesos could have been sent to the Attic to exist as nothing but fond memories of developers and admins. Fortunately, it missed that eventuality and will live on. However, even if Mesos was destined for the attic, it could have found its way back on GitHub as a fork, to live on under a different name with different developers.

It’s hard to say what will happen with Mesos after it finds itself helmed by a new leader. But for any developer (or team of developers), wishing to keep the project going, you might reach out to the Apache Software Foundation to find out what options are available.