What news from AWS re:Invent last week will have the most impact on you?
Amazon Q, an AI chatbot for explaining how AWS works.
Super-fast S3 Express storage.
New Graviton 4 processor instances.
Emily Freeman leaving AWS.
I don't use AWS, so none of this will affect me.
Open Source / Tech Life

How ‘Free Riders’ Make the Musl Open Source Project Grow

Some slag open source users off as “free riders” or “takers,” but for Musl, they’re the group that makes open source work.
Jun 16th, 2020 9:24am by
Featued image for: How ‘Free Riders’ Make the Musl Open Source Project Grow
This is part of a series on Open Source Builders. For a list of other articles in this series, check out the introductory post.

Amazon Web Services (AWS) sponsored this post.

Among the standard open source beatitudes (“Blessed are the… for they shall…”), basically no one remembers Rich Felker’s addition: “Blessed are the open source users, those that don’t contribute a single line of code, for they shall make your project prosper.” For Felker, founder of the musl libc project, users of his project may well trump all others — including code contributors — usually considered the sine qua non of open source communities.

Some slag open source users off as “free riders” or “takers,” but for Felker, they’re the group that makes open source work. After all, widespread use of open source software is what encourages others to do the hard work of ensuring interoperability, among other benefits.

A Proof of Concept

Matt Asay
Matt is a principal at AWS and has been involved in open source and all that it enables (cloud, machine learning, data infrastructure, mobile, etc.) for nearly two decades, working for a variety of open source companies and writing regularly for InfoWorld and TechRepublic. You can follow him on Twitter (@mjasay).

Most people won’t have heard of musl, an implementation of the C standard library built on top of the Linux system call API. Geeky stuff. As Felker explained in an interview, musl is the most basic, lowest level library code in userspace. In other words, it’s the “glue” or, if you will, the bridge between applications and the Linux kernel. Musl provides the interfaces these applications need to talk to the Linux kernel so they can run.

When Felker first started tinkering as a student in 2005, there were other libraries that developers could use to interface with the Linux kernel, including uClibc and glibc. The former lacked a stable API and the latter was difficult to upgrade, with a community that was, at the time, “not very friendly to criticism or bug reports from users.” A few years later Felker decided to build musl libc, not to compete with existing libraries, but rather as a place to showcase proofs of concept for what glibc could become.

“I wasn’t really imagining at the time that it would take off and be used by lots of people,” Felker says. (Narrator: Musl libc would, in fact, take off and be used by lots of people.)

Despite musl’s popularity, it’s always been a labor of love for Felker, not a “big income-producing activity.” There’s the occasional short-term engagement to improve musl in some way, but that usually only happens if a client wants something big that Felker and the musl community already intended to build. For the most part, therefore, most of Felker’s time spent on developing musl gets funded by Patreon or GitHub Sponsors. “Both of which have been really helpful in making this something that I can focus on,” Felker says.

‘More Valuable Than Writing Any Code’

Though Felker does much of the musl development, a community has grown up around it over the years. It’s a community, Felker says, that is welcoming to everyone. It’s not somewhere people feel like they’re going to be attacked or harassed. This isn’t an accident. Likely in response to glibc’s reputation for being allergic to contributions and the Linux kernel’s somewhat toxic reputation, Felker has gone out of his way to do the simple things, including the simplest of all. “The important thing is to really thank people. Convey how much you value what they’re doing,” he says.

Even, it turns out, people who stop by to use the code and give nothing back.

Except that, in Felker’s mind, they do give something back, simply by using musl. In addition to thanking people, Felker says, he goes out of his way to publicly correct people when they ask something like, “Who are the real contributors?” or, “Who’s writing code?”

Why? Because Felker says those writing the code are not the real contributors. “That’s a very small part,” he explains. “The real contributors are the people who are making this possible by telling us when something doesn’t work or coordinating between different people with potentially conflicting interests, to figure out how to fix the problem when something doesn’t work.”

Say that again? “The users, the testers, the adopters, the bug reporters are so much more valuable than writing any code.”

Indirect Code Contributions

While this may seem counterintuitive, it makes perfect sense in musl’s world. Though the C language is small, it quite possibly has the largest interface surface of any software library anywhere. Think of all the software written in C, or written in another language but whose runtime is built on C, including more than 50,000 open source programs. Every single one of them has expectations for how standard interfaces like musl should work. Without lots of users demanding upstream compatibility with musl, Felker says, “I would have had to spend 10 to 100 times the amount of time I actually have to get patches and fixes into the hundreds or even thousands of other software products that would otherwise have musl build problems. This just wouldn’t be possible. But with lots of people using it and wanting things to work and not wanting to have to maintain their own outer tree patches with their own builds, it all happened. People got in touch with the upstreams early on.”

Such users are “free riders” in the sense that they’re not contributing code back, sure, but their very act of using musl prompts others to write code that will ensure compatibility. No wonder, then, that Felker “considers the people who’ve built distributions based on musl and reported every problem they found in musl, reported things to upstreams of other projects when they found portability problems that needed to get fixed, doing all that work of coordinating between different parties; the most important contribution that’s happened to musl.” Such contributions started early on with musl and resulted in a situation where, today, most everything just works out of the box with musl.

Or, as Felker later emphasized, users create demand for other software to interoperate, and may fix interoperability with other software that musl’s code contributors may never see. Such “contributors-by-using-your-software” could, after all, simply use something else, forcing a project to do the incredibly arduous work of doing all compatibility testing itself. By contrast, Felker’s policy of thanking those who contribute to musl simply by virtue of using musl is not merely kind. It’s wise.

So how can you contribute to musl? Well, use it, of course. But there are other ways. Felker would love to spend more time with musl, as well as participating in the standards processes around the C language and POSIX. If you or your company would like to help contribute to musl’s ongoing success, consider becoming a sponsor through Patreon or GitHub Sponsors.

If you or someone you know needs help with a different open source project, visit the AWS Open Source site to learn how open source projects can apply for AWS promotional credits.

At this time, The New Stack does not allow comments directly on this website. We invite all readers who wish to discuss a story to visit us on Twitter or Facebook. We also welcome your news tips and feedback via email:

Feature image via Pixabay.

Group Created with Sketch.
TNS owner Insight Partners is an investor in: The New Stack.
THE NEW STACK UPDATE A newsletter digest of the week’s most important stories & analyses.