Hacks to Help Open Source Leadership Support Inclusion
We know open source has a diversity and inclusion problem. In fact, open source falls well short of the poorly performing tech industry as a whole by all measures of diversity, equity and inclusion (DEI), in part, because it relies on a significantly voluntary workforce.
We also know that the majority of maintainers are overworked and underpaid. Yet DEI remains siloed as a social problem. That’s a mistake.
Open source is the backbone of the tech industry, making it an inherently sociotechnical concern. That lack of diversity and inclusion echoes across community and code. But it’s possible to automate part of open source community management — like we do with technology itself, to reuse models and metrics for success.
Can’t Improve What You Can’t Measure
One of both of the benefits and challenges of open source communities is they can be as distributed as their codebases. In this remote-first world, it’s not easy to get a feel for even more visible signs of diversity, like you could in a colocated office or event.
And time-starved project leadership don’t always get a chance to survey their community as much as they’d like. Even when they do try to get insights into their communities, as Fábio Santos found in his recent research out of Northern Arizona University, classification is restricted to the nature of the issue — bug/non-bug, questions, documentation and the like.
The Linux Foundation’s Community Health Analytics Open Source Software project, or CHAOSS, was developed to create metrics and surveys in six buckets:
- Common metrics: contributions and the people who make them, where contributions are made (GitHub, chats, forums, events), time to respond.
- Diversity and inclusion: event diversity, governance D&I, community leadership health, community health.
- Evolution: the software development lifecycle of a project.
- Risk: project sustainability including licensing, security, code quality and dependencies.
- Value: offered to social, organizational, individual, community and academic stakeholders.
- App ecosystem: geared toward newcomers to building ecosystems.
While only one is specifically geared towards accruing diversity and inclusion metrics, all six topics interrelate and contribute to a sustainable open source community.
“Having diverse communities is important for the health of a project,” Amy Marrich, member of the DEI Working Group and CHAOSS board member, told The New Stack.
“By collecting metrics you get a better picture of where your community is, especially in a larger community where you can’t possibly know everyone within the community.”
CHAOSS, she said, doesn’t run surveys but simply creates the questions for open source projects to build their surveys on. In October, the CHAOSS community also released the beta of Augur, a Python library and web service to support data collection and measurement, including dashboarding, for open source project leadership.
Badges to Signal Inclusion
Traditionally, recognition in open source communities is relegated to technical contributions, like leaderboards based on a number of pull requests (PRs) made, as drawn from GitHub contribution graphs. However, participation in open source relies on non-technical contributions too, including documentation, translation, and community development and advocacy.
“When we talk about contributions, we’re talking about everything, we don’t just focus on PRs,” said Ruth Cheesley, sharing her experience as open source project lead for Mautic, a marketing automation tool, at Acquia, a digital experience platform
“In fact, a lot of our community are nontechnical marketers, because the software is marketing automation,” Cheesley told The New Stack. “A lot of them contribute in other ways, like writing or reviewing or running a podcast.”
But how does open source leadership, perpetually short on time, keep track of that? Demetris Cheatham, senior director of diversity and inclusion strategy at GitHub, said this request to facilitate acknowledgment of non-code contributions was a major finding from last year’s Maintainers Listening Tour.
GitHub Achievements was launched this past June, as “the next layer of contribution graph for volume and badges,” Cheatham said, describing it as, “let’s unpack what you’re doing, through a different lens.”
GitHub is rolling the badges out gradually, to help prevent gamification of its recognition algorithms — though, of course, someone has created a repo tracking badges. However, besides Galaxy Brain — which encourages answering community questions — the badges released thus far require pull requests, which in turn requires technical know-how.
However, there is a series of Pair Extraordinaire badges that seem to encourage collaboration with other contributors. These badges offer contributors proof of not only technical prowess, but the in-demand, but often intangible core skills, like teamwork.
The next step, Cheatham said, is to scale this sort of automated recognition up to the community, project and organizational levels. Her team, in collaboration with the Linux Foundation and its Diversity, Equity and Inclusion in Open Source Survey, has started to reveal general steps toward creating an inclusive community:
- Basic: Having a code of conduct (which GitHub recently made more visible within repos), turning on closed captioning for meetings.
- Next: Publishing documentation, and a contribution ladder to define progression within a community.
- Mid-level: A triage process so contributors know which issues to pick up, including flagging tasks for newcomers versus more experienced
- Senior or Unicorn Inclusion: Maintaining inclusive and diverse leadership, enforceable codes of conduct
“Once we build out what’s happening at each level, we’re also looking at who can we pay — developers, nonprofits, whoever — to help us develop these things,” Cheatham said.
The CHAOSS project also offers Diversity and Inclusion Event Badging, which acknowledges both in-person and online open source events that meet set demographic standards on and off the stage, as well as other checkpoints:
- Feedback, as an important way of fostering an inclusive experience across all participants.
- Code of conduct enforceability.
- Family-friendly facilities, like breastfeeding rooms and child care services.
- Accessibility requirements.
- Free event tickets for people who are marginalized in the tech industry.
- Retention rates.
The badging process is manual; CHAOSS is planning to extend this D&I process to project badging as well, according to Marrich.
A simple way for open source project leaders to start noticing active contributors beyond their pull requests is via their Slack Analytics. If premium users, they can see who is overall most active.
Bots to Battle Discriminatory Language
“Phrases rooted in racism, sexism, and other forms of discrimination have made their way into business vocabulary. Most of us do our best to use inclusive language, but sometimes we slip up — or we don’t even know that what we’re saying isn’t inclusive,” wrote Dannielle Sakher, in her article on the Zapier blog about why to build an inclusive language Slackbot. (A Slackbot is a chatbot built for the Slack communication platform.)
One area where we’ve seen a lot of automation in the last couple years has been around inclusive language, via tools that not only flag exclusionary written language, but explain why that language can be offensive and what alternatives exist.
The kind of Slackbot that Sahker described in her blog post is slightly different from others because it is done publicly, instead of a personal nudge. It “holds everyone in the channel accountable, discouraging others from using the word or phrase in the future too. This is something that a direct message (or an autocorrect) can’t do.”
A Slackbot programmed to nudge users to use inclusive language.
Other existing inclusive language tools, which all privately nudge, include:
- Inclusive Bot: A Slackbot that pings users with alternative language, while also adding a reaction (a rainbow above) so Slack community managers know it’s being handled. It costs up to $199 a month for communities over 50 users.
- AllieBot: A Slackbot more focused on offering a way for users to provide ongoing anonymous and non-anonymous feedback as well as anonymous pulse surveys. It also provides suggestions of micro-training, content and advice to encourage more inclusive behavior.
- Witty.works: Like Grammarly but for inclusion, too. Browser-based Witty, available for English and German, is free for up to three users so may be good to start with project maintainers, but aimed more as an enterprise team product. Also features branding enforcement.
- Dei.ai: A free Slackbot that nudges folks in the inclusive direction, even warning about metaphors like sports lingo, which would also help improve comprehension for non-native English speakers. There’s also a Chrome extension.
A DEI bot offers alternative suggestions for more inclusive language.
But bots are of limited usage in nuanced, real-world situations. “Simpler attempts at automation can be surprisingly ineffective,” said Coraline Ada Ehmke, creator of the Contributor Covenant, a code of conduct for open source communities.
“Consider the example of the ‘guys’ bots that were/are pretty popular in Discord and Slack,” Ehmke told The New Stack. “It seems simple and straightforward — the bot would be triggered by any message that used the word ‘guys,’ as in, ‘Hey guys, what time is the meeting?’
“The idea is to interrupt or draw attention to gendered language. But, the bot would trigger on the word and have no conception of [intent], leading to lots of false positives and generally becoming so annoying that the bot was eventually removed altogether.”
In fact, after initial adoption in 2020, there’s been a drop-off in usage of such tools, including within the Cloud Native Computing Foundation (CNCF) Slack community. Furthermore, the Inclusive Naming Initiative that was touted by several large open source organizations in 2021 seems to have done very little before fading away. If you want a much more thorough list of inclusive language, Self-Defined continues to be the most expansive service available.
“Inclusion comes at the community level. Maintainers and community leaders influence the inclusion. It’s that manager that’s influencing that culture.”
— Demetris Cheatham, senior director of diversity and inclusion strategy, GitHub
Indeed, pushing people to use inclusive language doesn’t mean they’ll use it, but it does automate a lot of the baseline inclusion education work of community managers and project leadership. Most of these tools aren’t free and most are English only — a consistent barrier to open source participation.
GitLab, the open source repository, open sources its translations to over 70 languages, and includes guides to translation and inclusive language — like formal and informal versions of “you” — within its documentation for English, German and French.
This advice could potentially be integrated into a tool like witty.works. In general, don’t forget to poll your users to know which are their main languages — any DEI effort should be translated for maximum impact, with translated push notifications reaching scale.
Another code repository, GitHub, is now available on mobile, which enables far more of the developing world to leverage and participate in open source and is available in seven different languages.
Code of Conduct Nudges
The code of conduct remains one of the most important baseline requirements for those marginalized by the tech industry to join open source communities. We’ve already covered how Vandana Singh, an associate professor at the University of Tennessee, has researched and documented the impact of codes of conduct. She has deemed them not only necessary for DEI success in open source, but that those customized for their audiences are even more successful.
The struggle is not only that codes of conduct exist, but that they are enforced. “There’s a key indicator to enforce: we see you, we value you, we do not tolerate XYZ,” she said.
Since a code of conduct is written, there are options to automate some of at least lower-level enforcement. However, we only found one example of a community doing so. The Drupal open source community has push nudging to encourage code of conduct adherence and to discourage harassment, or what could escalate to become harassment.
“In Drupal, they have a bunch of pre-worded nudges that you can copy and paste if someone’s doing a particular behavior, to just say, like, ‘Hey, let’s deescalate this,’ to try and bring things back before they get too far,” said Cheesley, a long-term open source contributor and external open source ambassador for the Drupal Community Health Team.
Drupal also keeps a log of every time a nudge is used in order to review effectiveness. It’s helpful, she said, because as an ally you can’t always know how to call out behavior.
“The Drupal Community Health Team has put in a lot of effort in crafting various ‘nudges,’ focusing on using non-escalating, inclusive language,” said Michael Anello, a Drupal developer and trainer. This includes discouraging gate-keeping of knowledge, like nudging alternatives to all-too-common phrases like “I can’t believe you don’t know how to do this” or “This is so easy.”
Overall, Cheesley isn’t sure if she is seeing more low-level activism in open source communities because of these nudges or because of overall increased awareness since the social justice reckoning of 2020. Either way, she welcomes it.
“Because of the general raising of awareness, of racism or bias, of all these kinds of things, in most of the communities that I’m active in, people are calling that kind of behavior out and I’m learning from it,” she said.
Software-Assisted Governance
Ehmke, who has previously experimented with prototypes around code of conduct reporting and enforcement platforms, pointed to the emerging trend of software-assisted governance.
The Metagovernance Project is an interdisciplinary research collective that collaborates to build standards and infrastructure for digital self-governance within individual communities and meta-governance across interconnected communities. Metagovernance collaborations range from the internet itself to Web3 and crypto to ethics.
Three experimental tools that have emerged from it are:
- CommunityRule, a governance toolkit for great communities, led by Nathan Schneider.
- PolicyKit, a tool for building and evolving governance in online communities, led by Amy Zhang.
- Kybern, a website for collectively managing communities and resources, led by Shauna Gordon-McKeon.
Sociotechnical meta-governance also derives from a need for better awareness and communication among different open source projects. Cheesley participates in the Open Source Community Health call, which she refers to as the United Nations of open source, in which maintainers and community managers share best practices, including codes of conduct violations.
Because the open source community is so inherently distributed, it’s not uncommon for someone to be kicked out for violating the code of conduct in one community, and then to go to another one with the same behavior. “Because we don’t talk,” Cheesley said.
Can You Automate Onboarding?
There are also indirect ways to support community inclusion — or to automate and optimize some essential open source leadership tasks, so limited focus can be better allocated to DEI concerns.
One important inclusion task is welcoming new members and thanking them for their first contributions. Setting up pings and push notifications or flagging pull requests can help create the necessary urgency, reminding maintainers to do that extra manual step — which means much more when it comes from a human rather than a bot.
Pinging project leadership about first-time contributors has many benefits, such as developing a feedback strategy and encouraging repeat contributions. Much of this can be set up through GitHub’s organization API, brainstormed Kubeshop’s Matheus Nogueira in the CNCF Slack group.
In his work on the Tracetest observability project, he has blocked messages from the core team, so that they are only pinged for outside contributors.
“Any other person that opens an issue will make the pipeline to send us a message to notify us,” Nogueira told The New Stack. “It works fine for small teams. For large teams, it would probably be better to filter, using either the project code owners or the Github’s organization API.”
Of course, as we’ve already written about, there is an increased security risk for malicious injections that target regular contributors, so security review should be over any and all contributions.
Santos, the machine learning analyst at Northern Arizona University mentioned earlier, recently published research examining the task assignment mismatch between newcomers and existing contributors and the potential of task-driven skill identification.
“Selecting an appropriate task is challenging for contributors to open source software, mainly for those who are contributing for the first time,” Santos told The New Stack.
“Therefore, researchers and OSS projects have proposed various strategies to aid newcomers, including labeling tasks.”
Labeling issues is a common indicator for technical level and priority. But, he said, it is often challenging and time-consuming. There’s also a misconception that a newcomer to a project is a newcomer to software development, and communities can misdirect technical talent to less important issues.
“One approach is to automatically label the issues [or] tasks with categories of APIs — instead of labeling the APIs directly,” he said. “Therefore, we focus on predicting the domains of the APIs, i.e. API-domain labels, which are defined as high-level labels designating categories of APIs or skills,” such as a user interface, database, etc.
“If the contributors know which types of APIs will be required to work on each issue, they could choose tasks that better match their skills or involve skills they want to learn.”
With all this in mind, Santos’s team is now applying machine learning and semantic taxonomy — trained on individual contributors’ previous public contributions across GitHub — to newcomers’ technical task assignments. So far, they have reached 84% accuracy in matching contributors’ skills to tasks. The automation of these activities would significantly lighten the maintainer load and take better advantage of drive-thru contributions.
Another benefit to labeling good issues for beginners to tackle is that it can be picked up by platforms like Good First Issue, which automatically pulls issues labeled “first contribution” or “beginner” from several popular open source projects. If you enroll your open source project in a platform like this, you can attract new volunteer contributors more easily.
You Can’t Automate It All (or Much, Yet)
Of course, the lack of diversity, equity and inclusion in open source communities isn’t just a technical problem. You cannot and should not automate everything. However, the open source world should continue to look for ways to lessen the burden on overworked project maintainers and to leverage automation and other technical solutions when possible.
“It is really impractical — at least at this point in time — to automate something as complex and nuanced as social interactions,” said Ehmke. “And any kind of automation is pretty easily abused and weaponized against the people it’s intended to protect. For example, Twitter is easy to game by piling on false abuse reports from multiple accounts and thus triggering auto-lockouts or bans on their target’s account.”
Indeed, the examples above are Band-Aids, not cure-alls The open source community still has a lot to reckon with in terms of DEI, including the continued reliance on a largely voluntary workforce. Companies need to invest in the open source technical stacks they are working on, so short-staffing issues are solved and community leadership can focus on these essential issues.
“It’s about creating the mechanisms for prioritizing DEI and promoting good outcomes, through fair and transparent enforcement of community norms,” Ehmke said. “These mechanisms are fundamentally about transparency, responsibility, and accountability, which I think are fundamental to equitable collaboration, sustainability and pro-social outcomes.
“The trick is not to automate away human attention, understanding, and intervention, but rather to ensure that the work that humans have to do in this regard is as safe, reliable and friction-free as possible.”
For more on DEI and open source, check out this podcast on open source contributor strategy, recorded at KubeCon + CloudNativeCon Europe, in Valencia, Spain.
Do you have more examples of technical solutions to diversity and inclusion in open source? Let us know @JKRiggins and @TheNewStack on Twitter.
