Mozilla BugBug Adds Machine Learning to Bugzilla
-
Emma Humphries, Bugmaster at Mozilla
Adding machine learning (ML) to Bugzilla is solving several problems, said Emma Humphries, Bugmaster at Mozilla, the organization behind the Firefox open source web browser. Bugzilla is a 20-year old bug tracking system where engineers and Firefox users can submit bugs they find.
BugBug, Mozilla’s new ML platform, automatically assesses each new bug and automatically assigns it to the correct product and component. This is just the first phase of this work.
Bugzilla is a noisy data source, wrote Marco Castelluccio on his blog at GitHub titled “It’s not a bug, it’s a feature.”
“Bugs are used to track anything, from ‘Create an LDAP account for contributor X’ to ‘Printing page Y doesn’t work,’” He wrote. “This makes it hard to know which bugs are bugs and which bugs are not bugs but e.g. feature requests, or meta bugs, or refactorings, and so on.”
Bugzilla uses a classification system for bugs, which tag every bug. Sixty percent of the bugs are tagged in a general component by users submitting the bugs, but bugs were only being picked up to be worked on from a specific component.
“People just weren’t picking up bugs that were general,” said Humphries.
Castelluccio wrote in the “Teaching machines to triage Firefox bugs” post, that “To help get bugs in front of the right Firefox engineers quickly, we developed BugBug, a machine learning tool that automatically assigns a product and component for each new untriaged bug.”
Not all people who submit bugs are engineers, so there’s a danger in training BugBug on bugs that were written by people who come from different backgrounds, said Humphries. It’s not like they could dump in data from the 1.5 million bugs and simply train off of that.
The challenge, she said, is how to use machine learning wisely.
Modernizing Bugzilla
But before the BugBug could be created, Bugzilla itself needed some fundamental changes.
What Humphries inherited when she arrived almost four years ago was a system that had been adapted and customized in ways that worked for the immediate needs of various groups throughout Mozilla, but made it impossible to scale. And with over 1.5 million bugs in the system, scaling was no longer an option.
Because Bugzilla had a database and access control list, it could be used to track both bugs and tasks. Twenty years ago, SaaS wasn’t a thing, so over the years, the system was adopted by teams across the company, from HR to legal and even for mascot requests for Foxy, all of whom asked for customizations to fit their specific use cases. These were built on an ad-hoc basis and Humphries inherited an unwieldy system desperately in need of cohesion and modernization.
Another issue was the way bugs were prioritized. It turns out, Humphries said, that voting is a rotten way to prioritize features. As Luis Villa noted, “Votes end up having no bearing on actual bug validity, importance, or severity.”
Then there was the issue of exploding metadata.
Humphries started by taking a step back and reviewing the system as a whole, starting with metadata. Over time there’s so much optional metadata that’s been added that “I started going through the metadata and asking ‘is this necessary?’,” she said.
Humphries also revisited what it means to track a bug throughout the entire system. She asked fundamental questions like, are keywords the right way to indicate a bug? Would it be better to have a multistate flag?
Because a question about a bug being a regression is non-binary, there’s several questions to be answered about how a bug is flagged, she said. “What you really want to know,” said Humphries, “is ‘which commit broke it?’”
Structure
Bugzilla was built-in TK/TCL, then ported over to PERL with Apache ModPerl and MySQL as a backend. Twenty years ago, this was cutting edge.
Tracking flags were added over five years ago. These flags can be attached to a product or a component. And they built their own multivalue flags, used mostly for fine-grained release management tracking within Mozilla.
The foundation is solid, Humphries said, so after her review, she decided against doing a teardown and re-build. She’s just remodeling and modernizing what’s already there.
Part of the challenge for Humphries and her team, as with all upgrades, is getting the Mozilla staff and volunteers to shift their perspective on change itself.
“I needed to get them used to the idea of experimenting with experimenting. There’s so much pushback against any little change,” she sighed.
To aid in this venture, Laura Thompson, Mozilla Senior Director of Engineering, coined the phrase “Make More Cupcakes.”
Because cupcakes are smaller than cakes, they are easier to consume and take less time to bake. They applied this analogy to the new features as well as product development itself.
“Incremental changes are more consumable,” said Humphries.
The product has three release “trains” daily. Engineers and bug fix volunteers can drop a feature into Normandy when they’re done.
Normandy sends an encounter (recipe) and if the client confirms a match, it’s dropped on the next train to release on its own schedule. As long as you can talk to Normandy, she said, you’ll get the recipe.
The beauty of the system, she said, is that rolling back simply means sending out a new recipe not a point release. “We’re sending rules, NOT code,” she said.
‘Beware of the Leopard’
Humphries insisted that all the changes be well documented and easy to find and use. To remind her team of this, she adopted a “Beware of the Leopard” tagline.
In the Douglas Adams’ science fiction classic, “The Hitchhikers Guide to the Galaxy,” Arthur Dent wakes up and sees a bulldozer ready to take down his house so they can build a freeway. He was told that the plans for this were in the council office in the basement, in a cabinet in the restroom with a sign saying “Beware of the Leopard.” The lesson: Don’t bury your documentation.
Work on BugBug is ongoing. Humphries is excited to add new learnings to make BugBug useful across a variety of projects.
Feature image by Lolame from Pixabay.
