A Practical Approach to Detecting and Correcting Bias in AI Systems

As companies look to bring artificial intelligence into the core of their business, calls for greater transparency into AI algorithms and accountability for the decisions they make are on the rise.

That makes sense: If people are going to rely on AI to make important decisions with real world consequences, they need to trust it. But trust comes in many forms — and that makes it difficult to pin down. First, AI needs to explain why it made a particular recommendation. That builds trust because people understand the reasoning. Deeper levels of trust come from knowing that a system is fair and unbiased. Showing this part is much harder.

This leaves companies in a tough spot when it comes to leveraging AI: they can either fly blind or fall behind. In 2018, Amazon — a clear frontrunner in AI — shut down its experimental AI recruiting tool after the team discovered major issues with bias in the system.

What’s needed is a more practical approach. Here’s what 15 years building AI and machine learning models at companies like Facebook and Branch, and now HackerRank, has taught me about detecting and correcting bias in AI systems.

Focus on What You Can Control

The problem set surrounding bias detection and correction in AI systems is enormous. To make real progress on improving the fidelity of your AI systems, you need to break the bias problem into smaller parts.

Start by figuring out which aspects you can actually control. For example, let’s say you want to use AI to accelerate the developer recruiting process, drive better hiring outcomes, and foster greater diversity. In this scenario, bias can creep into the process in any number of ways — some you can control, others you can’t. For instance, while there isn’t much you can do directly in the near term to eliminate the systemic bias in the candidate pipeline at a broad, societal level (e.g. expanding diversity in early STEM education), there are actions you can take to optimize your own recruiting pipeline and process, such as checking for implicit bias in job descriptions and omitting names and pronouns from the interview feedback process to keep it focused on the relevant facts.

Amazon built its data model on resumes submitted over a 10 year period, most of which came from men. Without better baseline data, the algorithms developed a strong bias in favor of male candidates.

The key is understanding that eliminating bias from your AI systems entirely is not yet possible. I wholeheartedly believe that AI researchers will get there one day. Until then, it’s better to focus on solving more practical aspects of the problem — which can actually be far more effective than you might at first expect.

Identify Areas Where Bias Can Creep into the System

Generally speaking, figuring out what is in your control is an exercise in analyzing inputs and identifying areas where human error (ie. bias) can enter into the equation. While it’s a highly contextual process that varies from system to system, there are three particularly high-risk areas to examine:

Baseline data: If the data your AI runs on is biased, then the AI itself will also be biased. There are two key steps to establishing baseline data: (1) population sampling — creating an accurate model of the world based on limited data points; and (2) data conversion — converting data points into a format required by an AI system to learn from. Both are highly vulnerable to unconscious bias (unwittingly mishandling the data), ignorance (not knowing any better), laziness (a lack of vigor in scrubbing data), and explicit bias (inability to separate personal views from objectivity when doing any of the above).

In the case of Amazon, the company built its data model on resumes submitted over a 10 year period, most of which came from men. Without better (ie. more accurate and representative) baseline data, the algorithms developed a strong bias in favor of male candidates.

Directives: The more simple and specific the tasks are that you assign to an AI algorithm, the less likely it is that bias will skew the results. A relatively straightforward process such as determining whether an image contains a cat, for instance, is less susceptible to bias than a complex task like making a prediction on whether a candidate is a good fit for a job based on their resume.

As complexity increases, there are two types of bias to contend with. The first is “labeling bias” — to train an AI algorithm to predict whether a resume is a good fit for a job role, one needs to label many resumes, both good and bad. Here bias can show up if the labels are created by people who are unconsciously biased in some way or another, or if the overall hiring process is biased (ie. hiring mostly men). The second is “correlation bias” — the data collected might not be properly correlated with the labels, introducing bias. For example, hair color has little bearing on whether someone gets hired, but a correlation may still exist. In other words, it’s quite difficult to teach AI that correlation does not necessarily imply causation

Action: If the insights or predictions the AI delivers are ignored by the people who ultimately make decisions, those individuals’ biases will inevitably impact the system. This “last mile” is perhaps the most problematic. It’s extremely difficult to detect and correct bias in these situations because the decision happens outside of the system and cannot be measured with accuracy.

Apply Artificial Intelligence

Once you zero in on the parts you can control and identify where they are vulnerable to human bias, the next step is taking action. In my experience, creating a second layer of AI that monitors the first is a highly practical strategy.

Essentially, you want a set of AI systems in place that monitors every model and process that is vulnerable to bias and flag anomalies for closer inspection. For example, if your goal is to build a team with similar levels of gender diversity as the general population, then it’s relatively easy to build a system that tracks the delta between the general population and your applicant pool by pulling in trusted third-party data. If a gap exists between the two, there is a bias. You can then use statistics and/or AI to correct the model by ensuring the desired distribution of gender. Proceed with caution though — the general population you might not have the same diversity as your pool of candidates, so it is important to understand what the goal is and what you are normalizing to.

In some cases, such as in the previous example, the issue can be corrected quickly and autonomously by the system. In others, it makes sense to bring humans into the loop. For example, let’s say your system tracks the deltas between (1) your applicant pool, (2) the people you ultimately hire, and (3) the general population, and there’s a small gap in gender diversity between the applicant pool and the general population, but a large one between applicants and hired candidates. Clearly, bias is creeping into your system — but where? Since this process involves many steps, a human is likely to figure it out much faster. Once they do, you can add AI oversight to the problem area to monitor for future biases.

It’s important to note, however, that while adding a human into the loop can improve outcomes in certain respects, it also introduces new opportunities for bias to creep into the system. To help mitigate this risk, consider adding additional layers of bias monitoring on top.

As these examples illustrate, there are many creative ways to detect and correct bias in AI systems using the technology available today. Just because there’s no perfect solution, it doesn’t mean you have to sit on the sidelines. But to do it right, it’s critical to understand that AI is only as good as the data it runs on, the directives you give it, and — perhaps most importantly — the humans that interact with it. So go ahead and start building — and be relentless in rooting out bias every step of the way.

Feature image via Pixabay.