Mar 9, 2017 · whitepaper

FairML: Auditing Black-Box Predictive Models

Fairness in ML

Machine learning models are used for important decisions like determining who has access to bail. The aim is to increase efficiency and spot patterns in data that humans would otherwise miss. But how do we know if a machine learning model is fair? And what does fairness in machine learning mean?

In this post, we’ll explore these questions using FairML, a new Python library that audits black-box predictive models and is based on work I did with my advisor while at MIT. We’ll apply it to a hypothetical risk model based on data collected by ProPublica in their investigation of the COMPAS algorithm. We’ll also go over the methodology behind FairML at a conceptual level and describe other work addressing bias in machine learning.

This post is a prelude to our upcoming research report and prototype on algorithmic interpretability, which we’ll release in the next few months. Understanding how algorithms use inputs to inform outputs is, in certain instances, a condition for organizations to adopt machine learning systems. This is particularly salient when algorithms are used for sensitive use cases, like criminal sentencing.

Recidivism & ProPublica

Until recently, judges and legal professionals were the sole arbiters deciding whether to release someone on bail. Legislation and other checks that limit discrimination in this process are designed with human biases in mind. As algorithms influence–or even take over–these types of decisions, it is not clear how to audit the process for fairness and discrimination.

Police and legal professionals in states like Virginia, Arizona, and Florida now use scores from the COMPAS algorithm to support decisions about who to set free, typically regarding bail and/or sentencing. The makers claim the COMPAS algorithm can fairly predict whether a person will re-offend. The exact details and methodology of the COMPAS algorithm are not publicly available, but several studies have assessed its performance.

In a recent investigation, ProPublica analyzed the risk scores produced by the COMPAS model for about 7000 individuals in Broward County, Florida and found that

[COMPAS] correctly predicts recidivism 61 percent of the time. But blacks are almost twice as likely as whites to be labeled a higher risk but not actually re-offend. It makes the opposite mistake among whites: they are much more likely than blacks to be labeled lower risk but go on to commit other crimes.

Multicollinearity and Indirect Effects

In its investigation, ProPublica ran several regression models to quantify how COMPAS risk scores depend on race and other factors. Regressions allow one to use the magnitude and sign of the coefficient estimate obtained as a measure of an model’s dependence on particular attributes. However, indirect effects might still not be accounted for if the attributes used in the regression are co-linear.

The variance inflation factor (VIF) allows us to quantify multicollinearity before running a regression. Despite low VIF scores, a variable correlated with other attributes -even midly- used in the regression can still have indirect effects. Consequently, we need a way to properly account for the issue of multicollinearity and indirect effects for black-box models. FairML provides this capability.

Audit of a Proxy Model

While used for public sentencing, Northpointe’s COMPAS risk assessment algorithm is proprietary, so ProPublica was not able to directly examine its model. Instead, ProPublica manually collected data on the COMPAS risk scores for thousands of cases in Broward, County FL. Using this data, they built regression models that measure the relationship between race and the risk scores COMPAS produces.

For our audit, we used a proxy model. We built a logistic regression proxy model using the attributes collected by ProPublica and assumed this model was an oracle, i.e., a reasonable approximation of the COMPAS model. We then audit this proxy model for bias using FairML. As a sanity check, we also built more sophisticated models using the ProPublica data, and our conclusions remain the same.



The basic idea behind FairML (and many other attempts to audit or interpret model behavior) is to measure a model’s dependence on its inputs by changing them. If a small change to an input feature dramatically changes the output, the model is sensitive to the feature.

For example, imagine we have a predictive model of income that takes as input attributes age, occupation, and educational level. One way of determining the model’s dependence on age is to see how much the model’s prediction of an individual’s income changes if the individual’s age is slightly perturbed. If the model places high importance on age, then a slight change would result in a big change to the prediction.

But what if the input attributes are correlated? This is certainly true of age and education: there aren’t many 16-year olds with PhDs! This means that perturbing age alone will not provide an accurate measure of the model’s dependency on age. One has to perturb the other input attributes as well.

Orthogonal Projection as a Perturbation Scheme

The trick FairML uses to counter this multicollinearity is orthogonal projection. FairML orthogonally projects the input to measure the dependence of the predictive model on each attribute.

Why does this help? Orthogonal projection is a type of vector projection. A projection maps one vector onto another. If you project a vector a onto vector b, then you get the component of vector a that lies in the direction of vector b.

Orthogonal Projection Demonstration

An orthogonal projection is a particular type of vector projection that maps a vector onto a direction orthogonal (or perpendicular) to reference vector. For example, if we orthogonally project vector v onto s (in Euclidean space), we get the component of vector v that is at perpendicular to vector s.

Orthogonal Projection Demonstration

Orthogonal projection of vectors is important for FairML because it allows us to completely remove the linear dependence between attributes. If two vectors are orthogonal to one another, then no linear transformation of one vector can produce the other. This intuition underlies the feature dependence measure in FairML.

More concretely, if we have a model, F, trained on two features x and y, the dependence of model F on x is determined by quantifying the change in output on a transformed input. This transformed input perturbs x, and the other feature y is made orthogonal to x.

The change in output between the perturbed input and the original input indicates the dependence of the model on a given attribute, and we can be confident that there are no hidden collinearity effects.

FairML Process Diagram

If you’re familiar with vector projection, you may note that orthogonal projection is a linear transformation. This means it does not account for non-linear dependence among attributes. This is a valid concern. FairML accounts for this by a basis expansion and a greedy search over such expansions (see chapter 4 of thesis, and FairML repo).

FairML - Demo

Finally, here’s how to use the FairML Python package to audit a black-box model using the audit_model function.

from fairml import audit_model

audit_model takes two required arguments:

  • A function that classifies data, i.e., the model of interest.
  • Sample data to be perturbed by the querying function (a pandas DataFrame with no missing data). This sample data should represent data the model will encounter in the real world.

It also takes optional keywords that control the mechanics of the auditing process, including

  • the number of iterations to perform
  • a Boolean flag to enable/disable checking model dependence on interactions
  • distance metric to to quantify dependence.

See the FairML repository for a more detailed explanation of how these parameters affect the auditing process.

audit_model returns a dictionary whose keys are the column names of the input pandas DataFrame and whose values are lists containing model dependence on that particular feature for each iteration performed.

Let’s use scikit-learn to make a model for FairML to audit. We’re using a LogisticRegression model here, but one advantage of FairML is that it can audit any classifier or regressor. FairML only requires that it has a predict function.

from sklearn.linear_model import LogisticRegression

# Read in the propublica data to be used for our analysis.
propublica_data = pd.read_csv("./doc/example_notebooks/propublica_data_for_fairml.csv")

# Create feature and design matrix for model building.
compas_rating = propublica_data.score_factor.values
propublica_data = propublica_data.drop("score_factor", 1)

# Train simple model
clf = LogisticRegression(penalty='l2', C=0.01), compas_rating)

Now let’s audit the model built with FairML.

importances, _ = audit_model( clf.predict, propublica_data)


Feature: Number_of_Priors,   Importance: 0.3608
Feature: Age_Above_FourtyFive,   Importance: -0.006805
Feature: Misdemeanor,    Importance: -0.05266
Feature: Hispanic,   Importance: -0.008425
Feature: Age_Below_TwentyFive,   Importance: 0.1533
Feature: Other,  Importance: -0.004861
Feature: Two_yr_Recidivism,  Importance: 0.2289
Feature: African_American,   Importance: 0.2349
Feature: Asian,  Importance: -0.00032404
Feature: Female,     Importance: 0.045366
Feature: Native_American,    Importance: 0.0004861

As a convenience, FairML includes a basic plotting function to visualize these dependencies:

from fairml import plot_dependencies

    title="FairML feature dependence"

That’s it. FairML allows us to quickly perform an end-to-end audit of a model simply by passing in a predict method and sample data.

Auditing COMPAS

Now that we know how FairML works, let’s use it to audit a hypothetical COMPAS model. Here’s the output of the demo code above.

Fairness in ML

Relative feature dependence ranking obtained from FariML. Red indicates that the factor highly contributes to a high recidivism rating by COMPAS. Note: We omit the intercept term.

Let’s compare the dependence above with that of the logistic regression model obtained in ProPublica’s analysis. Below, we show a bar chart of each attribute and then feature dependence obtained in the two analyses.

Fairness in ML

Relative ranking of Logistic Regression Coefficients obtained by ProPublica. Red indicates that the factor highly contributes to a high recidivism rating by COMPAS. Note: We omit the intercept term.

The crucial point to note is: when we account for multicollinearity using FairML, we actually strengthen ProPublica’s claim of bias against African Americans, i.e., we see that the model’s dependence on that attribute increases. FairML shows that the most significant variable is the number of priors an individual has, but this is followed by the attribute African American. In their analysis, ProPublica found that the importance of the African American attribute was not as strong as attributes such as Age, Native American, and Past Recidivism, which differs from the result of our audit.

As noted earlier, our audit results are for the proxy logistic regression model similar to ProPublica’s proxy model. While this audit is not of the original COMPAS model, we, like ProPublica, take the proxy model as an reasonable approximation. Ultimately, to make concrete conclusions, we’ll need access to the COMPAS model. We also built other more complicated models based on this data and audited them. Our results are consistent with the graph shown above.

Other Work

There are many other examples of research about auditing black-box models. All use input perturbation to measure attribute influence on a black-box model, but use different strategies to perturb the input. Two projects stand out: Adler et al. also propose a data perturbation method (whose open source code is available) and Datta et. al. attack the problem of auditing black-box models.

Beyond auditing black box models, the research on fairness and discrimination in machine learning has become a hot button issue. The Fairness, Accountability and Transparency in Machine Learning community is active and dedicated to this line of research. Recent work has articulated new definitions of fairness, and the inherent tradeoffs that different notions of fairness lead to.

Governments and international organizations have become increasingly concerned about algorithmic decision-making too. In fact, the EU recently adopted a resolution that allows for a ‘right to explanation’ of algorithmic decisions that ‘significantly’ affect an individual. The United States White House has also published several reports on big data and discrimination. Developing and deploying ‘artificial intelligence’ in a safe manner is becoming topical among researchers, industry, and government. Going forward, there seems to be dedicated effort on multiple fronts to tackle this issue.

Read more

Mar 10, 2017 · interview
Feb 27, 2017 · post

Latest posts

Sep 22, 2021 · post

Automatic Summarization from TextRank to Transformers

by Melanie Beck · Automatic summarization is a task in which a machine distills a large amount of data into a subset (the summary) that retains the most relevant and important information from the whole. While traditionally applied to text, automatic summarization can include other formats such as images or audio. In this article we’ll cover the main approaches to automatic text summarization, talk about what makes for a good summary, and introduce Summarize. – a summarization prototype we built that showcases several automatic summarization techniques. more
Sep 21, 2021 · post

Extractive Summarization with Sentence-BERT

by Victor Dibia · In extractive summarization, the task is to identify a subset of text (e.g., sentences) from a document that can then be assembled into a summary. Overall, we can treat extractive summarization as a recommendation problem. That is, given a query, recommend a set of sentences that are relevant. The query here is the document, relevance is a measure of whether a given sentence belongs in the document summary. How we go about obtaining this measure of relevance varies (a common dilemma for any recommendation system). more
Sep 20, 2021 · post

How (and when) to enable early stopping for Gensim's Word2Vec

by Melanie Beck · The Gensim library is a staple of the NLP stack. While it primarily focuses on topic modeling and similarity for documents, it also supports several word embedding algorithms, including what is likely the best-known implementation of Word2Vec. Word embedding models like Word2Vec use unlabeled data to learn vector representations for each token in a corpus. These embeddings can then be used as features in myriad downstream tasks such as classification, clustering, or recommendation systems. more
Jul 7, 2021 · post

Exploring Multi-Objective Hyperparameter Optimization

By Chris and Melanie. The machine learning life cycle is more than data + model = API. We know there is a wealth of subtlety and finesse involved in data cleaning and feature engineering. In the same vein, there is more to model-building than feeding data in and reading off a prediction. ML model building requires thoughtfulness both in terms of which metric to optimize for a given problem, and how best to optimize your model for that metric! more
Jun 9, 2021 ·

Deep Metric Learning for Signature Verification

By Victor and Andrew. TLDR; This post provides an overview of metric learning loss functions (constrastive, triplet, quadruplet, and group loss), and results from applying contrastive and triplet loss to the task of signature verification. A complete list of the posts in this series is outlined below: Pretrained Models as Baselines for Signature Verification -- Part 1: Deep Learning for Automatic Offline Signature Verification: An Introduction Part 2: Pretrained Models as Baselines for Signature Verification Part 3: Deep Metric Learning for Signature Verification In our previous blog post, we discussed how pretrained models can serve as strong baselines for the task of signature verification. more
May 27, 2021 · post

Pretrained Models as a Strong Baseline for Automatic Signature Verification

By Victor and Andrew. Figure 1. Baseline approach for automatic signature verification using pretrained models TLDR; This post describes how pretrained image classification models can be used as strong baselines for the task of signature verification. The full list of posts in the series is outlined below: Pretrained Models as Baselines for Signature Verification -- Part 1: Deep Learning for Automatic Offline Signature Verification: An Introduction Part 2: Pretrained Models as Baselines for Signature Verification Part 3: Deep Metric Learning for Signature Verification As discussed in our introductory blog post, offline signature verification is a biometric verification task that aims to discriminate between genuine and forged samples of handwritten signatures. more

Popular posts

Oct 30, 2019 · newsletter
Exciting Applications of Graph Neural Networks
Nov 14, 2018 · post
Federated learning: distributed machine learning with data locality and privacy
Apr 10, 2018 · post
PyTorch for Recommenders 101
Oct 4, 2017 · post
First Look: Using Three.js for 2D Data Visualization
Aug 22, 2016 · whitepaper
Under the Hood of the Variational Autoencoder (in Prose and Code)
Feb 24, 2016 · post
"Hello world" in Keras (or, Scikit-learn versus Keras)


In-depth guides to specific machine learning capabilities


Machine learning prototypes and interactive notebooks


A usable library for question answering on large datasets.

Explain BERT for Question Answering Models

Tensorflow 2.0 notebook to explain and visualize a HuggingFace BERT for Question Answering model.

NLP for Question Answering

Ongoing posts and code documenting the process of building a question answering model.

Interpretability Revisited: SHAP and LIME

Explore how to use LIME and SHAP for interpretability.


Cloudera Fast Forward is an applied machine learning reseach group.
Cloudera   Blog   Twitter