Whitepaper
FairML: Auditing BlackBox Predictive Models
Mar 9 2017 · by Julius
Machine learning models are used for important decisions like determining who has access to bail. The aim is to increase efficiency and spot patterns in data that humans would otherwise miss. But how do we know if a machine learning model is fair? And what does fairness in machine learning mean?
In this post, we’ll explore these questions using FairML, a new Python library that audits blackbox predictive models and is based on work I did with my advisor while at MIT. We’ll apply it to a hypothetical risk model based on data collected by ProPublica in their investigation of the COMPAS algorithm. We’ll also go over the methodology behind FairML at a conceptual level and describe other work addressing bias in machine learning.
This post is a prelude to our upcoming research report and prototype on algorithmic interpretability, which we’ll release in the next few months. Understanding how algorithms use inputs to inform outputs is, in certain instances, a condition for organizations to adopt machine learning systems. This is particularly salient when algorithms are used for sensitive use cases, like criminal sentencing.
Recidivism & ProPublica
Until recently, judges and legal professionals were the sole arbiters deciding whether to release someone on bail. Legislation and other checks that limit discrimination in this process are designed with human biases in mind. As algorithms influence–or even take over–these types of decisions, it is not clear how to audit the process for fairness and discrimination.
Police and legal professionals in states like Virginia, Arizona, and Florida now use scores from the COMPAS algorithm to support decisions about who to set free, typically regarding bail and/or sentencing. The makers claim the COMPAS algorithm can fairly predict whether a person will reoffend. The exact details and methodology of the COMPAS algorithm are not publicly available, but several studies have assessed its performance.
In a recent investigation, ProPublica analyzed the risk scores produced by the COMPAS model for about 7000 individuals in Broward County, Florida and found that
[COMPAS] correctly predicts recidivism 61 percent of the time. But blacks are almost twice as likely as whites to be labeled a higher risk but not actually reoffend. It makes the opposite mistake among whites: they are much more likely than blacks to be labeled lower risk but go on to commit other crimes.
Multicollinearity and Indirect Effects
In its investigation, ProPublica ran several regression models to quantify how COMPAS risk scores depend on race and other factors. Regressions allow one to use the magnitude and sign of the coefficient estimate obtained as a measure of an model’s dependence on particular attributes. However, indirect effects might still not be accounted for if the attributes used in the regression are colinear.
The variance inflation factor (VIF) allows us to quantify multicollinearity before running a regression. Despite low VIF scores, a variable correlated with other attributes even midly used in the regression can still have indirect effects. Consequently, we need a way to properly account for the issue of multicollinearity and indirect effects for blackbox models. FairML provides this capability.
Audit of a Proxy Model
While used for public sentencing, Northpointe’s COMPAS risk assessment algorithm is proprietary, so ProPublica was not able to directly examine its model. Instead, ProPublica manually collected data on the COMPAS risk scores for thousands of cases in Broward, County FL. Using this data, they built regression models that measure the relationship between race and the risk scores COMPAS produces.
For our audit, we used a proxy model. We built a logistic regression proxy model using the attributes collected by ProPublica and assumed this model was an oracle, i.e., a reasonable approximation of the COMPAS model. We then audit this proxy model for bias using FairML. As a sanity check, we also built more sophisticated models using the ProPublica data, and our conclusions remain the same.
FairML
Perturbation
The basic idea behind FairML (and many other attempts to audit or interpret model behavior) is to measure a model’s dependence on its inputs by changing them. If a small change to an input feature dramatically changes the output, the model is sensitive to the feature.
For example, imagine we have a predictive model of income that takes as input attributes age, occupation, and educational level. One way of determining the model’s dependence on age is to see how much the model’s prediction of an individual’s income changes if the individual’s age is slightly perturbed. If the model places high importance on age, then a slight change would result in a big change to the prediction.
But what if the input attributes are correlated? This is certainly true of age and education: there aren’t many 16year olds with PhDs! This means that perturbing age alone will not provide an accurate measure of the model’s dependency on age. One has to perturb the other input attributes as well.
Orthogonal Projection as a Perturbation Scheme
The trick FairML uses to counter this multicollinearity is orthogonal projection. FairML orthogonally projects the input to measure the dependence of the predictive model on each attribute.
Why does this help? Orthogonal projection is a type of vector projection. A projection maps one vector onto another. If you project a vector a onto vector b, then you get the component of vector a that lies in the direction of vector b.
An orthogonal projection is a particular type of vector projection that maps a vector onto a direction orthogonal (or perpendicular) to reference vector. For example, if we orthogonally project vector v onto s (in Euclidean space), we get the component of vector v that is at perpendicular to vector s.
Orthogonal projection of vectors is important for FairML because it allows us to completely remove the linear dependence between attributes. If two vectors are orthogonal to one another, then no linear transformation of one vector can produce the other. This intuition underlies the feature dependence measure in FairML.
More concretely, if we have a model, F, trained on two features x and y, the dependence of model F on x is determined by quantifying the change in output on a transformed input. This transformed input perturbs x, and the other feature y is made orthogonal to x.
The change in output between the perturbed input and the original input indicates the dependence of the model on a given attribute, and we can be confident that there are no hidden collinearity effects.
If you’re familiar with vector projection, you may note that orthogonal projection is a linear transformation. This means it does not account for nonlinear dependence among attributes. This is a valid concern. FairML accounts for this by a basis expansion and a greedy search over such expansions (see chapter 4 of thesis, and FairML repo).
FairML  Demo
Finally, here’s how to use the FairML Python package to audit a
blackbox model using the audit_model
function.
from fairml import audit_model
audit_model
takes two required arguments:
 A function that classifies data, i.e., the model of interest.
 Sample data to be perturbed by the querying function (a pandas DataFrame with no missing data). This sample data should represent data the model will encounter in the real world.
It also takes optional keywords that control the mechanics of the auditing process, including
 the number of iterations to perform
 a Boolean flag to enable/disable checking model dependence on interactions
 distance metric to to quantify dependence.
See the FairML repository for a more detailed explanation of how these parameters affect the auditing process.
audit_model
returns a dictionary whose keys are the column names of the input
pandas DataFrame and whose values are lists containing model dependence on that
particular feature for each iteration performed.
Let’s use scikitlearn to make a model for FairML to audit. We’re using a
LogisticRegression
model here, but one advantage of FairML is that it can audit any
classifier or regressor. FairML only requires that it has a
predict function.
from sklearn.linear_model import LogisticRegression
# Read in the propublica data to be used for our analysis.
propublica_data = pd.read_csv("./doc/example_notebooks/propublica_data_for_fairml.csv")
# Create feature and design matrix for model building.
compas_rating = propublica_data.score_factor.values
propublica_data = propublica_data.drop("score_factor", 1)
# Train simple model
clf = LogisticRegression(penalty='l2', C=0.01)
clf.fit(propublica_data.values, compas_rating)
Now let’s audit the model built with FairML.
importances, _ = audit_model( clf.predict, propublica_data)
print(importances)
Feature: Number_of_Priors, Importance: 0.3608
Feature: Age_Above_FourtyFive, Importance: 0.006805
Feature: Misdemeanor, Importance: 0.05266
Feature: Hispanic, Importance: 0.008425
Feature: Age_Below_TwentyFive, Importance: 0.1533
Feature: Other, Importance: 0.004861
Feature: Two_yr_Recidivism, Importance: 0.2289
Feature: African_American, Importance: 0.2349
Feature: Asian, Importance: 0.00032404
Feature: Female, Importance: 0.045366
Feature: Native_American, Importance: 0.0004861
As a convenience, FairML includes a basic plotting function to visualize these dependencies:
from fairml import plot_dependencies
plot_dependencies(
total.median(),
reverse_values=False,
title="FairML feature dependence"
)
That’s it. FairML allows us to quickly perform an endtoend audit of a model simply by passing in a predict method and sample data.
Auditing COMPAS
Now that we know how FairML works, let’s use it to audit a hypothetical COMPAS model. Here’s the output of the demo code above.
Relative feature dependence ranking obtained from FariML. Red indicates that the factor highly contributes to a high recidivism rating by COMPAS. Note: We omit the intercept term.
Let’s compare the dependence above with that of the logistic regression model obtained in ProPublica’s analysis. Below, we show a bar chart of each attribute and then feature dependence obtained in the two analyses.
Relative ranking of Logistic Regression Coefficients obtained by ProPublica. Red indicates that the factor highly contributes to a high recidivism rating by COMPAS. Note: We omit the intercept term.
The crucial point to note is: when we account for multicollinearity using FairML, we actually strengthen ProPublica’s claim of bias against African Americans, i.e., we see that the model’s dependence on that attribute increases. FairML shows that the most significant variable is the number of priors an individual has, but this is followed by the attribute African American. In their analysis, ProPublica found that the importance of the African American attribute was not as strong as attributes such as Age, Native American, and Past Recidivism, which differs from the result of our audit.
As noted earlier, our audit results are for the proxy logistic regression model similar to ProPublica’s proxy model. While this audit is not of the original COMPAS model, we, like ProPublica, take the proxy model as an reasonable approximation. Ultimately, to make concrete conclusions, we’ll need access to the COMPAS model. We also built other more complicated models based on this data and audited them. Our results are consistent with the graph shown above.
Other Work
There are many other examples of research about auditing blackbox models. All use input perturbation to measure attribute influence on a blackbox model, but use different strategies to perturb the input. Two projects stand out: Adler et al. also propose a data perturbation method (whose open source code is available) and Datta et. al. attack the problem of auditing blackbox models.
Beyond auditing black box models, the research on fairness and discrimination in machine learning has become a hot button issue. The Fairness, Accountability and Transparency in Machine Learning community is active and dedicated to this line of research. Recent work has articulated new definitions of fairness, and the inherent tradeoffs that different notions of fairness lead to.
Governments and international organizations have become increasingly concerned about algorithmic decisionmaking too. In fact, the EU recently adopted a resolution that allows for a ‘right to explanation’ of algorithmic decisions that ‘significantly’ affect an individual. The United States White House has also published several reports on big data and discrimination. Developing and deploying ‘artificial intelligence’ in a safe manner is becoming topical among researchers, industry, and government. Going forward, there seems to be dedicated effort on multiple fronts to tackle this issue.
More from the Blog

Older ↓
Post
Feb 27 2017
Mobile Behavioural Authentication
by — As mobile devices become central to our personal and professional lives, their security is more and more important. Passcodes in particular can be lost (or forcibly surrendered) to law enforcement. Recent research has focussed on behavioural authentication based on patterns of user interaction. This could provide an unintrusive authentication method that operates during normal use. Figure fr...
…read more
by

Newer ↓
Interview
Mar 10 2017
Trees, Layers, and Speed: Talking Optimization with Patrick Hayes
by — In a modern twist on Claude Shannon’s Theseus, SigOpt explains optimization by teaching a mouse to solve a randomly generated maze The learning of machine learning refers to the process of updating and tuning the parameters of a model. For example, if we take the function f(x) = ax^2 + bx + c, learning would mean to change the values of a, b, and c so that our function does a better job desc...
…read more
by Kathryn

Newest ↓
Post
Jul 17 2019
New research: transfer learning for natural language processing
by — We’ll be discussing this research as part of our virtual event on Wednesday, July 24th at 10:00am PT (1:00pm ET). Register for the event here. Machine learning powers systems that can translate language, guide searches, and interact with humans. All around us we are seeing automated systems that are getting better and better at processing natural language. Machines that can work directly wi...
…read more
by Seth