Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments

Alexandra Chouldechova

article2017Big Data2,543 citations

Proves that risk assessment instruments satisfying predictive parity inevitably generate disparate error rates across demographic groups when baseline recidivism rates differ, exposing an inherent mathematical trade-off in algorithmic fairness.

Listen

Recidivism prediction instruments are increasingly used in criminal justice decisions such as pretrial release, parole, and sentencing, yet they have sparked debate over potential racial bias. The article addresses the tension between two fairness notions: one from psychometrics requiring that risk scores reflect the same recidivism probability across groups, and another focused on equal error rates. Differences in underlying recidivism prevalence between groups make it impossible for an instrument to satisfy both at once, raising the risk of unintended disproportionate penalties.

The article set out to demonstrate how a recidivism prediction instrument that meets the psychometric test-fairness standard can still produce unequal false positive and false negative rates when group recidivism rates differ, and to show how those rate differences translate into disparate impact under penalty policies.

The analysis draws on publicly released Broward County data covering COMPAS scores, two-year recidivism outcomes, and demographics for Black and White defendants. It applies basic probability relationships to link calibration to error rates, then models a simple binary penalty policy to quantify expected differences in outcomes.

The COMPAS instrument approximately satisfies test fairness, with observed recidivism rates aligning across score levels for both groups. Because Black defendants had a 51 percent recidivism rate compared with 39 percent for White defendants, the resulting false positive rate reached 45 percent for Black defendants versus 23 percent for White defendants, while the false negative rate was 28 percent versus 48 percent. Under a policy that assigns stricter penalties to high-risk classifications, non-recidivating Black defendants would face penalties roughly twice as often as non-recidivating White defendants; the pattern reverses but remains unequal among those who do recidivate. These error-rate gaps persist even within subgroups defined by prior record and offense severity.

The findings indicate that an instrument free of predictive bias can nevertheless produce measurable disparate impact when recidivism prevalence differs across groups and high-risk labels trigger harsher treatment. This effect arises directly from the mathematical relationship between prevalence, calibration, and classification errors rather than from flaws in the instrument itself.

Decision makers should therefore choose which fairness criterion to prioritize based on the specific use case, and they should verify error-rate balance at the level of granularity relevant to the policy. Where error-rate parity is desired, instruments may need to be adjusted even if that reduces calibration. Continued reliance on data-driven tools is warranted, provided they are tested for the forms of bias that matter in their intended context.

The results rest on one jurisdiction’s data and a simplified binary penalty model; actual sentencing practices and subgroup dynamics may moderate the magnitude of impact. The relationships shown are deterministic given the observed prevalence differences, supporting high confidence in the core mechanism while leaving room for variation in real-world settings.

Carnegie Mellon University prediction modes proportional fairness disparate impact recidivism prediction predictive bias

arXiv: 1610.07524

Paper: Equality of Opportunity in Supervised Learning, Moritz Hardt et al. (2016). This foundational paper establishes the equalized odds and equal opportunity criteria for supervised learning, formalizing the error-rate parity definitions that contrast with predictive calibration in recidivism models.
Paper: Certifying and Removing Disparate Impact, Michael Feldman et al. (2014). It provides the formal statistical framework linking disparate impact in decision algorithms to balanced error rates and underlying data distributions.
Paper: Fairness through awareness, Cynthia Dwork et al. (2012). This pioneering work introduces formal definitions of algorithmic fairness and analyzes the core conflicts between individual and group-level fairness metrics.
Paper: Big Data's Disparate Impact, Solon Barocas et al. (2016). It lays out the foundational legal and conceptual taxonomy for how predictive algorithms unintentionally create disparate impact despite being facially neutral.
Paper: Predicting good probabilities with supervised learning, Alexandru Niculescu-Mizil et al. (2005). It provides essential background on probability calibration in supervised learning, which underpins the test-fairness criterion analyzed in the source.

Paper: Inherent Trade-Offs in the Fair Determination of Risk Scores, Jon Kleinberg et al. (2017). This paper establishes the rigorous mathematical impossibility theorems proving that calibration within groups and error-rate balance cannot be satisfied simultaneously under unequal base rates.
Paper: A Reductions Approach to Fair Classification, Alekh Agarwal et al. (2018). It develops an algorithmic reductions framework to optimize classifiers under explicit fairness constraints like equalized odds, directly addressing how to handle the trade-offs identified in recidivism prediction.
Paper: Delayed Impact of Fair Machine Learning, Lydia T. Liu et al. (2018). It extends static analyses of error-rate and calibration fairness metrics to dynamic settings, examining their downstream, long-term societal impacts on protected populations over time.
Paper: Counterfactual Fairness, Matt J. Kusner et al. (2017). It introduces a causal and counterfactual framework to address the underlying data-generating mechanisms and limitations inherent in purely observational fairness metrics.
Paper: Fairness and Abstraction in Sociotechnical Systems, Andrew D. Selbst et al. (2019). It critiques the reduction of complex sociotechnical challenges—such as pretrial risk assessment—to narrow mathematical fairness definitions, expanding on the policy trade-offs examined in the source.
Paper: AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias, Rachel Bellamy et al. (2019). It operationalizes multiple competing fairness metrics and mitigation algorithms into a unified open-source software toolkit evaluated on the COMPAS recidivism dataset.
Paper: A Survey on Bias and Fairness in Machine Learning, Ninareh Mehrabi et al. (2019). It offers a comprehensive survey synthesizing the broad taxonomy of algorithmic fairness definitions, trade-offs, and mitigation strategies across the machine learning literature.

Cover for Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments

Abstract

Recidivism prediction instruments provide decision makers with an assessment of the likelihood that a criminal defendant will reoffend at a future point in time. While such instruments are gaining increasing popularity across the country, their use is attracting tremendous controversy. Much of the controversy concerns potential discriminatory bias in the risk assessments that are produced. This paper discusses a fairness criterion originating in the field of educational and psychological testing that has recently been applied to assess the fairness of recidivism prediction instruments. We demonstrate how adherence to the criterion may lead to considerable disparate impact when recidivism prevalence differs across groups.

Topics

prediction modes proportional fairness

Keywords

disparate impact recidivism prediction predictive bias risk assessment COMPAS decile score false-positive rates racial bias fair prediction

1 Introduction
1.1 Data description and setup
2 Assessing fairness
2.1 Implied constraints on the false positive and false negative rates
3 Assessing impact
3.1 Connections to measures of effect size
4 Discussion
References

Knowls

Knowl 1 — Fundamental Trade-off Between Calibration and Error Rate Balance Under Unequal Prevalence
theoretical result
For a binary ground-truth outcome $Y \in \{0, 1\}$ (where $Y=1$ indicates recidivism) and a binary classification score $S_c \in \{\text{LR}, \text{HR}\}$ (indicating low risk and high risk) derived from a risk assessment instrument, let:
- $\text{FPR} = P(S_c = \text{HR} \mid Y = 0)$ be the false positive rate,
- $\text{FNR} = P(S_c = \text{LR} \mid Y = 1)$ be the false negative rate,
- $\text{PPV} = P(Y = 1 \mid S_c = \text{HR})$ be the positive predictive value,
- $p = P(Y = 1)$ be the base rate prevalence of the outcome, where $p \in (0, 1)$ and $\text{PPV} \in (0, 1)$ .
The classification error rates and predictive value are linked by the exact relation:

$\text{FPR} = \frac{p}{1 - p} \frac{1 - \text{PPV}}{\text{PPV}} (1 - \text{FNR})$

Consequently, if two demographic groups $b$ and $w$ exhibit different outcome prevalences ( $p_b \neq p_w$ ) and the classifier satisfies predictive parity or test fairness (meaning $\text{PPV}_b = \text{PPV}_w$ ), the classifier cannot simultaneously achieve equal false positive rates ( $\text{FPR}_b = \text{FPR}_w$ ) and equal false negative rates ( $\text{FNR}_b = \text{FNR}_w$ ). Specifically, the group with the higher prevalence will necessarily experience a higher false positive rate, a lower false negative rate, or both.
Knowl 2 — Test Fairness and Calibration of Risk Prediction Instruments
definition
Let $S = S(x) \in \mathbb{R}$ denote a continuous or ordinal risk score computed from covariates $X = x$ , where higher values correspond to greater assessed risk. Let $R \in \{b, w\}$ denote a sensitive group attribute (such as race), and let $Y \in \{0, 1\}$ denote the binary outcome (such as recidivism).

A score $S$ satisfies test fairness (or is well-calibrated) across groups if the probability of the outcome given the score is independent of group membership. That is, for all possible score values $s$ :

$P(Y = 1 \mid S = s, R = b) = P(Y = 1 \mid S = s, R = w)$

When $S$ is thresholded at a cutoff $s_{\text{HR}}$ to produce a coarsened binary prediction $S_c \in \{\text{LR}, \text{HR}\}$ ( $S_c = \text{HR}$ if $S > s_{\text{HR}}$ and $S_c = \text{LR}$ if $S \le s_{\text{HR}}$ ), test fairness of $S$ implies that the positive predictive value (PPV) of the coarsened classifier is equal across groups:

$\text{PPV}(S_c \mid R = r) \equiv P(Y = 1 \mid S_c = \text{HR}, R = r) \quad \text{is independent of } r.$
Knowl 3 — Disparate Penalty Impact Under Binary Risk-Based Policies
theoretical result
Consider a decision-making policy $\mathcal{T}_{\text{MinMax}}$ where individuals assessed as low risk ( $S_c = \text{LR}$ ) receive a lower penalty $t_L$ and those assessed as high risk ( $S_c = \text{HR}$ ) receive a stricter penalty $t_H$ , with $t_H > t_L$ :

$\mathcal{T}_{\text{MinMax}} = \begin{cases} t_L & \text{if } S_c = \text{LR} \\ t_H & \text{if } S_c = \text{HR} \end{cases}$

Let $T_{r, y}$ denote the penalty assigned to an individual from group $R = r \in \{b, w\}$ with true outcome $Y = y \in \{0, 1\}$ . The expected difference in penalty $\Delta(y_1, y_2) \equiv \mathbb{E}_{\text{MinMax}}[T_{b, y_1} - T_{w, y_2}]$ between group $b$ (higher prevalence) and group $w$ (lower prevalence) is:

$\Delta(y_1, y_2) = (t_H - t_L) \left[ P(S_c = \text{HR} \mid R = b, Y = y_1) - P(S_c = \text{HR} \mid R = w, Y = y_2) \right]$

This yields two immediate cases for disparate impact:
1. Among non-recidivators ( $y_1 = y_2 = 0$ ):
$\Delta(0, 0) = (t_H - t_L)(\text{FPR}_b - \text{FPR}_w)$

where $\text{FPR}_r = P(S_c = \text{HR} \mid R = r, Y = 0)$ . In the special case where $t_L = 0$ and $t_H = 1$ (e.g., probability of incarceration), a non-recidivating individual in group $b$ is $\text{FPR}_b / \text{FPR}_w$ times more likely to be penalized than in group $w$ .
1. Among recidivators ( $y_1 = y_2 = 1$ ):
$\Delta(1, 1) = (t_H - t_L)(\text{FNR}_w - \text{FNR}_b)$

where $\text{FNR}_r = P(S_c = \text{LR} \mid R = r, Y = 1)$ .
Knowl 4 — Total Variation Bound on Disparate Impact
theoretical result
Under the binary $\mathcal{T}_{\text{MinMax}}$ penalty policy assigning penalty $t_L$ for low risk and $t_H$ for high risk ( $t_H > t_L$ ), let $f_{r, y}(s)$ denote the probability distribution of the continuous or ordinal risk score $S$ for demographic group $R = r$ and true outcome $Y = y$ .

The expected difference in penalty between group $b$ and group $w$ among individuals with outcome $y \in \{0, 1\}$ satisfies the upper bound:

$\Delta(y, y) \le (t_H - t_L) \, d_{\text{TV}}(f_{b, y}, f_{w, y})$

where $d_{\text{TV}}(f_{b, y}, f_{w, y})$ is the total variation distance between the score distributions:

$d_{\text{TV}}(f_{b, y}, f_{w, y}) = \frac{1}{2} \sum_{s} |f_{b, y}(s) - f_{w, y}(s)|$

(or the corresponding integral in the continuous case), representing a generalization of the percentage of non-overlap for non-Gaussian distributions.
Knowl 5 — Recidivism Prevalence and Error Rate Disparities in COMPAS Broward County Data
empirical result
In an empirical evaluation using the ProPublica Broward County dataset of defendants evaluated by the COMPAS recidivism risk instrument:
- The observed two-year recidivism prevalence differs between racial groups: $51\%$ for Black defendants ( $p_b = 0.51$ ) compared to $39\%$ for White defendants ( $p_w = 0.39$ ).
- The COMPAS decile risk score adheres closely to test fairness (calibration), with observed recidivism rates $P(Y = 1 \mid S = s, R)$ being approximately equal between Black and White defendants across all decile levels $s \in \{1, \dots, 10\}$ .
- When thresholding at decile score $>4$ $> 4$ to define a high-risk classifier, substantial error rate disparities emerge as a mathematical consequence of calibration under unequal prevalence:
  - False Positive Rate ( $\text{FPR}$ ): $45\%$ for Black defendants versus $23\%$ for White defendants.
  - False Negative Rate ( $\text{FNR}$ ): $28\%$ for Black defendants versus $48\%$ for White defendants.
- The COMPAS decile distributions across races yield Cohen's $d = 0.60$ and a total variation non-overlap distance of $d_{\text{TV}}(f_b, f_w) = 24.5\%$ .
Knowl 6 — Persistence of False Positive Rate Disparities Across Granular Prior Record Subgroups
empirical result
When analyzing criminal defendants in the Broward County dataset charged exclusively with misdemeanor offenses (the lowest offense severity category) thresholded at COMPAS decile score $> 4$ :
- The false positive rate (FPR) remains substantially higher for Black defendants than for White defendants across every prior record count subgroup ( $0$ , $1\text{--}3$ , $4\text{--}6$ , $7\text{--}10$ , and $>10$ priors).
- For defendants with zero prior offenses, the FPR is approximately $20\%$ for Black defendants versus under $10\%$ for White defendants.
- This demonstrates that between-group error rate imbalances arising from calibrated scores are not simply an aggregate confounding artifact of differences in criminal histories or charge severity, but persist within granular criminal history strata.

Coverage note — Deliberately omitted narrative background on the public COMPAS controversy and general qualitative discussion on clinical vs. actuarial prediction in sentencing, retaining all formal definitions, mathematical proofs/propositions, bounds, and empirical findings.

References

1.Thomas Blomberg, William Bales, Karen Mann, Ryan Meldrum, and Joe Nedelec. Validation of the compas risk assessment classification instrument. 2010.
2.Ben Casselman Anna Maria Barry-Jester and Dana Goldstein. Should prison sentences be based on crimes that haven’t been committed yet?
3.Northpointe. Compas risk & need assessment system: Selected questions posed by inquiring agencies.
4.Administrative Office of the United States Courts. An overview of the federal post conviction risk assessment, September 2011.
5.Jay P Singh. Predictive validity performance indicators in violence risk assessment: A methodological primer. Behavioral Sciences & the Law, 31(1):8–22, 2013.
6.Jennifer L Skeem and Christopher T Lowenkamp. Risk, race, & recidivism: Predictive bias and disparate impact. Available at SSRN, 2015.
7.Jennifer L Skeem, John Monahan, and Christopher T Lowenkamp. Gender, risk assessment, and sanctioning: The cost of treating women like men. Available at SSRN 2718460, 2016.
8.Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias: There’s software used across the country to predict future criminals. and it’s biased against blacks. 2016. URL https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing.
9.Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. How we analyzed the compas recidivism algorithm. 2016. URL https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm.
10.Anthony W Flores, Kristin Bechtel, and Christopher T Lowenkamp. False positives, false negatives, and false analyses: A rejoinder to “machine bias: There’s software used across the country to predict future criminals. and it’s biased against blacks.”. Unpublished manuscript, 2016.
11.Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807, 2016.
12.Jacob Cohen. Statistical Power Analysis for the Behavioral Sciences (2nd Edition). Lawrence Erlbaum Associates, 1988.
13.Paul E Meehl. Clinical versus statistical prediction: A theoretical analysis and a review of the evidence. University of Minnesota Press, 1954.
14.William M Grove, David H Zald, Boyd S Lebow, Beth E Snitz, and Chad Nelson. Clinical versus mechanical prediction: a meta-analysis. Psychological assessment, 12(1):19, 2000.
15.Shamena Anwar and Hanming Fang. Testing for racial prejudice in the parole board release process: Theory and evidence. Technical report, National Bureau of Economic Research, 2012.
16.Laura T Sweeney and Craig Haney. The influence of race on sentencing: A meta-analytic review of experimental studies. Behavioral Sciences & the Law, 10(2):179–195, 1992.

Citation

MLA

Chouldechova, A. “Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments”. Big Data, vol. 5, no. 2, 2017, pp. 153–63, https://doi.org/10.1089/big.2016.0047.

APA

Chouldechova, A. (2017). Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data, 5(2), 153–163. https://doi.org/10.1089/big.2016.0047

Chicago

Chouldechova, A. 2017. “Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments”. Big Data 5 (2): 153–63. https://doi.org/10.1089/big.2016.0047.

Harvard

Chouldechova, A. (2017) “Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments”, Big Data, 5(2), pp. 153–163. Available at: https://doi.org/10.1089/big.2016.0047.

Vancouver

1. Chouldechova A (2017) Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments. Big Data 5:153–163

BibTeX

@article{Chouldechova_2017, title={Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments}, volume={5}, ISSN={2167-647X}, url={http://dx.doi.org/10.1089/big.2016.0047}, DOI={10.1089/big.2016.0047}, number={2}, journal={Big Data}, publisher={SAGE Publications}, author={Chouldechova, Alexandra}, year={2017}, month=June, pages={153–163} }

Metadata:Crossref

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — Fundamental Trade-off Between Calibration and Error Rate Balance Under Unequal Prevalence

Knowl 2 — Test Fairness and Calibration of Risk Prediction Instruments

Knowl 3 — Disparate Penalty Impact Under Binary Risk-Based Policies

Knowl 4 — Total Variation Bound on Disparate Impact

Knowl 5 — Recidivism Prevalence and Error Rate Disparities in COMPAS Broward County Data

Knowl 6 — Persistence of False Positive Rate Disparities Across Granular Prior Record Subgroups

References

Citation

BibTeX

Access the Paper