Back

A Survey on Bias and Fairness in Machine Learning

Ninareh MehrabiFred MorstatterNripsuta SaxenaKristina LermanAram Galstyan

article2019ACM Computing Surveys6,348 citations

Categorizes the sources of bias throughout the machine learning pipeline and establishes a structured taxonomy of mathematical fairness definitions and mitigation strategies across real-world artificial intelligence applications.

ai applicationsdeep learningengineeringalgorithmic fairnessalgorithmic biasfairness-aware classification
arXiv: 1908.09635
  • Paper: Fairness through awareness, Cynthia Dwork et al. (2012). It introduces foundational definitions of individual and group fairness that form the bedrock of the survey's fairness taxonomy.
  • Paper: Equality of Opportunity in Supervised Learning, Moritz Hardt et al. (2016). It formalizes key fairness criteria like equalized odds and equal opportunity in supervised learning, which are centrally categorized in the survey.
  • Paper: Inherent Trade-Offs in the Fair Determination of Risk Scores, Jon Kleinberg et al. (2017). It proves the fundamental mathematical trade-offs and impossibilities among core fairness metrics that the survey discusses.
  • Paper: Certifying and Removing Disparate Impact, Michael Feldman et al. (2014). It provides the mathematical framework for certifying and removing disparate impact via data pre-processing, a primary mitigation strategy reviewed in the survey.
  • Paper: Learning Fair Representations, Richard Zemel et al. (2013). It establishes representation learning techniques to mitigate bias before downstream classification, providing a core pre-processing reference for the survey.
  • Paper: Counterfactual Fairness, Matt J. Kusner et al. (2017). It develops counterfactual fairness using causal modeling, offering a vital theoretical perspective reviewed within the survey's fairness taxonomy.
  • Paper: Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification, Joy Buolamwini et al. (2018). It provides seminal empirical evidence of intersectional bias in commercial vision systems, illustrating the real-world harms analyzed in the survey.
  • Paper: Semantics derived automatically from language corpora contain human-like biases, Aylin Caliskan et al. (2016). It demonstrates how human-like biases are automatically embedded in NLP models and embeddings, establishing foundational evidence for the survey's NLP domain analysis.
  • Paper: A Reductions Approach to Fair Classification, Alekh Agarwal et al. (2018). It provides the reductions-based in-processing framework for constrained fair classification that is categorized among standard algorithmic mitigation approaches.
  • Paper: Big Data's Disparate Impact, Solon Barocas et al. (2016). It outlines the technical mechanisms through which data mining can introduce legal disparate impact, motivating the survey's taxonomy of bias sources.
Cover for A Survey on Bias and Fairness in Machine Learning

Abstract

With the widespread use of AI systems and applications in our everyday lives, it is important to take fairness issues into consideration while designing and engineering these types of systems. Such systems can be used in many sensitive environments to make important and life-changing decisions; thus, it is crucial to ensure that the decisions do not reflect discriminatory behavior toward certain groups or populations. We have recently seen work in machine learning, natural language processing, and deep learning that addresses such challenges in different subdomains. With the commercialization of these systems, researchers are becoming aware of the biases that these applications can contain and have attempted to address them. In this survey we investigated different real-world applications that have shown biases in various ways, and we listed different sources of biases that can affect AI applications. We then created a taxonomy for fairness definitions that machine learning researchers have defined in order to avoid the existing bias in AI systems. In addition to that, we examined different domains and subdomains in AI showing what researchers have observed with regard to unfair outcomes in the state-of-the-art methods and how they have tried to address them. There are still many future directions and solutions that can be taken to mitigate the problem of bias in AI systems. We are hoping that this survey will motivate researchers to tackle these issues in the near future by observing existing work in their respective fields.

Topics

ai applicationsdeep learningengineeringtaxonomynatural language understanding

Keywords

algorithmic fairnessalgorithmic biasfairness-aware classificationcounterfactual fairnessindividual fairnessgroup fairnessbias mitigationfair representationsstatistical discriminationfair regression

Table of Contents

  • 1 Introduction
  • 2 Real-World Examples of Algorithmic Unfairness
  • 2.1 Systems that Demonstrate Discrimination
  • 2.2 Assessment Tools
  • 3 Bias in Data, Algorithms, and User Experiences
  • 3.1 Types of Bias
  • 3.1.1 Data to Algorithm
  • 3.1.2 Algorithm to User
  • 3.1.3 User to Data
  • 3.2 Data Bias Examples
  • 3.2.1 Examples of Bias in Machine Learning Data
  • 3.2.2 Examples of Data Bias in Medical Applications
  • 3.3 Discrimination
  • 3.3.1 Explainable Discrimination
  • 3.3.2 Unexplainable Discrimination
  • 3.3.3 Sources of Discrimination
  • 4 Algorithmic Fairness
  • 4.1 Definitions of Fairness
  • 5 Methods for Fair Machine Learning
  • 5.1 Unbiasing Data
  • 5.2 Fair Machine Learning
  • 5.2.1 Fair Classification
  • 5.2.2 Fair Regression
  • 5.2.3 Structured Prediction
  • 5.2.4 Fair PCA
  • 5.2.5 Community Detection/Graph Embedding/Clustering
  • 5.2.6 Causal Approach to Fairness
  • 5.3 Fair Representation Learning
  • 5.3.1 Variational Auto Encoders
  • 5.3.2 Adversarial Learning
  • 5.4 Fair NLP
  • 5.4.1 Word Embedding
  • 5.4.2 Coreference Resolution
  • 5.4.3 Language Model
  • 5.4.4 Sentence Encoder
  • 5.4.5 Machine Translation
  • 5.4.6 Named Entity Recognition
  • 5.5 Comparison of Different Mitigation Algorithms
  • 6 Challenges and Opportunities for Fairness Research
  • 6.1 Challenges
  • 6.2 Opportunities
  • 7 Conclusion
  • 8 Acknowledgments
  • 9 Appendix
  • 9.1 Datasets for Fairness Research
  • 9.1.1 UCI Adult Dataset
  • 9.1.2 German Credit Dataset
  • 9.1.3 WinoBias
  • 9.1.4 Communities and Crime Dataset
  • 9.1.5 COMPAS Dataset
  • 9.1.6 Recidivism in Juvenile Justice Dataset
  • 9.1.7 Pilot Parliaments Benchmark Dataset
  • 9.1.8 Diversity in Faces Dataset
  • References

Knowls

  1. Knowl 1 — Group Fairness Definitions in Machine Learning

    definition

    Group fairness requires that algorithmic decisions satisfy statistical parity or equal error rates across demographic groups defined by a protected attribute A∈{0,1}A \in \{0, 1\} and true binary outcome label Y∈{0,1}Y \in \{0, 1\}, for a predictor outcome Y^∈{0,1}\hat{Y} \in \{0, 1\}:

    • Demographic Parity (Statistical Parity): The likelihood of receiving a positive outcome is independent of protected attribute status: P(Y^=1∣A=0)=P(Y^=1∣A=1)P(\hat{Y}=1 \mid A=0) = P(\hat{Y}=1 \mid A=1)

    • Equalized Odds: The predictor Y^\hat{Y} is independent of the protected attribute AA conditional on the actual outcome YY, requiring true positive rates and false positive rates to be equal across groups: P(Y^=1∣A=0,Y=y)=P(Y^=1∣A=1,Y=y),∀y∈{0,1}P(\hat{Y}=1 \mid A=0, Y=y) = P(\hat{Y}=1 \mid A=1, Y=y), \quad \forall y \in \{0, 1\}

    • Equal Opportunity: A relaxation of equalized odds requiring equal true positive rates across protected and unprotected groups for the positive class: P(Y^=1∣A=0,Y=1)=P(Y^=1∣A=1,Y=1)P(\hat{Y}=1 \mid A=0, Y=1) = P(\hat{Y}=1 \mid A=1, Y=1)

    • Treatment Equality: The ratio of false negatives to false positives is identical across both protected and unprotected groups: P(Y^=0∣A=0,Y=1)P(Y^=1∣A=0,Y=0)=P(Y^=0∣A=1,Y=1)P(Y^=1∣A=1,Y=0)\frac{P(\hat{Y}=0 \mid A=0, Y=1)}{P(\hat{Y}=1 \mid A=0, Y=0)} = \frac{P(\hat{Y}=0 \mid A=1, Y=1)}{P(\hat{Y}=1 \mid A=1, Y=0)}

    • Test Fairness (Calibration within Groups): For a continuous prediction score S=S(x)S = S(x), the probability of correctly belonging to the positive outcome given score ss is identical across demographic groups RR: P(Y=1∣S=s,R=b)=P(Y=1∣S=s,R=w),∀sP(Y=1 \mid S=s, R=b) = P(Y=1 \mid S=s, R=w), \quad \forall s

    • Conditional Statistical Parity: For a set of legitimate factors LL, the outcome is conditionally independent of protected attribute AA given LL: P(Y^=1∣L=l,A=0)=P(Y^=1∣L=l,A=1)P(\hat{Y}=1 \mid L=l, A=0) = P(\hat{Y}=1 \mid L=l, A=1)

  2. Knowl 2 — Individual, Counterfactual, and Relational Fairness Definitions

    definition

    Individual, counterfactual, and relational fairness criteria evaluate algorithmic fairness at the level of individual instances, causal mechanisms, or network relationships:

    • Fairness Through Awareness (Individual Fairness): An algorithm satisfies individual fairness if it produces similar predictions for similar individuals, formalised via a task-specific metric D(x,x′)D(x, x') such that if two individuals x,x′x, x' are close under DD, their predictive distributions are correspondingly close.

    • Fairness Through Unawareness: A decision system satisfies fairness through unawareness if protected attributes AA are explicitly omitted from the features supplied to the model during decision-making.

    • Counterfactual Fairness: For input features XX, protected attribute AA, background/latent variables UU, and predictor Y^\hat{Y}, a predictor is counterfactually fair if the predictive distribution for an individual is invariant to counterfactually setting their protected attribute from aa to any alternative value a′a': P(Y^A←a(U)=y∣X=x,A=a)=P(Y^A←a′(U)=y∣X=x,A=a)P(\hat{Y}_{A \leftarrow a}(U) = y \mid X=x, A=a) = P(\hat{Y}_{A \leftarrow a'}(U) = y \mid X=x, A=a) for all possible outcomes yy and attainable values a′a'.

    • Fairness in Relational Domains: A graph-oriented notion of fairness that evaluates constraints over individuals by incorporating relational structures, social network links, and organizational connections rather than treating instances as independent and identically distributed.

    • Subgroup Fairness: Enforces group fairness constraints (such as equal false positive rates) across a rich, combinatorially defined collection of structured subgroups formed by intersections of multiple sensitive and non-sensitive attributes, bridging individual and group fairness.

  3. Knowl 3 — Feedback Loop Taxonomy of Machine Learning Bias

    model/method

    Biases in machine learning propagate across a continuous feedback loop connecting data generation, algorithmic processing, and user interaction. The taxonomy structures these biases into three distinct directional segments:

    1. Data to Algorithm: Biases originating in data collection and measurement that distort model learning:

      • Measurement Bias: Arises from mismeasuring attributes or using skewed proxy variables (e.g., arrest counts as a proxy for criminal risk).
      • Omitted Variable Bias: Occurs when one or more key explanatory variables are absent from the model.
      • Representation Bias: Non-representative sampling from a target population, leading to missing subgroups.
      • Aggregation Bias: Incorrectly concluding individual relationships from aggregate population trends (including Simpson's Paradox and Modifiable Areal Unit Problems).
      • Sampling Bias: Non-random subgroup sampling causing training data distributions to diverge from deployment distributions.
      • Longitudinal Data Fallacy: Misleading conclusions drawn from cross-sectional snapshots taken at single time points rather than tracking cohorts over time.
      • Linking Bias: User connection graphs misrepresenting true user behaviors due to network sampling artifacts.

    2. Algorithm to User: Biases induced by algorithmic models that modulate subsequent user actions:

      • Algorithmic Bias: Disparities introduced purely by algorithmic design choices (loss functions, regularizers, or biased estimators) even when input data is balanced.
      • User Interaction Bias: User behavioral changes shaped by interface designs (including Presentation Bias and Ranking Bias where top positions collect disproportionate engagement).
      • Popularity Bias: Feedback amplification of already popular items driven by recommendation mechanisms rather than underlying quality.
      • Emergent Bias: Unanticipated bias arising post-deployment due to evolving cultural norms, user demographics, or societal context.
      • Evaluation Bias: Skewed performance assessment arising from unbalanced or disproportionate benchmark datasets.

    3. User to Data: Biases introduced during user-generated content creation:

      • Historical Bias: Societal inequities reflected in data generation despite technically sound sampling.
      • Population Bias: Demographic mismatch between platform user bases and target populations.
      • Self-Selection Bias: Subjects selectively choosing whether to participate in surveys or platforms.
      • Social Bias: Social influence and peer ratings distorting individual judgment.
      • Behavioral, Temporal, and Content Production Biases: Systematic demographic variations in language, emoji usage, and timing across platforms.

  4. Knowl 4 — Three-Stage Categorization of Bias Mitigation Techniques

    model/method

    Algorithmic bias mitigation strategies in machine learning are classified into three operational categories based on where they intervene in the machine learning lifecycle:

    1. Pre-processing: Techniques applied directly to the training data before model optimization. When data modification is permissible, pre-processing transforms sample distributions or feature representations to remove discriminatory signals. Methods include sample reweighting, preferential sampling, message perturbation, dataset nutrition labels/datasheets, and removing disparate impact from feature representations.

    2. In-processing: Techniques applied during model training. When the learning objective or optimization procedure can be modified, in-processing incorporates fairness constraints directly into the loss function. Methods include adding regularizers penalizing demographic disparity, adversarial objectives to minimize protected attribute mutual information in latent embeddings, Wasserstein-based constraint optimization, and multitask learning frameworks.

    3. Post-processing: Techniques applied after model training. When the model must be treated as a fixed black box without access to modify training data or the learning objective, post-processing adjusts or remaps predictions on a holdout validation set based on optimal decision thresholds to satisfy targeted fairness constraints.

  5. Knowl 5 — Fairness Penalty Objectives for Continuous Regression

    equation

    For continuous regression models parameterised by weights ww, where a dataset is partitioned into demographic subgroups S1S_1 of size n1n_1 and S2S_2 of size n2n_2, fairness can be enforced by adding penalty functions to the regression objective, weighted by a distance metric d(yi,yj)=f(∣yi−yj∣)d(y_i, y_j) = f(|y_i - y_j|) between target labels:

    • Individual Fairness Penalty: Penalizes the model ww for producing differing predictions on cross-pair samples (xi,yi)∈S1(x_i, y_i) \in S_1 and (xj,yj)∈S2(x_j, y_j) \in S_2 with similar target outcomes: f1(w,S)=1n1n2∑(xi,yi)∈S1∑(xj,yj)∈S2d(yi,yj)(w⋅xi−w⋅xj)2f_1(w, S) = \frac{1}{n_1 n_2} \sum_{(x_i, y_i) \in S_1} \sum_{(x_j, y_j) \in S_2} d(y_i, y_j) (w \cdot x_i - w \cdot x_j)^2

    • Group Fairness Penalty: Penalizes average cross-group predictive disparities: f2(w,S)=(1n1n2∑(xi,yi)∈S1∑(xj,yj)∈S2d(yi,yj)(w⋅xi−w⋅xj))2f_2(w, S) = \left( \frac{1}{n_1 n_2} \sum_{(x_i, y_i) \in S_1} \sum_{(x_j, y_j) \in S_2} d(y_i, y_j) (w \cdot x_i - w \cdot x_j) \right)^2

    • Hybrid Fairness Penalty: Enforces average parity separately across positive-labeled cross-pairs (yi=yj=1y_i = y_j = 1, with group sizes n1,1n_{1,1} and n2,1n_{2,1}) and negative-labeled cross-pairs (yi=yj=−1y_i = y_j = -1, with group sizes n1,−1n_{1,-1} and n2,−1n_{2,-1}): f3(w,S)=(∑(xi,yi)∈S1(xj,yj)∈S2yi=yj=1d(yi,yj)(w⋅xi−w⋅xj)n1,1n2,1)2+(∑(xi,yi)∈S1(xj,yj)∈S2yi=yj=−1d(yi,yj)(w⋅xi−w⋅xj)n1,−1n2,−1)2f_3(w, S) = \left( \sum_{\substack{(x_i, y_i) \in S_1 \\ (x_j, y_j) \in S_2 \\ y_i = y_j = 1}} \frac{d(y_i, y_j)(w \cdot x_i - w \cdot x_j)}{n_{1,1} n_{2,1}} \right)^2 + \left( \sum_{\substack{(x_i, y_i) \in S_1 \\ (x_j, y_j) \in S_2 \\ y_i = y_j = -1}} \frac{d(y_i, y_j)(w \cdot x_i - w \cdot x_j)}{n_{1,-1} n_{2,-1}} \right)^2

  6. Knowl 6 — Minimax Optimization Framework for Fair Principal Component Analysis

    model/method

    Standard Principal Component Analysis (PCA) can produce unequal reconstruction errors across demographic groups even when groups are sampled with equal weights. Fair PCA formulates dimensionality reduction to target dimension dd as a minimax optimization problem that equalizes reconstruction fidelity across demographic subgroups AA and BB.

    Given mm data points in Rn\mathbb{R}^n partitioned into subgroups AA and BB, let U∈Rm×nU \in \mathbb{R}^{m \times n} denote the low-rank projection matrix, and let UAU_A and UBU_B denote the row submatrices corresponding to members of AA and BB. The Fair PCA objective minimizes the maximum normalized reconstruction loss between the two groups: min⁡U∈Rm×nrank(U)≤dmax⁡(1∣A∣loss(A,UA),  1∣B∣loss(B,UB))\min_{\substack{U \in \mathbb{R}^{m \times n} \\ \text{rank}(U) \le d}} \max \left( \frac{1}{|A|} \text{loss}(A, U_A), \; \frac{1}{|B|} \text{loss}(B, U_B) \right)

    This optimization problem is solved via a two-step procedure:

    1. Relax the Fair PCA objective to a semidefinite program (SDP) and solve it to obtain an initial relaxed matrix.
    2. Solve a linear program (LP) to reduce the rank of the SDP solution to at most dd.

  7. Knowl 7 — Causal Graph Criteria for Unresolved and Proxy Discrimination

    definition

    In causal Directed Acyclic Graphs (DAGs), where nodes represent variables and directed edges represent causal mechanisms, discrimination is identified by specific causal paths originating from a protected attribute AA:

    • Unresolved Discrimination: A variable VV in a causal DAG exhibits unresolved discrimination if there exists a directed path from protected attribute AA to VV that is not blocked by a resolving variable, and VV itself is non-resolving (where a resolving variable is one whose conditional dependence on AA is accepted as legitimate in the specific decision context).

    • Proxy Discrimination: A variable VV in a causal DAG exhibits potential proxy discrimination if there exists a directed path from AA to VV that is blocked by a proxy variable, and VV itself is not designated as a proxy.

    • Causal Risk Difference (RDcRD^c): Causal discrimination discovery quantifies disparity between groups via: RDc=p1−p2cRD^c = p_1 - p_2^c where p1p_1 is the observed probability of positive decision in group 1, and p2cp_2^c is the adjusted probability under propensity score weighting: p2c=∑s∈S,dec(s)=⊖w(s)∑s∈Sw(s)p_2^c = \frac{\sum_{s \in S, dec(s) = \ominus} w(s)}{\sum_{s \in S} w(s)} A non-zero value of RDcRD^c indicates causal discrimination or unaccounted confounding covariates (omitted variable bias).

  8. Knowl 8 — Multi-Metric Evaluation Framework for Gender Bias in Named Entity Recognition

    equation

    In Named Entity Recognition (NER) systems, gender bias occurs when female names are disproportionately unrecognised or misclassified compared to male names across templated sentences. Given female name set NfN_f, predicted tag type ntypen_{type}, and census frequency weighting function freqf(n)freq_f(n), six metrics quantify these disparities:

    • Error Type-1 Unweighted (Proportion of names tagged as non-PERSON or untagged): Error1unweighted=∑n∈NfI(ntype≠PERSON)∣Nf∣\text{Error}_1^{\text{unweighted}} = \frac{\sum_{n \in N_f} \mathbb{I}(n_{type} \neq \text{PERSON})}{|N_f|}

    • Error Type-1 Weighted (Frequency-weighted proportion of names tagged as non-PERSON): Error1weighted=∑n∈Nffreqf(n)⋅I(ntype≠PERSON)∑n∈Nffreqf(n)\text{Error}_1^{\text{weighted}} = \frac{\sum_{n \in N_f} freq_f(n) \cdot \mathbb{I}(n_{type} \neq \text{PERSON})}{\sum_{n \in N_f} freq_f(n)}

    • Error Type-2 Unweighted (Proportion of names misclassified as non-person entities such as LOCATION, excluding untagged names ∅\emptyset): Error2unweighted=∑n∈NfI(ntype∉{∅,PERSON})∣Nf∣\text{Error}_2^{\text{unweighted}} = \frac{\sum_{n \in N_f} \mathbb{I}(n_{type} \notin \{\emptyset, \text{PERSON}\})}{|N_f|}

    • Error Type-2 Weighted: Error2weighted=∑n∈Nffreqf(n)⋅I(ntype∉{∅,PERSON})∑n∈Nffreqf(n)\text{Error}_2^{\text{weighted}} = \frac{\sum_{n \in N_f} freq_f(n) \cdot \mathbb{I}(n_{type} \notin \{\emptyset, \text{PERSON}\})}{\sum_{n \in N_f} freq_f(n)}

    • Error Type-3 Unweighted (Proportion of names left completely untagged): Error3unweighted=∑n∈NfI(ntype=∅)∣Nf∣\text{Error}_3^{\text{unweighted}} = \frac{\sum_{n \in N_f} \mathbb{I}(n_{type} = \emptyset)}{|N_f|}

    • Error Type-3 Weighted: Error3weighted=∑n∈Nffreqf(n)⋅I(ntype=∅)∑n∈Nffreqf(n)\text{Error}_3^{\text{weighted}} = \frac{\sum_{n \in N_f} freq_f(n) \cdot \mathbb{I}(n_{type} = \emptyset)}{\sum_{n \in N_f} freq_f(n)}

    Corresponding male metrics are computed analogously over male names NmN_m using male census frequencies freqm(n)freq_m(n).

  9. Knowl 9 — Benchmark Datasets for Algorithmic Fairness Evaluation

    data/table

    Empirical evaluation of bias detection and fairness mitigation algorithms relies on a collection of standard benchmark datasets spanning social, financial, legal, natural language processing, and computer vision domains:

    Dataset Name Size Area / Concentration
    UCI Adult (Census Income) 48,842 income records Social / Gender and racial income inequality
    German Credit 1,000 credit records Financial / Gender inequality in credit approval
    Pilot Parliaments Benchmark (PPB) 1,270 images Computer Vision / Intersectional gender and skin tone parity
    WinoBias 3,160 sentences NLP / Gender bias in coreference resolution across occupations
    Communities and Crime 1,994 crime records Social / Socio-economic and racial crime statistics
    COMPAS Dataset 18,610 crime records Social / Racial disparities in recidivism risk scoring
    Recidivism in Juvenile Justice 4,753 crime records Social / Juvenile recidivism risk assessment
    Diversity in Faces (DiF) 1,000,000 images Computer Vision / Facial diversity (skin tone, craniofacial ratios, pose)

    These datasets provide standard empirical testbeds for measuring demographic parity, error calibration, counterfactual outcomes, and representation fairness across protected attributes such as race, sex, and age.

  10. Knowl 10 — Inherent Incompatibility of Calibration and Error Rate Parity

    theoretical result

    In algorithmic classification and risk assessment, satisfying calibration within groups (test fairness) and balance across positive and negative classes (equal true positive rates and equal false positive rates across demographic groups) simultaneously is mathematically impossible except under highly constrained degenerate conditions (namely, when the predictor achieves perfect deterministic classification accuracy or when base rate prevalences across all demographic groups are identical).

    If the base rates of the true outcome YY differ across protected groups (P(Y=1∣A=0)≠P(Y=1∣A=1)P(Y=1 \mid A=0) \neq P(Y=1 \mid A=1)), any decision rule based on a score SS calibrated within each group cannot equalize both false positive rates P(Y^=1∣Y=0,A)P(\hat{Y}=1 \mid Y=0, A) and false negative rates P(Y^=0∣Y=1,A)P(\hat{Y}=0 \mid Y=1, A) across group boundaries. Consequently, machine learning practitioners must select mutually compatible fairness criteria tailored to the domain constraints.

Coverage note — Domain-specific empirical debiasing case studies (such as specific word embedding projection methods, coreference resolution augmentation rules, and community detection pipelines) were summarized within the broader algorithmic taxonomy and benchmark knowls rather than extracted as individual knowls.

References

  1. 1.Alekh Agarwal, Miroslav Dudik, and Zhiwei Steven Wu. 2019. Fair Regression: Quantitative Definitions and Reduction-Based Algorithms. In International Conference on Machine Learning. 120–129.
  2. 2.Sina Aghaei, Mohammad Javad Azizi, and Phebe Vayanos. 2019. Learning optimal and fair decision trees for non-discriminative decision-making. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33. 1418–1426.
  3. 3.Nazanin Alipourfard, Peter G Fennell, and Kristina Lerman. 2018. Can you Trust the Trend?: Discovering Simpson’s Paradoxes in Social Data. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining. ACM, 19–27.
  4. 4.Nazanin Alipourfard, Peter G Fennell, and Kristina Lerman. 2018. Using Simpson’s Paradox to Discover Interesting Patterns in Behavioral Data. In Twelfth International AAAI Conference on Web and Social Media.
  5. 5.Alexander Amini, Ava Soleimany, Wilko Schwarting, Sangeeta Bhatia, and Daniela Rus. 2019. Uncovering and Mitigating Algorithmic Bias through Learned Latent Structure. (2019).
  6. 6.Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. 2016. Machine Bias: there’s software used across the country to predict future criminals. And it’s biased against blacks. ProPublica 2016.
  7. 7.A. Asuncion and D.J. Newman. 2007. UCI Machine Learning Repository. http://www.ics.uci.edu/$\sim$mlearn/{MLR}epository.html
  8. 8.Arturs Backurs, Piotr Indyk, Krzysztof Onak, Baruch Schieber, Ali Vakilian, and Tal Wagner. 2019. Scalable Fair Clustering. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, Long Beach, California, USA, 405–413. http://proceedings.mlr.press/v97/backurs19a.html
  9. 9.Ricardo Baeza-Yates. 2018. Bias on the Web. Commun. ACM 61, 6 (May 2018), 54–61. https://doi.org/10.1145/3209581
  10. 10.Samuel Barbosa, Dan Cosley, Amit Sharma, and Roberto M. Cesar-Jr. 2016. Averaging Gone Wrong: Using Time-Aware Analyses to Better Understand Behavior. (April 2016), 829–841.
  11. 11.Rachel KE Bellamy, Kuntal Dey, Michael Hind, Samuel C Hoffman, Stephanie Houde, Kalapriya Kannan, Pranay Lohia, Jacquelyn Martino, Sameep Mehta, Aleksandra Mojsilovic, et al. 2018. Ai fairness 360: An extensible toolkit for detecting, understanding, and mitigating unwanted algorithmic bias. arXiv preprint arXiv:1810.01943 (2018).
  12. 12.Emily M. Bender and Batya Friedman. 2018. Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science. Transactions of the Association for Computational Linguistics 6 (2018), 587–604. https://doi.org/10.1162/tacl_a_00041
  13. 13.Misha Benjamin, Paul Gagnon, Negar Rostamzadeh, Chris Pal, Yoshua Bengio, and Alex Shee. [n.d.]. TOWARDS STANDARDIZATION OF DATA LICENSES: THE MONTREAL DATA LICENSE. ([n. d.]).
  14. 14.Richard Berk, Hoda Heidari, Shahin Jabbari, Matthew Joseph, Michael Kearns, Jamie Morgenstern, Seth Neel, and Aaron Roth. 2017. A Convex Framework for Fair Regression. arXiv:1706.02409 [cs.LG]
  15. 15.Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. [n.d.]. Fairness in criminal justice risk assessments: The state of the art. Sociological Methods & Research ([n. d.]), 0049124118782533.
  16. 16.Peter J Bickel, Eugene A Hammel, and J William O’Connell. 1975. Sex bias in graduate admissions: Data from Berkeley. Science 187, 4175 (1975), 398–404.
  17. 17.RDP Binns. 2018. Fairness in machine learning: Lessons from political philosophy. Journal of Machine Learning Research (2018).
  18. 18.Colin R Blyth. 1972. On Simpson’s paradox and the sure-thing principle. J. Amer. Statist. Assoc. 67, 338 (1972), 364–366.
  19. 19.Miranda Bogen and Aaron Rieke. 2018. Help wanted: an examination of hiring algorithms, equity. Technical Report. and bias. Technical report, Upturn.
  20. 20.Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In Advances in neural information processing systems. 4349–4357.
  21. 21.Shikha Bordia and Samuel Bowman. 2019. Identifying and Reducing Gender Bias in Word-Level Language Models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop. 7–15.
  22. 22.Avishek Bose and William Hamilton. 2019. Compositional Fairness Constraints for Graph Embeddings. In International Conference on Machine Learning. 715–724.
  23. 23.Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, and Richard Zemel. 2019. Understanding the Origins of Bias in Word Embeddings. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, Long Beach, California, USA, 803–811. http://proceedings.mlr.press/v97/brunet19a.html
  24. 24.Joy Buolamwini and Timnit Gebru. 2018. Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 77–91. http://proceedings.mlr.press/v81/buolamwini18a.html
  25. 25.Toon Calders and Sicco Verwer. 2010. Three naive Bayes approaches for discrimination-free classification. Data Mining and Knowledge Discovery 21, 2 (2010), 277–292.
  26. 26.Aylin Caliskan, Joanna J Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science 356, 6334 (2017), 183–186.
  27. 27.Flavio Calmon, Dennis Wei, Bhanukiran Vinzamuri, Karthikeyan Natesan Ramamurthy, and Kush R Varshney. 2017. Optimized Pre-Processing for Discrimination Prevention. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 3992–4001. http://papers.nips.cc/paper/6988-optimized-pre-processing-for-discrimination-prevention.pdf
  28. 28.Manel Capdevila, Marta Ferrer, and Eulália Luque. 2005. La reincidencia en el delito en la justicia de menores. Centro de estudios jurídicos y formación especializada, Generalitat de Catalunya. Documento no publicado (2005).
  29. 29.Allison JB Chaney, Brandon M Stewart, and Barbara E Engelhardt. 2018. How algorithmic confounding in recommendation systems increases homogeneity and decreases utility. In Proceedings of the 12th ACM Conference on Recommender Systems. ACM, 224–232.
  30. 30.Jiahao Chen, Nathan Kallus, Xiaojie Mao, Geoffry Svacha, and Madeleine Udell. 2019. Fairness Under Unawareness: Assessing Disparity When Protected Class Is Unobserved. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 339–348.
  31. 31.Xingyu Chen, Brandon Fain, Liang Lyu, and Kamesh Munagala. 2019. Proportionally Fair Clustering. In International Conference on Machine Learning. 1032–1041.
  32. 32.S. Chiappa. 2019. Path-specific Counterfactual Fairness. In Thirty-Third AAAI Conference on Artificial Intelligence. 7801–7808.
  33. 33.S. Chiappa and W. S. Isaac. 2019. A Causal Bayesian Networks Viewpoint on Fairness. In E. Kosta, J. Pierson, D. Slamanig, S. Fischer-Hübner, S. Krenn (eds) Privacy and Identity Management. Fairness, Accountability, and Transparency in the Age of Big Data. Privacy and Identity 2018. IFIP Advances in Information and Communication Technology, Vol. 547. Springer, Cham.
  34. 34.Alexandra Chouldechova. 2017. Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big data 5, 2 (2017), 153–163.
  35. 35.Alexandra Chouldechova, Diana Benavides-Prado, Oleksandr Fialko, and Rhema Vaithianathan. 2018. A case study of algorithm-assisted decision making in child maltreatment hotline screening decisions. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 134–148. http://proceedings.mlr.press/v81/chouldechova18a.html
  36. 36.Alexandra Chouldechova and Aaron Roth. 2018. The frontiers of fairness in machine learning. arXiv preprint arXiv:1810.08810 (2018).
  37. 37.John S Chuang, Olivier Rivoire, and Stanislas Leibler. 2009. Simpson’s paradox in a synthetic microbial system. Science 323, 5911 (2009), 272–275.
  38. 38.Kevin A Clarke. 2005. The phantom menace: Omitted variable bias in econometric research. Conflict management and peace science 22, 4 (2005), 341–352.
  39. 39.Lee Cohen, Zachary C. Lipton, and Yishay Mansour. 2019. Efficient candidate screening under multiple tests and implications for fairness. arXiv:1905.11361 [cs.LG]
  40. 40.United States. Equal Employment Opportunity Commission. [n.d.]. EEOC compliance manual. [Washington, D.C.] : U.S. Equal Employment Opportunity Commission, [1992].
  41. 41.Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, and Aziz Huq. 2017. Algorithmic decision making and the cost of fairness. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 797–806.
  42. 42.Elliot Creager, David Madras, Joern-Henrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. 2019. Flexibly Fair Representation Learning by Disentanglement. In International Conference on Machine Learning. 1436–1445.
  43. 43.Brian d’Alessandro, Cathy O’Neil, and Tom LaGatta. 2017. Conscientious classification: A data scientist’s guide to discrimination-aware classification. Big data 5, 2 (2017), 120–134.
  44. 44.David Danks and Alex John London. 2017. Algorithmic Bias in Autonomous Systems.. In IJCAI. 4691–4697.
  45. 45.Shai Danziger, Jonathan Levav, and Liora Avnaim-Pesso. 2011. Extraneous factors in judicial decisions. Proceedings of the National Academy of Sciences 108, 17 (2011), 6889–6892.
  46. 46.Julia Dressel and Hany Farid. 2018. The accuracy, fairness, and limits of predicting recidivism. Science Advances 4, 1 (2018). https://doi.org/10.1126/sciadv.aao5580 arXiv:https://advances.sciencemag.org/content/4/1/eaao5580.full.pdf
  47. 47.Dheeru Dua and Casey Graff. 2017. UCI Machine Learning Repository. http://archive.ics.uci.edu/ml
  48. 48.Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. 2012. Fairness Through Awareness. In Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (Cambridge, Massachusetts) (ITCS ’12). ACM, New York, NY, USA, 214–226. https://doi.org/10.1145/2090236.2090255
  49. 49.Cynthia Dwork, Nicole Immorlica, Adam Tauman Kalai, and Max Leiserson. 2018. Decoupled Classifiers for Group-Fair and Efficient Machine Learning. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 119–133. http://proceedings.mlr.press/v81/dwork18a.html
  50. 50.Golnoosh Farnadi, Behrouz Babaki, and Lise Getoor. 2018. Fairness in Relational Domains. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society (New Orleans, LA, USA) (AIES ’18). ACM, New York, NY, USA, 108–114. https://doi.org/10.1145/3278721.3278733
  51. 51.Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Scheidegger, and Suresh Venkatasubramanian. 2015. Certifying and Removing Disparate Impact. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (Sydney, NSW, Australia) (KDD ’15). Association for Computing Machinery, New York, NY, USA, 259–268. https://doi.org/10.1145/2783258.2783311
  52. 52.Joel Escudé Font and Marta R Costa-jussà. 2019. Equalizing Gender Biases in Neural Machine Translation with Word Embeddings Techniques. arXiv preprint arXiv:1901.03116 (2019).
  53. 53.Batya Friedman and Helen Nissenbaum. 1996. Bias in Computer Systems. ACM Trans. Inf. Syst. 14, 3 (July 1996), 330–347. https://doi.org/10.1145/230538.230561
  54. 54.Anna Fry, Thomas J Littlejohns, Cathie Sudlow, Nicola Doherty, Ligia Adamska, Tim Sprosen, Rory Collins, and Naomi E Allen. 2017. Comparison of Sociodemographic and Health-Related Characteristics of UK Biobank Participants With Those of the General Population. American Journal of Epidemiology 186, 9 (06 2017), 1026–1034. https://doi.org/10.1093/aje/kwx246 arXiv:http://oup.prod.sis.lan/aje/article-pdf/186/9/1026/24330720/kwx246.pdf
  55. 55.Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. [n.d.]. Datasheets for Datasets. ([n. d.]).
  56. 56.C. E. Gehlke and Katherine Biehl. 1934. Certain effects of grouping upon the size of the correlation coefficient in census tract material. J. Amer. Statist. Assoc. 29, 185A (1934), 169–170. https://doi.org/10.2307/2277827
  57. 57.Naman Goel, Mohammad Yaghini, and Boi Faltings. 2018. Non-discriminatory machine learning through convex fairness criteria. In Thirty-Second AAAI Conference on Artificial Intelligence.
  58. 58.Hila Gonen and Yoav Goldberg. 2019. Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them. arXiv preprint arXiv:1903.03862 (2019).
  59. 59.Sandra González-Bailón, Ning Wang, Alejandro Rivero, Javier Borge-Holthoefer, and Yamir Moreno. 2014. Assessing the bias in samples of large online networks. Social Networks 38 (2014), 16–27.
  60. 60.Susan T Gooden. 2015. Race and social equity: A nervous area of government. Routledge.
  61. 61.Nina Grgic-Hlaca, Muhammad Bilal Zafar, Krishna P Gummadi, and Adrian Weller. 2016. The case for process fairness in learning: Feature selection for fair decision making. In NIPS Symposium on Machine Learning and the Law, Vol. 1. 2.
  62. 62.S. Hajian and J. Domingo-Ferrer. 2013. A Methodology for Direct and Indirect Discrimination Prevention in Data Mining. IEEE Transactions on Knowledge and Data Engineering 25, 7 (July 2013), 1445–1459. https://doi.org/10.1109/TKDE.2012.72
  63. 63.Moritz Hardt, Eric Price, Nati Srebro, et al. 2016. Equality of opportunity in supervised learning. In Advances in neural information processing systems. 3315–3323.
  64. 64.Eszter Hargittai. 2007. Whose Space? Differences among Users and Non-Users of Social Network Sites. Journal of Computer-Mediated Communication 13, 1 (10 2007), 276–297. https://doi.org/10.1111/j.1083-6101.2007.00396.x arXiv:http://oup.prod.sis.lan/jcmc/article-pdf/13/1/276/22317170/jjcmcom0276.pdf
  65. 65.Yuzi He, Keith Burghardt, and Kristina Lerman. 2020. A Geometric Solution to Fair Representations. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society. 279–285.
  66. 66.Sarah Holland, Ahmed Hosny, Sarah Newman, Joshua Joseph, and Kasia Chmielinski. 2018. The dataset nutrition label: A framework to drive higher data quality standards. arXiv preprint arXiv:1805.03677 (2018).
  67. 67.Ayanna Howard and Jason Borenstein. 2018. The ugly truth about ourselves and our robot creations: the problem of bias and social inequity. Science and engineering ethics 24, 5 (2018), 1521–1536.
  68. 68.Gary B Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. 2008. Labeled faces in the wild: A database forstudying face recognition in unconstrained environments.
  69. 69.Lingxiao Huang and Nisheeth Vishnoi. 2019. Stable and Fair Classification. In International Conference on Machine Learning. 2879–2890.
  70. 70.Ben Hutchinson and Margaret Mitchell. 2019. 50 Years of Test (Un) fairness: Lessons for Machine Learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 49–58.
  71. 71.L. Introna and H. Nissenbaum. 2000. Defining the Web: the politics of search engines. Computer 33, 1 (Jan 2000), 54–62. https://doi.org/10.1109/2.816269
  72. 72.Ayush Jaiswal, Yue Wu, Wael AbdAlmageed, and Premkumar Natarajan. 2018. Unsupervised Adversarial Invariance. arXiv:1809.10083 [cs.LG]
  73. 73.Ray Jiang, Aldo Pacchiano, Tom Stepleton, Heinrich Jiang, and Silvia Chiappa. [n.d.]. Wasserstein Fair Classification. ([n. d.]).
  74. 74.F. Kamiran and T. Calders. 2009. Classifying without discriminating. In 2009 2nd International Conference on Computer, Control and Communication. 1–6. https://doi.org/10.1109/IC4.2009.4909197
  75. 75.Faisal Kamiran and Toon Calders. 2010. Classification with no discrimination by preferential sampling. In Proc. 19th Machine Learning Conf. Belgium and The Netherlands. Citeseer, 1–6.
  76. 76.Faisal Kamiran and Toon Calders. 2012. Data preprocessing techniques for classification without discrimination. Knowledge and Information Systems 33, 1 (01 Oct 2012), 1–33. https://doi.org/10.1007/s10115-011-0463-8
  77. 77.Faisal Kamiran and İndre Žliobaitȧe. 2013. Explainable and Non-explainable Discrimination in Classification. Springer Berlin Heidelberg, Berlin, Heidelberg, 155–170. https://doi.org/10.1007/978-3-642-30487-3_8
  78. 78.Toshihiro Kamishima, Shotaro Akaho, Hideki Asoh, and Jun Sakuma. 2012. Fairness-aware classifier with prejudice remover regularizer. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 35–50.
  79. 79.Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2018. Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness. In International Conference on Machine Learning. 2569–2577.
  80. 80.Michael Kearns, Seth Neel, Aaron Roth, and Zhiwei Steven Wu. 2019. An empirical study of rich subgroup fairness for machine learning. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 100–109.
  81. 81.Rogier Kievit, Willem Eduard Frankenhuis, Lourens Waldorp, and Denny Borsboom. 2013. Simpson’s paradox in psychological science: a practical guide. Frontiers in psychology 4 (2013), 513.
  82. 82.Niki Kilbertus, Mateo Rojas Carulla, Giambattista Parascandolo, Moritz Hardt, Dominik Janzing, and Bernhard Schölkopf. 2017. Avoiding discrimination through causal reasoning. In Advances in Neural Information Processing Systems. 656–666.
  83. 83.Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan. 2016. Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807 (2016).
  84. 84.Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In MT summit, Vol. 5. 79–86.
  85. 85.Emmanouil Krasanakis, Eleftherios Spyromitros-Xioufis, Symeon Papadopoulos, and Yiannis Kompatsiaris. 2018. Adaptive Sensitive Reweighting to Mitigate Bias in Fairness-aware Classification. In Proceedings of the 2018 World Wide Web Conference (Lyon, France) (WWW ’18). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 853–862. https://doi.org/10.1145/3178876.3186133
  86. 86.Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, et al. 2017. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://github. com/openimages 2, 3 (2017), 2–3.
  87. 87.Matt J Kusner, Joshua Loftus, Chris Russell, and Ricardo Silva. 2017. Counterfactual Fairness. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 4066–4076. http://papers.nips.cc/paper/6995-counterfactual-fairness.pdf
  88. 88.Anja Lambrecht and Catherine E Tucker. 2018. Algorithmic bias? An empirical study into apparent gender-based discrimination in the display of STEM career ads. An Empirical Study into Apparent Gender-Based Discrimination in the Display of STEM Career Ads (March 9, 2018) (2018).
  89. 89.J Larson, S Mattu, L Kirchner, and J Angwin. 2016. Compas analysis. GitHub, available at: https://github.com/propublica/compas-analysis[Google Scholar] (2016).
  90. 90.Blake Lemoine, Brian Zhang, and M Mitchell. 2018. Mitigating Unwanted Biases with Adversarial Learning. (2018).
  91. 91.Kristina Lerman. 2018. Computational social scientist beware: Simpson’s paradox in behavioral data. Journal of Computational Social Science 1, 1 (2018), 49–58.
  92. 92.Kristina Lerman and Tad Hogg. 2014. Leveraging position bias to improve peer recommendation. PLoS One 9, 6 (2014), e98914. http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0098914
  93. 93.Kristina Lerman and Tad Hogg. 2014. Leveraging position bias to improve peer recommendation. PloS one 9, 6 (2014), e98914.
  94. 94.Zachary C Lipton, Alexandra Chouldechova, and Julian McAuley. 2017. Does mitigating ML’s disparate impact require disparate treatment? stat 1050 (2017), 19.
  95. 95.Lydia T Liu, Sarah Dean, Esther Rolf, Max Simchowitz, and Moritz Hardt. 2018. Delayed Impact of Fair Machine Learning. In Proceedings of the 35th International Conference on Machine Learning.
  96. 96.Joshua R Loftus, Chris Russell, Matt J Kusner, and Ricardo Silva. 2018. Causal reasoning for algorithmic fairness. arXiv preprint arXiv:1805.05859 (2018).
  97. 97.Christos Louizos, Kevin Swersky, Yujia Li, Max Welling, and Richard Zemel. 2016. THE VARIATIONAL FAIR AUTOENCODER. stat 1050 (2016), 4.
  98. 98.Arjun K. Manrai, Birgit H. Funke, Heidi L. Rehm, Morten S. Olesen, Bradley A. Maron, Peter Szolovits, David M. Margulies, Joseph Loscalzo, and Isaac S. Kohane. 2016. Genetic Misdiagnoses and the Potential for Health Disparities. New England Journal of Medicine 375, 7 (2016), 655–665. https://doi.org/10.1056/NEJMsa1507092 arXiv:https://doi.org/10.1056/NEJMsa1507092 PMID: 27532831.
  99. 99.Ray Marshall. 1974. The economics of racial discrimination: A survey. Journal of Economic Literature 12, 3 (1974), 849–871.
  100. 100.Chandler May, Alex Wang, Shikha Bordia, Samuel R Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. arXiv preprint arXiv:1903.10561 (2019).
  101. 101.Ninareh Mehrabi, Thamme Gowda, Fred Morstatter, Nanyun Peng, and Aram Galstyan. 2019. Man is to Person as Woman is to Location: Measuring Gender Bias in Named Entity Recognition. arXiv preprint arXiv:1910.10872 (2019).
  102. 102.Ninareh Mehrabi, Umang Gupta, Fred Morstatter, Greg Ver Steeg, and Aram Galstyan. 2021. Attributing Fair Decisions with Attention Interventions. arXiv preprint arXiv:2109.03952 (2021).
  103. 103.Ninareh Mehrabi, Yuzhong Huang, and Fred Morstatter. 2020. Statistical Equity: A Fairness Classification Objective. arXiv preprint arXiv:2005.07293 (2020).
  104. 104.Ninareh Mehrabi, Fred Morstatter, Nanyun Peng, and Aram Galstyan. 2019. Debiasing Community Detection: The Importance of Lowly-Connected Nodes. arXiv preprint arXiv:1903.08136 (2019).
  105. 105.Ninareh Mehrabi, Pei Zhou, Fred Morstatter, Jay Pujara, Xiang Ren, and Aram Galstyan. 2021. Lawyers are Dishonest? Quantifying Representational Harms in Commonsense Knowledge Resources. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 5016–5033. https://doi.org/10.18653/v1/2021.emnlp-main.410
  106. 106.Aditya Krishna Menon and Robert C Williamson. 2018. The cost of fairness in binary classification. In Proceedings of the 1st Conference on Fairness, Accountability and Transparency (Proceedings of Machine Learning Research, Vol. 81), Sorelle A. Friedler and Christo Wilson (Eds.). PMLR, New York, NY, USA, 107–118. http://proceedings.mlr.press/v81/menon18a.html
  107. 107.Michele Merler, Nalini Ratha, Rogerio S Feris, and John R Smith. 2019. Diversity in Faces. arXiv preprint arXiv:1901.10436 (2019).
  108. 108.Hannah Jean Miller, Jacob Thebault-Spieker, Shuo Chang, Isaac Johnson, Loren Terveen, and Brent Hecht. 2016. “Blissfully Happy” or “Ready toFight”: Varying Interpretations of Emoji. In Tenth International AAAI Conference on Web and Social Media.
  109. 109.I Minchev, G Matijevic, DW Hogg, G Guiglion, M Steinmetz, F Anders, C Chiappini, M Martig, A Queiroz, and C Scannapieco. 2019. Yule-Simpson’s paradox in Galactic Archaeology. arXiv preprint arXiv:1902.01421 (2019).
  110. 110.Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (Atlanta, GA, USA) (FAT* ’19). ACM, New York, NY, USA, 220–229. https://doi.org/10.1145/3287560.3287596
  111. 111.Fred Morstatter, Jürgen Pfeffer, Huan Liu, and Kathleen M Carley. 2013. Is the sample good enough? Comparing data from twitter’s streaming API with Twitter’s firehose. In 7th International AAAI Conference on Weblogs and Social Media, ICWSM 2013. AAAI press.
  112. 112.Daniel Moyer, Shuyang Gao, Rob Brekelmans, Aram Galstyan, and Greg Ver Steeg. 2018. Invariant Representations without Adversarial Training. In Advances in Neural Information Processing Systems. 9084–9093.
  113. 113.Amitabha Mukerjee, Rita Biswas, Kalyanmoy Deb, and Amrit P Mathur. 2002. Multi–objective evolutionary algorithms for the risk–return trade–off in bank loan management. International Transactions in operational research 9, 5 (2002), 583–597.
  114. 114.David B Mustard. 2003. Reexamining criminal behavior: the importance of omitted variable bias. Review of Economics and Statistics 85, 1 (2003), 205–211.
  115. 115.Razieh Nabi, Daniel Malinsky, and Ilya Shpitser. 2018. Learning Optimal Fair Policies. arXiv preprint arXiv:1809.02244 (2018).
  116. 116.Razieh Nabi and Ilya Shpitser. 2018. Fair inference on outcomes. In Thirty-Second AAAI Conference on Artificial Intelligence.
  117. 117.Azadeh Nematzadeh, Giovanni Luca Ciampaglia, Filippo Menczer, and Alessandro Flammini. 2017. How algorithmic popularity bias hinders or promotes quality. arXiv preprint arXiv:1707.00574 (2017).
  118. 118.Dong-Phuong Nguyen, Rilana Gravel, Rudolf Berend Trieschnigg, and Theo Meder. 2013. "How old do you think I am?": A study of language and age in Twitter. In Proceedings of the Seventh International AAAI Conference on Weblogs and Social Media, ICWSM 2013. AAAI Press, 439–448. eemcs-eprint-23604.
  119. 119.Anne O’Keeffe and Michael McCarthy. 2010. The Routledge handbook of corpus linguistics. Routledge.
  120. 120.Alexandra Olteanu, Carlos Castillo, Fernando Diaz, and Emre Kiciman. 2016. Social data: Biases, methodological pitfalls, and ethical boundaries. (2016).
  121. 121.Cathy O’Neil. 2016. Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. Crown Publishing Group, New York, NY, USA.
  122. 122.Luca Oneto, Michele Doninini, Amon Elders, and Massimiliano Pontil. 2019. Taking advantage of multitask learning for fair classification. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. 227–237.
  123. 123.Osonde A Osoba and William Welser IV. 2017. An intelligence in our image: The risks of bias and errors in artificial intelligence. Rand Corporation.
  124. 124.Edmund S Phelps. 1972. The statistical theory of racism and sexism. The american economic review 62, 4 (1972), 659–661.
  125. 125.Geoff Pleiss, Manish Raghavan, Felix Wu, Jon Kleinberg, and Kilian Q Weinberger. 2017. On Fairness and Calibration. In Advances in Neural Information Processing Systems 30, I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (Eds.). Curran Associates, Inc., 5680–5689. http://papers.nips.cc/paper/7151-on-fairness-and-calibration.pdf
  126. 126.Marcelo OR Prates, Pedro H Avelar, and Luís C Lamb. 2018. Assessing gender bias in machine translation: a case study with Google Translate. Neural Computing and Applications (2018), 1–19.
  127. 127.Bilal Qureshi, Faisal Kamiran, Asim Karim, and Salvatore Ruggieri. 2016. Causal discrimination discovery through propensity score analysis. arXiv preprint arXiv:1608.03735 (2016).
  128. 128.Inioluwa Deborah Raji and Joy Buolamwini. 2019. Actionable auditing: Investigating the impact of publicly naming biased performance results of commercial ai products.
  129. 129.M Redmond. 2011. Communities and crime unnormalized data set. UCI Machine Learning Repository. In website: http://www. ics. uci. edu/mlearn/MLRepository. html (2011).
  130. 130.Willy E Rice. 1996. Race, Gender, Redlining, and the Discriminatory Access to Loans, Credit, and Insurance: An Historical and Empirical Analysis of Consumers Who Sued Lenders and Insurers in Federal and State Courts, 1950-1995. San Diego L. Rev. 33 (1996), 583.
  131. 131.Stephanie K Riegg. 2008. Causal inference and omitted variable bias in financial aid research: Assessing solutions. The Review of Higher Education 31, 3 (2008), 329–354.
  132. 132.Lauren A Rivera. 2012. Hiring as cultural matching: The case of elite professional service firms. American sociological review 77, 6 (2012), 999–1022.
  133. 133.Andrea Romei and Salvatore Ruggieri. 2011. A multidisciplinary survey on discrimination analysis.
  134. 134.Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender Bias in Coreference Resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers). Association for Computational Linguistics, New Orleans, Louisiana, 8–14. https://doi.org/10.18653/v1/N18-2002
  135. 135.Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision 115, 3 (2015), 211–252.
  136. 136.Pedro Saleiro, Benedict Kuester, Abby Stevens, Ari Anisfeld, Loren Hinkson, Jesse London, and Rayid Ghani. 2018. Aequitas: A Bias and Fairness Audit Toolkit. arXiv preprint arXiv:1811.05577 (2018).
  137. 137.Samira Samadi, Uthaipon Tantipongpipat, Jamie Morgenstern, Mohit Singh, and Santosh Vempala. 2018. The Price of Fair PCA: One Extra Dimension. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems (Montréal, Canada) (NIPS’18). Curran Associates Inc., USA, 10999–11010. http://dl.acm.org/citation.cfm?id=3327546.3327755
  138. 138.Nripsuta Ani Saxena. 2019. Perceptions of Fairness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society (Honolulu, HI, USA) (AIES ’19). ACM, New York, NY, USA, 537–538. https://doi.org/10.1145/3306618.3314314
  139. 139.Nripsuta Ani Saxena, Karen Huang, Evan DeFilippis, Goran Radanovic, David C Parkes, and Yang Liu. 2019. How Do Fairness Definitions Fare?: Examining Public Attitudes Towards Algorithmic Definitions of Fairness. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society. ACM, 99–106.
  140. 140.Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims. 2016. Recommendations as Treatments: Debiasing Learning and Evaluation. In International Conference on Machine Learning. 1670–1679.
  141. 141.Andrew D Selbst, Danah Boyd, Sorelle A Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. Fairness and abstraction in sociotechnical systems. In Proceedings of the Conference on Fairness, Accountability, and Transparency. ACM, 59–68.
  142. 142.Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. 2017. No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World. stat 1050 (2017), 22.
  143. 143.Richard Shaw and Manuel Corpas. [n.d.]. Further bias in personal genomics? ([n. d.]).
  144. 144.Harini Suresh and John V Guttag. 2019. A Framework for Understanding Unintended Consequences of Machine Learning. arXiv preprint arXiv:1901.10002 (2019).
  145. 145.Songül Tolan, Marius Miron, Emilia Gómez, and Carlos Castillo. 2019. Why Machine Learning May Lead to Unfairness: Evidence from Risk Assessment for Juvenile Justice in Catalonia. (2019).
  146. 146.Zeynep Tufekci. 2014. Big questions for social media big data: Representativeness, validity and other methodological pitfalls. In Eighth International AAAI Conference on Weblogs and Social Media.
  147. 147.Berk Ustun, Yang Liu, and David Parkes. 2019. Fairness without Harm: Decoupled Classifiers with Preference Guarantees. In Proceedings of the 36th International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 97), Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, Long Beach, California, USA, 6373–6382. http://proceedings.mlr.press/v97/ustun19a.html
  148. 148.Eva Vanmassenhove, Christian Hardmeier, and Andy Way. 2018. Getting gender right in neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 3003–3008.
  149. 149.Sahil Verma and Julia Rubin. 2018. Fairness definitions explained. In 2018 IEEE/ACM International Workshop on Software Fairness (FairWare). IEEE, 1–7.
  150. 150.Selwyn Vickers, Mona Fouad, and Moon S Chen Jr. 2014. Enhancing Minority Participation in Clinical Trials (EMPaCT): laying the groundwork for improving minority clinical trial accrual. Cancer 120 (2014), vi–vii.
  151. 151.Ting Wang and Dashun Wang. 2014. Why Amazon’s ratings might mislead you: The story of herding effects. Big data 2, 4 (2014), 196–204.
  152. 152.Steven L Willborn. 1984. The disparate impact model of discrimination: Theory and limits. Am. UL Rev. 34 (1984), 799.
  153. 153.Christo Wilson, Bryce Boe, Alessandra Sala, Krishna PN Puttaswamy, and Ben Y Zhao. 2009. User interactions in social networks and their implications. In Proceedings of the 4th ACM European conference on Computer systems. Acm, 205–218.
  154. 154.Blake Woodworth, Suriya Gunasekar, Mesrob I Ohannessian, and Nathan Srebro. 2017. Learning non-discriminatory predictors. arXiv preprint arXiv:1702.06081 (2017).
  155. 155.Yongkai Wu, Lu Zhang, and Xintao Wu. 2018. Fairness-aware Classification: Criterion, Convexity, and Bounds. arXiv:1809.04737 [cs.LG]
  156. 156.Depeng Xu, Shuhan Yuan, Lu Zhang, and Xintao Wu. 2018. Fairgan: Fairness-aware generative adversarial networks. In 2018 IEEE International Conference on Big Data (Big Data). IEEE, 570–575.
  157. 157.Irene Y Chen, Peter Szolovits, and Marzyeh Ghassemi. 2019. Can AI Help Reduce Disparities in General Medical and Mental Health Care? AMA journal of ethics 21 (02 2019), E167–179. https://doi.org/10.1001/amajethics.2019.167
  158. 158.Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. 2017. Fairness beyond disparate treatment & disparate impact: Learning classification without disparate mistreatment. In Proceedings of the 26th international conference on world wide web. 1171–1180.
  159. 159.Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi. 2015. Fairness constraints: Mechanisms for fair classification. arXiv preprint arXiv:1507.05259 (2015).
  160. 160.Lu Zhang and Xintao Wu. 2017. Anti-discrimination learning: a causal modeling-based framework. International Journal of Data Science and Analytics 4, 1 (01 Aug 2017), 1–16. https://doi.org/10.1007/s41060-017-0058-x
  161. 161.Lu Zhang, Yongkai Wu, and Xintao Wu. 2016. On Discrimination Discovery Using Causal Networks. In Social, Cultural, and Behavioral Modeling, Kevin S. Xu, David Reitter, Dongwon Lee, and Nathaniel Osgood (Eds.). Springer International Publishing, Cham, 83–93.
  162. 162.Lu Zhang, Yongkai Wu, and Xintao Wu. 2016. Situation Testing-based Discrimination Discovery: A Causal Inference Approach. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (New York, New York, USA) (IJCAI’16). AAAI Press, 2718–2724. http://dl.acm.org/citation.cfm?id=3060832.3061001
  163. 163.Lu Zhang, Yongkai Wu, and Xintao Wu. 2017. Achieving non-discrimination in data release. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. ACM, 1335–1344.
  164. 164.Lu Zhang, Yongkai Wu, and Xintao Wu. 2017. A Causal Framework for Discovering and Removing Direct and Indirect Discrimination. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17. 3929–3935. https://doi.org/10.24963/ijcai.2017/549
  165. 165.L. Zhang, Y. Wu, and X. Wu. 2018. Causal Modeling-Based Discrimination Discovery and Removal: Criteria, Bounds, and Algorithms. IEEE Transactions on Knowledge and Data Engineering (2018), 1–1. https://doi.org/10.1109/TKDE.2018.2872988
  166. 166.Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, and Kai-Wei Chang. 2019. Gender Bias in Contextualized Word Embeddings. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 629–634.
  167. 167.Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2017. Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing.
  168. 168.Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods. arXiv:1804.06876 [cs.CL]
  169. 169.Jieyu Zhao, Yichao Zhou, Zeyu Li, Wei Wang, and Kai-Wei Chang. 2018. Learning Gender-Neutral Word Embeddings. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 4847–4853.
  170. 170.James Zou and Londa Schiebinger. 2018. AI can be sexist and racist it’s time to make it fair. Nature Publishing Group.

Citation

MLA
Mehrabi, N., et al. “A Survey on Bias and Fairness in Machine Learning”. arXiv, 2019, http://arxiv.org/abs/1908.09635v3.
APA
Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2019). A Survey on Bias and Fairness in Machine Learning. arXiv. http://arxiv.org/abs/1908.09635v3
Chicago
Mehrabi, N., F. Morstatter, N. Saxena, K. Lerman, and A. Galstyan. 2019. “A Survey on Bias and Fairness in Machine Learning”. arXiv. http://arxiv.org/abs/1908.09635v3.
Harvard
Mehrabi, N. et al. (2019) “A Survey on Bias and Fairness in Machine Learning”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1908.09635v3.
Vancouver
1. Mehrabi N, Morstatter F, Saxena N, Lerman K, Galstyan A (2019) A Survey on Bias and Fairness in Machine Learning. arXiv

BibTeX

@article{mehrabi2019survey,
  title = {A Survey on Bias and Fairness in Machine Learning},
  author = {Mehrabi, Ninareh and Morstatter, Fred and Saxena, Nripsuta and Lerman, Kristina and Galstyan, Aram},
  year = {2019},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1908.09635v3},
  eprint = {1908.09635}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF