---
id: SAFE-ETHICS-001
title: "AI Ethics, Bias and Fairness"
alternative_names:
  - "Algorithmic Fairness"
  - "Responsible AI"
  - "Fairness, Accountability and Transparency (FAccT)"
category: "Safety & Governance → Ethics → Bias and Fairness"
volume: VIII
chapter: 47
entry_type: "Standard Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-18
version: "0.2.0"
prerequisites:
  - "FOUND-001"
related_entries:
  - "SAFE-PRIV-001"
  - "SAFE-ROBUST-001"
  - "SAFE-ALIGN-001"
  - "GOV-LAW-001"
  - "TECH-ML-001"
  - "TECH-DATA-001"
---

# AI Ethics, Bias and Fairness

## 1. Concise Definition
**AI ethics** is the study of moral questions raised by the design, deployment and governance of AI systems. **Algorithmic bias** refers to systematic and unfair differences in a system's outputs or error rates across individuals or groups, and **algorithmic fairness** is the research area that defines, measures and attempts to reduce such differences (Mehrabi et al., 2021; Dwork et al., 2012).

## 2. Overview & Significance
AI systems are used to score credit, allocate health care, screen applicants, assess criminal risk and moderate content. Errors in these settings are not evenly distributed, and statistical learning can reproduce or amplify patterns in historical data. Mehrabi et al. (2021) survey real-world applications that have shown bias, catalogue sources of bias that can affect AI applications, and propose a taxonomy of fairness definitions. The central technical finding of the fairness literature is that several intuitively appealing fairness criteria cannot all be satisfied at once, so "fairness" is a set of value-laden choices, not a single measurable property (Kleinberg et al., 2016; Chouldechova, 2017).

Ethics debates extend beyond group fairness to transparency, accountability, privacy, labour, environmental cost and concentration of power. Jobin et al. (2019) mapped the corpus of AI ethics principles and guidelines and reported a global convergence emerging around five principles (transparency, justice and fairness, non-maleficence, responsibility and privacy), with substantive divergence on interpretation and implementation. This article focuses on bias and fairness; privacy is treated in [SAFE-PRIV-001], and legal regimes in [GOV-LAW-001].

## 3. Historical Development
The papers below, from 2011 to 2021, are among the widely cited works in algorithmic fairness. Dwork et al. (2012) formalised *individual fairness* ("fairness through awareness"). Bolukbasi et al. (2016) showed that word embeddings trained on Google News articles exhibit gender stereotypes and that gender bias is captured by a direction in the embedding space. Public attention intensified with the 2016 ProPublica analysis of the COMPAS recidivism-risk tool, which found that Black defendants were almost twice as likely as white defendants to be wrongly labelled higher risk (44.9% versus 23.5%) while white defendants were more often mislabelled lower risk (47.7% versus 28.0%) (Angwin et al., 2016). The developer of COMPAS disputed the methodology (as reported in the same article). Within roughly nine months, Kleinberg et al. (arXiv, September 2016) and Chouldechova (arXiv, February 2017) showed formally that the competing criteria at the heart of that dispute cannot generally be satisfied together. Hardt et al. (2016) proposed *equality of opportunity* as a criterion. Caliskan et al. (2017) showed that word-association tests applied to corpora reproduce human-like biases, and Buolamwini and Gebru (2018) documented intersectional accuracy disparities in commercial facial-analysis systems. Obermeyer et al. (2019) analysed a health-care algorithm in wide use in the United States. Bender et al. (2021) discussed risks of ever-larger language models.

## 4. Technical / Conceptual Explanation

### 4.1 Sources of bias
Mehrabi et al. (2021) distinguish biases arising in data (for example unrepresentative samples), in algorithms (design and optimisation choices) and in user interaction. Two concrete mechanisms illustrate this. *Representation:* Buolamwini and Gebru (2018) found that two widely used face benchmarks were overwhelmingly composed of lighter-skinned subjects (79.6% in IJB-A and 86.2% in Adience), and that in three commercial gender-classification systems darker-skinned women were the most misclassified group, with error rates up to 34.7% versus a maximum of 0.8% for lighter-skinned men (systems as tested in 2018; the authors' results are not a statement about current systems). *Proxy labels:* Obermeyer et al. (2019) found that one widely used algorithm used health costs as a proxy for health needs; because less money was spent on Black patients with the same level of need, Black patients assigned the same risk score were sicker, and the authors estimated that the bias reduced the number of Black patients identified for extra care by more than half. Reformulating the algorithm to stop using cost as a proxy removed the bias in predicting who needs extra care.

### 4.2 Formal fairness criteria
Let $Y$ be the true outcome, $\hat Y$ the decision, $S$ a risk score and $A$ a protected attribute.
- **Demographic (statistical) parity:** $\Pr(\hat Y=1\mid A=a)=\Pr(\hat Y=1\mid A=b)$.
- **Equalised odds / equality of opportunity:** $\Pr(\hat Y=1\mid Y=y, A=a)=\Pr(\hat Y=1\mid Y=y, A=b)$ for $y\in\{0,1\}$ (equalised odds), or for $y=1$ only (equality of opportunity) (Hardt et al., 2016).
- **Calibration within groups:** $\Pr(Y=1\mid S=s, A=a)=s$ for each group.
- **Individual fairness:** similar individuals receive similar outcomes; formally, a mapping $M$ from individuals to distributions over outcomes is Lipschitz with respect to a task-specific similarity metric $d$ (Dwork et al., 2012).

### 4.3 Impossibility results
Kleinberg et al. (2016) formalise three fairness conditions for probabilistic classification and prove that, except in highly constrained special cases, no method satisfies all three simultaneously. Chouldechova (2017) shows that criteria such as predictive parity and error-rate balance cannot all hold when prevalence differs across groups, and that disparate impact can arise when a recidivism instrument fails to satisfy error-rate balance. The practical consequence is that choosing a criterion is a normative decision that statistics alone cannot make.

### 4.4 Mitigation
Approaches are conventionally grouped into pre-processing (re-weighting or debiasing data or representations, for example removing a gender direction from embeddings; Bolukbasi et al., 2016), in-processing (constrained or regularised training) and post-processing (adjusting decisions or thresholds, as in Hardt et al., 2016). Mehrabi et al. (2021) review such methods. Methods that reduce one disparity metric can worsen another.

## 5. Types, Components & Classifications
| Category | Question | Example source |
| :--- | :--- | :--- |
| Group fairness | Do outcomes or error rates match across groups? | Hardt et al. (2016); Kleinberg et al. (2016) |
| Individual fairness | Are similar people treated similarly? | Dwork et al. (2012) |
| Representational harm | Does the system reproduce stereotypes? | Bolukbasi et al. (2016); Caliskan et al. (2017) |
| Allocative harm | Does it unfairly allocate resources or opportunities? | Obermeyer et al. (2019) |
| Performance disparity | Does accuracy differ across subgroups? | Buolamwini & Gebru (2018) |
| Systemic / societal | Risks of ever-larger language models | Bender et al. (2021) |

## 6. Applications & Concrete Examples
- **Criminal justice.** Risk-assessment instruments such as COMPAS were analysed by journalists and academics (Angwin et al., 2016; Chouldechova, 2017); the dispute illustrates incompatible fairness criteria.
- **Health care.** A proxy-label bias in a deployed risk algorithm and its correction (Obermeyer et al., 2019).
- **Computer vision.** Subgroup performance auditing of commercial gender classifiers (Buolamwini & Gebru, 2018).
- **Language technology.** Embedding debiasing and association tests (Bolukbasi et al., 2016; Caliskan et al., 2017). Bender et al. (2021) discuss risks of large language models. [VERIFICATION NEEDED: the specific risks (training-data documentation, encoded bias) were not checked against the paper text.]
- **Policy instruments.** UNESCO's *Recommendation on the Ethics of Artificial Intelligence* was adopted in November 2021 by UNESCO's 193 member states (as stated on the UNESCO page) and lists ten principles including fairness and non-discrimination, transparency and explainability, and human oversight and determination. The NIST AI Risk Management Framework 1.0 (NIST AI 100-1, 26 January 2023) is a US framework "intended for voluntary use" (NIST page, read 2026-10-18); that page also states it is being revised as part of the White House AI Action Plan, so check for a newer version. For binding law see [GOV-LAW-001], [GOV-EU-001] and [GOV-US-001].

## 7. Advantages, Limitations & Unresolved Questions
- **No single metric.** Impossibility results mean trade-offs are unavoidable when base rates differ (Kleinberg et al., 2016; Chouldechova, 2017).
- **Measurement.** Audits require demographic data that may be unavailable or sensitive to collect, creating tension with privacy (see [SAFE-PRIV-001]).
- **Label and proxy bias.** Biased labels or proxies can make a model "accurate" with respect to a flawed target (Obermeyer et al., 2019).
- **Context dependence.** Whether an observed disparity is unjust depends on the decision context and values, which is a normative question outside the statistical framework.
- **Principles versus practice.** Jobin et al. (2019) report substantive divergence in how convergent principles are interpreted and implemented, so agreement on principles does not settle practice.
- **Generative models.** Fairness research developed mainly for classifiers; extensions to open-ended generation lack agreed metrics. [VERIFICATION NEEDED: current consensus on evaluating bias in generative models; not surveyed.]

## 8. Current State of Knowledge
As of 2026-10-18:
- That machine-learning systems can exhibit group-level performance and outcome disparities, and that they have in documented cases (Buolamwini & Gebru, 2018; Obermeyer et al., 2019), is **Well Supported**.
- The impossibility of simultaneously satisfying certain fairness criteria is **Established** (mathematical results under stated conditions: Kleinberg et al., 2016; Chouldechova, 2017).
- Which fairness criterion is appropriate in a given setting is **Contested**, because it is a normative judgement.
- Whether specific mitigation techniques generalise across settings is **Preliminary**; reviews list many methods but outcomes vary (Mehrabi et al., 2021).
- The COMPAS dispute is documented in a journalistic source (Tier 4) and an academic analysis; the accuracy of ProPublica's headline interpretation was disputed by the developer.

## 9. Future Trajectory & Strategic Outlook
This section contains no registered forecasts. Observed trends and open questions:
- **Near-term (0-3 years).** Bias measurement is extending to language-model and multimodal systems, and legal requirements touching on bias are appearing (see [GOV-EU-001] and [GOV-LAW-001]). [VERIFICATION NEEDED: specific audit obligations not checked in this audit.]
- **Medium-term (3-10 years).** Open questions include how to audit systems whose behaviour is context-dependent and how to reconcile fairness with privacy.
- **Long-term (10+ years).** Integration of fairness with broader alignment questions about whose values AI systems should reflect (see [SAFE-ALIGN-001]).

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[FOUND-001] Artificial Intelligence](../vol01_foundations/FOUND-001_artificial_intelligence.md): The field and its aims.
  - [[TECH-ML-001] Machine Learning](../vol03_technical/TECH-ML-001_machine_learning.md): Learning from data and the sources of bias.
- **Related Methods & Technologies**:
  - [[TECH-DATA-001] Training Data, Contamination and Provenance](../vol03_technical/TECH-DATA-001_training_data_contamination_and_provenance.md): Data representation and documentation.
  - [[SAFE-PRIV-001] Privacy and Data Protection in AI](./SAFE-PRIV-001_privacy_and_data_protection_in_ai.md): Data protection and the data needed to audit bias.
  - [[SAFE-ROBUST-001] Technical Safety and Robustness](./SAFE-ROBUST-001_technical_safety_and_robustness.md): Performance under distribution shift.
- **Advanced Implications & Future Trajectories**:
  - [[SAFE-ALIGN-001] The AI Alignment Problem](./SAFE-ALIGN-001_ai_alignment.md): Whose values and which objectives.
  - [[GOV-LAW-001] Global AI Governance & Regulation](./GOV-LAW-001_ai_governance.md): Legal responses.

## 11. References & Foundational Literature
1. Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., & Galstyan, A. (2021). *A Survey on Bias and Fairness in Machine Learning*. ACM Computing Surveys, 54(6), 1-35. https://doi.org/10.1145/3457607; arXiv:1908.09635. [Sources of bias and taxonomy of fairness definitions. Venue confirmed via Crossref.]
2. Dwork, C., Hardt, M., Pitassi, T., Reingold, O., & Zemel, R. (2012). *Fairness Through Awareness*. Proceedings of the 3rd Innovations in Theoretical Computer Science Conference (ITCS 2012), 214-226. https://doi.org/10.1145/2090236.2090255; arXiv:1104.3913. [Individual fairness. Venue and pages confirmed via Crossref; arXiv posting dated 2011.]
3. Hardt, M., Price, E., & Srebro, N. (2016). *Equality of Opportunity in Supervised Learning*. arXiv:1610.02413. [Equalised odds and equality of opportunity.]
4. Kleinberg, J., Mullainathan, S., & Raghavan, M. (2016). *Inherent Trade-Offs in the Fair Determination of Risk Scores*. arXiv:1609.05807. [Impossibility theorem for three fairness conditions.]
5. Chouldechova, A. (2017). *Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments*. Big Data, 5(2), 153-163. https://doi.org/10.1089/big.2016.0047; arXiv:1703.00056. [Criteria cannot all hold when prevalence differs.]
6. Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016, May 23). *Machine Bias*. ProPublica. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing. [Analysis of COMPAS scores in Broward County, Florida; developer disputes methodology.]
7. Buolamwini, J., & Gebru, T. (2018). *Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification*. Proceedings of the 1st Conference on Fairness, Accountability and Transparency, PMLR 81, 77-91. https://proceedings.mlr.press/v81/buolamwini18a.html. [Subgroup error rates.]
8. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). *Dissecting racial bias in an algorithm used to manage the health of populations*. Science, 366(6464), 447-453. https://doi.org/10.1126/science.aax2342. [Health-cost proxy bias.]
9. Bolukbasi, T., Chang, K.-W., Zou, J., Saligrama, V., & Kalai, A. (2016). *Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings*. arXiv:1607.06520. [Gender stereotypes in embeddings and debiasing.]
10. Caliskan, A., Bryson, J. J., & Narayanan, A. (2017). *Semantics derived automatically from language corpora contain human-like biases*. Science, 356(6334), 183-186. https://doi.org/10.1126/science.aal4230. [Word associations from text mirror human implicit associations.]
11. Bender, E. M., Gebru, T., McMillan-Major, A., & Shmitchell, S. (2021). *On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?* Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 610-623. https://doi.org/10.1145/3442188.3445922. [Risks of large language models.]
12. Jobin, A., Ienca, M., & Vayena, E. (2019). *The global landscape of AI ethics guidelines*. Nature Machine Intelligence, 1(9), 389-399. https://doi.org/10.1038/s42256-019-0088-2. [Convergence on five principles; divergence in interpretation. Bibliographic details confirmed via Crossref; abstract read from the arXiv version, arXiv:1906.11668 (titled "Artificial Intelligence: the global landscape of ethics guidelines").]
13. UNESCO. (2021). *Recommendation on the Ethics of Artificial Intelligence*. https://www.unesco.org/en/artificial-intelligence/recommendation-ethics. [Adopted November 2021 and endorsed by all 193 member states; four values and ten principles as listed on the page, read 2026-10-18.]
14. Tabassi, E. (2023). *Artificial Intelligence Risk Management Framework (AI RMF 1.0)*. NIST AI 100-1, 26 January 2023. https://doi.org/10.6028/NIST.AI.100-1. [Voluntary US risk-management framework; released 26 January 2023 per the NIST page.]

Note: arXiv entries are cited as preprints. Final venues for Refs 3, 4 and 9 (understood to be NeurIPS 2016, ITCS 2017 and NeurIPS 2016 respectively) were not confirmed against a primary record and are left unstated. [VERIFICATION NEEDED: venues for Refs 3, 4, 9.]

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Commercial gender classifiers had large subgroup error disparities (2018 systems) | Well Supported | Buolamwini & Gebru (2018) | High (dated) |
| Proxy-label bias in a health algorithm | Well Supported | Obermeyer et al. (2019) | High |
| Fairness criteria cannot all be satisfied (differing base rates) | Established | Kleinberg et al. (2016); Chouldechova (2017) | High |
| Embeddings encode gender stereotypes | Well Supported | Bolukbasi et al. (2016); Caliskan et al. (2017) | High |
| COMPAS racial error-rate disparities (ProPublica analysis) | Contested (methodology disputed) | Angwin et al. (2016); Chouldechova (2017) | Moderate |
| Convergence on five ethical principles | Well Supported (corpus analysis) | Jobin et al. (2019) | Moderate; divergence in interpretation |
| Generalisability of bias mitigation | Preliminary | Mehrabi et al. (2021) | Moderate-Low |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.
- **v0.2.0** (2026-10-18): Fact-check audit (docs/audits/SAFE-ETHICS-001_audit.md). Added confirmed venues for Refs 1, 2, 5; confirmed ProPublica, Gender Shades, Obermeyer, Jobin, UNESCO and NIST figures against primary pages; softened unsupported historical, Bender and audit-law statements; kept remaining verification markers.