---
id: CAP-LIMITS-001
title: "Hallucination, Reliability and Calibration"
alternative_names:
  - "LLM Hallucination"
  - "Confabulation"
  - "Model Calibration"
category: "Capabilities → Limitations"
volume: IV
chapter: 23
entry_type: "Standard Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "TECH-LLM-001"
related_entries:
  - "CAP-BENCH-001"
  - "CAP-REASON-001"
  - "CAP-AGENTEVAL-001"
  - "CAP-GENAI-001"
  - "SAFE-ROBUST-001"
  - "SAFE-EVAL-001"
---

# Hallucination, Reliability and Calibration

## 1. Concise Definition
**Hallucination** is the generation of content that is fluent but unsupported, either by the model's source input (a faithfulness failure) or by established fact (a factuality failure) (Ji et al., 2023; Huang et al., 2023). **Calibration** is the property that a system's expressed confidence matches its empirical accuracy: among answers given with 80% confidence, about 80% should be correct (Guo et al., 2017). **Reliability** is the consistency with which a system produces acceptable outputs across repeated use, inputs and conditions.

## 2. Overview & Significance
Language models are trained to produce plausible continuations, not to verify claims. Consequently, they can state false information in the same register as true information, which is the central obstacle to using them in settings where errors are costly. The three terms in the title are related but distinct. A system can hallucinate rarely yet be poorly calibrated (it is rarely wrong but cannot tell when it is). It can be well calibrated yet unreliable (it knows it is wrong half the time). And it can be accurate on average while unreliable per user, because average accuracy hides variance across repeated attempts.

The terminology itself is contested. "Hallucination" borrows a perceptual term, and some authors prefer "confabulation" for plausible fabrications. This article uses "hallucination" because it is the dominant term in the literature, without implying any mechanism of perception.

## 3. Historical Development
Before large language models, "hallucination" was documented in neural machine translation and summarisation, where systems produced output unsupported by the source text. Ji et al. (2023) surveyed this work across natural-language-generation tasks and organised it into intrinsic cases (output contradicts the source) and extrinsic cases (output adds material that cannot be verified from the source). Calibration of neural networks was studied separately: Guo et al. (2017) reported that modern deep networks were substantially more miscalibrated than earlier, smaller networks, and that simple post-hoc temperature scaling was an effective remedy for classifiers.

With instruction-following language models, attention shifted to factuality in open-ended answers. Lin et al. (2021) introduced TruthfulQA to test whether models reproduce common human misconceptions. Kadavath et al. (2022) examined whether models can evaluate their own correctness. Huang et al. (2023) proposed a taxonomy covering factuality and faithfulness hallucination in LLMs, and Kalai and Vempala (2023) and Kalai et al. (2025) put forward statistical accounts of why hallucination arises and persists.

## 4. Technical / Conceptual Explanation

### 4.1 Why hallucination occurs: three explanations
1. **Statistical inevitability for rare facts.** Kalai and Vempala (2023) show that a language model with calibrated predictions must hallucinate at a rate bounded below by a statistic of the training data related to facts seen once, even on idealised training data. The result concerns arbitrary facts that cannot be inferred from patterns (a particular person's birthday, say), not facts derivable by reasoning.
2. **Pretraining as classification of valid versus invalid text.** Kalai et al. (2025) argue that if incorrect statements cannot be distinguished from facts during training, errors arise from ordinary statistical pressures, in the way that misclassifications arise in binary classification.
3. **Evaluation incentives.** The same paper argues that hallucinations persist because most benchmarks grade only correct answers, so guessing scores at least as well as abstaining, and models are optimised to be good test-takers; the authors propose modifying the scoring of existing benchmarks to credit appropriate expressions of uncertainty. These are arguments from a theoretical analysis; their relative contribution in specific production systems has not been established by the sources cited here.

### 4.2 Calibration: definition and measurement
A model is perfectly calibrated if $P(\hat{Y} = Y \mid \hat{P} = p) = p$ for all $p \in [0,1]$, where $\hat{P}$ is the reported confidence and $\hat{Y}$ the prediction. The commonly reported summary, **expected calibration error (ECE)**, partitions predictions into $M$ confidence bins $B_m$ and computes

$$\mathrm{ECE} = \sum_{m=1}^{M} \frac{|B_m|}{n} \left| \mathrm{acc}(B_m) - \mathrm{conf}(B_m) \right|,$$

where $n$ is the number of predictions, $\mathrm{acc}(B_m)$ the accuracy and $\mathrm{conf}(B_m)$ the mean confidence in bin $m$ (Guo et al., 2017). ECE is sensitive to binning and does not capture whether confidence is useful for separating correct from incorrect answers; ranking quality is measured separately, for example by failure-prediction performance (Xiong et al., 2023).

### 4.3 Sources of confidence for language models
Three signals are in use. (a) **Token probabilities**: Kadavath et al. (2022) reported that larger models were well calibrated on multiple-choice questions in a suitable format and that models could be trained to estimate the probability that their own answer is correct. (b) **Verbalised confidence**: Xiong et al. (2023) evaluated black-box methods and found that models tended to be overconfident when verbalising confidence, that calibration and failure prediction improved with model capability, and that strategies such as human-inspired prompts and consistency across multiple samples helped. (c) **Semantic entropy**: Farquhar et al. (2024) estimate uncertainty over the meaning of generated answers rather than over exact token sequences, and use it to detect a class of confabulated answers. Each depends on the model family and task, and none is a general guarantee.

### 4.4 Reliability as a repeated-trial property
Average accuracy conceals inconsistency. $\tau$-bench proposed a metric, pass^k, for the probability that an agent succeeds on all of $k$ independent trials of the same task, and reported that even the state-of-the-art function-calling agents it tested (such as gpt-4o, as of the 2024 paper) succeeded on fewer than 50% of tasks and were inconsistent across trials (Yao et al., 2024). The principle, which does not depend on the particular figure, is that deployment in repeated use depends on pass^k rather than pass@1. See [CAP-AGENTEVAL-001].

## 5. Types, Components & Classifications
- **Faithfulness hallucination**: output contradicts or is unsupported by a supplied source (a summary that adds a claim not in the document).
- **Factuality hallucination**: output conflicts with world knowledge (a false statement of fact, a fabricated citation, an invented API function).
- **Intrinsic versus extrinsic**: contradicts the source versus cannot be verified from it (Ji et al., 2023).
- **Reasoning errors**: incorrect inference from correct premises; related to but distinct from hallucination (see [CAP-REASON-001]).
- **Mitigation families**: retrieval and grounding on trusted sources; verification against external tools; abstention training and reward design that credits "I don't know" (Kalai et al., 2025); decoding-time uncertainty estimation (Farquhar et al., 2024); human review.

## 6. Applications & Concrete Examples
Hallucination matters most where output is trusted without checking: citations and legal or medical references, code that calls non-existent functions, summaries of documents, and agents that act on generated content. TruthfulQA illustrates a measured form: the authors tested whether models give false answers that mimic popular misconceptions, and reported (for the models and date of their 2021 study) that the best model was substantially less truthful than humans, and that larger models were not uniformly more truthful (Lin et al., 2021). This finding concerns 2021-era models and is not evidence about current systems; it is cited for the measurement design and the observation that scale alone does not guarantee truthfulness.

## 7. Advantages, Limitations & Unresolved Questions
**Limits of mitigation.** Retrieval reduces unsupported claims only if the retrieved text is correct and the model uses it faithfully. Verification by tools is possible only where a checker exists. Abstention trades coverage for accuracy, which many benchmarks penalise (Kalai et al., 2025).

**Contested or open.** (1) Whether hallucination can be eliminated, or only reduced: Kalai and Vempala (2023) give a lower bound for calibrated models on arbitrary rare facts, which is a statement about a theoretical class of models and calibrated predictions, and does not by itself say what is achievable with retrieval and tools. (2) Whether models "know what they know" in a sense useful for deployment (Kadavath et al., 2022, report positive results in specific formats; Xiong et al., 2023, report systematic overconfidence in verbalised confidence). (3) Whether post-training for helpfulness affects calibration; the sources cited here do not settle this. (4) Metrics: no single accepted measure combines factuality, faithfulness and calibration, and results depend on prompt format and grading method (see [CAP-COMPARE-001]).

## 8. Current State of Knowledge
As of 2026-10-15 the following is supported by the cited sources. Hallucination is documented across natural-language-generation tasks (Established; Ji et al., 2023). Modern deep networks are often miscalibrated and can be recalibrated post hoc in classification settings (Well Supported; Guo et al., 2017). LLM confidence can carry useful information, with the usefulness depending on the elicitation method, model and task (Well Supported to Preliminary; Kadavath et al., 2022; Xiong et al., 2023; Farquhar et al., 2024). Theoretical accounts of why hallucination occurs and persists exist and are published, but their empirical weight in deployed systems is Preliminary (Kalai & Vempala, 2023; Kalai et al., 2025). Current hallucination rates for named models are not given here; they vary by task, grading method and date, and belong in the Living Reference (`MODEL-001`).

## 9. Future Trajectory & Strategic Outlook
Observed direction (not a forecast): evaluation practice is shifting toward scoring that credits calibrated abstention and toward repeated-trial reliability metrics rather than single-attempt accuracy (Kalai et al., 2025; Yao et al., 2024). Whether this changes model behaviour in deployed systems depends on whether widely used leaderboards adopt such scoring, which is a coordination question, not a technical one. No forecast is registered in this article.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-LLM-001] Large Language Models](../vol03_technical/TECH-LLM-001_large_language_models.md): The systems whose outputs hallucinate.
  - [[TECH-ML-001] Machine Learning](../vol03_technical/TECH-ML-001_machine_learning.md): Probability and calibration background.
- **Related Methods & Technologies**:
  - [[CAP-BENCH-001] AI Evaluation, Benchmarking & Contamination](CAP-BENCH-001_ai_evaluation.md): Benchmark design and the incentive effects discussed here.
  - [[CAP-REASON-001] Reasoning, Planning and Long-Horizon Capability](CAP-REASON-001_reasoning_planning_and_long_horizon_capability.md): Reasoning errors and per-step reliability.
  - [[CAP-AGENTEVAL-001] Evaluating Autonomous Agents](CAP-AGENTEVAL-001_evaluating_autonomous_agents.md): Repeated-trial reliability.
  - [[CAP-COMPARE-001] Methodology for Comparing AI Systems](CAP-COMPARE-001_methodology_for_comparing_ai_systems.md): Why hallucination metrics are hard to compare.
  - [[CAP-GENAI-001] Generative AI](CAP-GENAI-001_generative_ai.md): Generation across modalities.
- **Advanced Implications & Future Trajectories**:
  - [[SAFE-ROBUST-001] Technical Safety and Robustness](../vol08_safety_governance/SAFE-ROBUST-001_technical_safety_and_robustness.md): Reliability as a safety property.
  - [[SAFE-EVAL-001] Dangerous Capability and Alignment Evaluations](../vol08_safety_governance/SAFE-EVAL-001_dangerous_capability_and_alignment_evaluations.md): Evaluation of risky behaviours.

## 11. References & Foundational Literature
1. Farquhar, S., Kossen, J., Kuhn, L., & Gal, Y. (2024). *Detecting hallucinations in large language models using semantic entropy*. Nature, 630(8017), 625–630. https://doi.org/10.1038/s41586-024-07421-0. [Uncertainty estimation over meanings to detect confabulated answers].
2. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). *On Calibration of Modern Neural Networks*. arXiv:1706.04599 (ICML 2017 proceedings version not checked). [Defines ECE usage, shows modern networks are miscalibrated, and evaluates temperature scaling].
3. Huang, L., et al. (2023). *A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions*. arXiv:2311.05232. [Taxonomy of LLM hallucination].
4. Ji, Z., et al. (2023). *Survey of Hallucination in Natural Language Generation*. ACM Computing Surveys, 55(12), 1–38. https://doi.org/10.1145/3571730. [Survey and intrinsic/extrinsic taxonomy across generation tasks].
5. Kadavath, S., et al. (2022). *Language Models (Mostly) Know What They Know*. arXiv:2207.05221. [Studies model self-evaluation and calibration].
6. Kalai, A. T., Nachum, O., Vempala, S. S., & Zhang, E. (2025). *Why Language Models Hallucinate*. arXiv:2509.04664. [Argues training and evaluation incentives reward guessing over expressing uncertainty].
7. Kalai, A. T., & Vempala, S. S. (2023). *Calibrated Language Models Must Hallucinate*. arXiv:2311.14648. [Statistical lower bound on hallucination for calibrated models on arbitrary facts].
8. Lin, S., Hilton, J., & Evans, O. (2021). *TruthfulQA: Measuring How Models Mimic Human Falsehoods*. arXiv:2109.07958. [Benchmark of questions where models may reproduce common misconceptions].
9. Xiong, M., et al. (2023). *Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs*. arXiv:2306.13063. [Benchmarks verbalised and consistency-based confidence methods].
10. Yao, S., Shinn, N., Razavi, P., & Narasimhan, K. (2024). *$\tau$-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains*. arXiv:2406.12045. [Introduces pass^k and reports inconsistency of tested agents].

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Hallucination occurs across NLG tasks; intrinsic/extrinsic taxonomy | Established | Ji et al. (2023) | High |
| Modern deep networks are often miscalibrated; temperature scaling helps for classifiers | Well Supported | Guo et al. (2017) | High |
| Calibrated models must hallucinate on arbitrary rare facts | Established as a theorem under its assumptions; applicability to deployed systems is Preliminary | Kalai & Vempala (2023) | Moderate |
| Evaluation incentives sustain hallucination | Preliminary / Contested (theoretical argument, not a controlled field study) | Kalai et al. (2025) | Moderate |
| Verbalised LLM confidence tends to be overconfident | Well Supported (for models evaluated in 2023) | Xiong et al. (2023) | Moderate |
| Models can estimate whether their own answers are correct | Preliminary (format-dependent) | Kadavath et al. (2022) | Moderate |
| Semantic entropy detects a class of confabulations | Preliminary | Farquhar et al. (2024) | Moderate |
| Average accuracy hides trial-to-trial inconsistency | Well Supported | Yao et al. (2024) | Moderate |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.