---
id: SAFE-INTERP-001
title: "Interpretability and Mechanistic Interpretability"
alternative_names:
  - "Explainable AI (XAI)"
  - "Neural Network Interpretability"
  - "Mechanistic Interpretability (MI)"
  - "Reverse-Engineering Neural Networks"
category: "Safety & Governance → Technical Safety → Interpretability"
volume: VIII
chapter: 34
entry_type: "Technical Article"
level: 4
status: "Draft"
last_reviewed: 2026-10-18
version: "0.2.0"
prerequisites:
  - "TECH-NN-001"
  - "TECH-ARCH-001"
related_entries:
  - "SAFE-ALIGN-001"
  - "SAFE-OVERSIGHT-001"
  - "SAFE-EVAL-001"
  - "SAFE-ROBUST-001"
  - "TECH-LLM-001"
  - "SAFE-XRISK-001"
---

# Interpretability and Mechanistic Interpretability

## 1. Concise Definition
**Interpretability** is the study of how to explain, in terms humans can understand, why a machine-learning model produces the outputs it does. **Mechanistic interpretability** is the subfield that tries to reverse-engineer a trained neural network into its internal components (features, circuits and algorithms) and to verify those explanations with causal interventions on the network's weights and activations (Olah et al., 2020; Elhage et al., 2021; Bereska & Gavves, 2024).

## 2. Overview & Significance
Modern networks are trained by optimisation, not written by engineers, so their learned algorithms are not documented anywhere. Interpretability aims to close that gap. For safety, the motivation is that behavioural testing alone cannot show *why* a system behaves well. A model that passes every test might do so for the wrong internal reason, or might behave differently when it infers it is not being tested. Internal evidence is a possible complement to behavioural evaluation (see [SAFE-EVAL-001]) and to oversight methods that depend on a human or weaker model judging outputs (see [SAFE-OVERSIGHT-001]).

Two distinctions organise the field. The first separates **post-hoc explanation** of individual predictions (for example saliency maps) from **mechanistic** accounts of how the computation is implemented. The second separates the goal of *understanding* (a scientific aim) from the goal of *control* (steering or monitoring behaviour), which can succeed even without a complete mechanistic account; representation engineering is an example of the control-oriented approach (Zou et al., 2023).

The practical state of the art falls well short of explaining a frontier model end to end. Reviews and a 2025 multi-author agenda paper describe open conceptual and practical problems that must be solved before many of the promised safety benefits can be realised (Sharkey et al., 2025; Bereska & Gavves, 2024).

## 3. Historical Development
Interpretability predates deep learning: linear models, decision trees and rule lists were chosen partly because their structure is inspectable. With deep networks, early work concentrated on attributing a prediction to input features (saliency and gradient methods). Lipton (2016) argued that "interpretability" was used for several diverse and occasionally discordant aims, which complicates evaluation. Adebayo et al. (2018) showed that some popular saliency methods produced outputs independent of the trained model and of the data-generating process, so visual plausibility was not evidence of explanatory value. Rudin (2019) argued that for high-stakes decisions, inherently interpretable models should be preferred over post-hoc explanations of black boxes.

The mechanistic programme was articulated in the "Circuits" line of work on vision models (Olah et al., 2020), which proposed that networks are composed of *features* connected by *circuits*, and offered a speculative "universality" claim that analogous features recur across models. In 2021 Elhage et al. developed a mathematical framework for small attention-only transformers, decomposing attention heads into query-key and output-value circuits acting on a shared residual stream. Olsson et al. (2022) identified "induction heads" and presented evidence linking them to in-context learning. Wang et al. (2022) explained a natural-language behaviour (indirect object identification) in GPT-2 small. From 2022 onward, attention shifted to the problem of *superposition* (Elhage et al., 2022) and to sparse autoencoders as a tool for resolving it (Cunningham et al., 2023; Bricken et al., 2023; Templeton et al., 2024).

## 4. Technical / Conceptual Explanation

### 4.1 Features, circuits and the residual stream
A **feature** is a property of the input that the network represents internally; a **circuit** is a subgraph of the network's computation that implements a behaviour using such features. In a transformer, every layer reads from and writes to a shared *residual stream*, so a component's effect can be analysed as an additive contribution (Elhage et al., 2021). Explanations are tested by **activation patching** (also called causal intervention): replace a component's activation on one input with its activation on a contrasting input and measure the change in a chosen metric (Wang et al., 2022; Conmy et al., 2023).

### 4.2 Polysemanticity and superposition
Individual neurons often respond to several unrelated inputs, a property called **polysemanticity**. The *superposition hypothesis* proposes that networks represent more features than they have dimensions by assigning features to nearly orthogonal directions in activation space, tolerating some interference because each feature is sparse. Elhage et al. (2022) demonstrate this in toy models, including phase changes between regimes of representation, but the demonstration is in synthetic settings; how fully it describes large language models is not settled.

If superposition holds, the natural unit of analysis is a *direction*, not a neuron. Writing an activation vector $x \in \mathbb{R}^d$ as a sparse combination of $m \gg d$ feature directions,

$$x \approx \sum_{i=1}^{m} f_i(x)\, d_i, \qquad f_i(x) \ge 0,\ \text{most } f_i(x)=0,$$

where $d_i$ are dictionary directions and $f_i$ are feature activations.

### 4.3 Sparse autoencoders
A **sparse autoencoder (SAE)** learns such a dictionary without labels. It encodes $x$ into a wide, sparse code and reconstructs it, trained with an objective of the form

$$\mathcal{L}(x) = \lVert x - \hat{x}\rVert_2^2 + \lambda \lVert f(x) \rVert_1, \qquad \hat{x} = \sum_i f_i(x)\, d_i + b,$$

trading reconstruction error against sparsity of the feature activations. Cunningham et al. (2023) report that SAE features are more interpretable, by automated measures, than directions found by alternative methods, and that they localise causal effects on the indirect-object task more finely. Bricken et al. (2023) applied dictionary learning to a small one-layer transformer, and Templeton et al. (2024) applied it to a production model (Claude 3 Sonnet), both published as lab reports on transformer-circuits.pub rather than in peer-reviewed venues.

### 4.4 Circuit analysis and automation
Hand-tracing circuits is labour-intensive. Conmy et al. (2023) systematise the workflow (choose a behaviour, metric and dataset; patch to find involved components) and automate circuit identification with the ACDC algorithm. In their reproduction on a GPT-2 Small "greater-than" circuit, ACDC selected 68 of 32,000 edges, all of which earlier manual work had also found.

### 4.5 Top-down and editing approaches
Representation engineering takes population-level representations, rather than neurons or circuits, as the unit of analysis, and reports simple methods for reading and steering properties such as honesty (Zou et al., 2023). Locating and editing factual associations (Meng et al., 2022) uses causal tracing to identify mid-layer feed-forward modules mediating factual recall and edits them with a rank-one update. These results show that behaviour can be influenced by interventions on internals; they do not by themselves show that the localisation is the whole story.

## 5. Types, Components & Classifications
| Family | Question answered | Representative work | Main limitation |
| :--- | :--- | :--- | :--- |
| Attribution / saliency | Which inputs mattered for this output? | Adebayo et al. (2018) | Can be insensitive to the model; visual plausibility misleads |
| Inherently interpretable models | Can the model be inspected by design? | Rudin (2019) | Performance trade-offs are task-dependent and debated |
| Circuit analysis | Which components implement behaviour X? | Wang et al. (2022); Nanda et al. (2023) | Manual effort; mostly small models and narrow behaviours |
| Dictionary learning (SAEs) | What features does the model represent? | Cunningham et al. (2023); Templeton et al. (2024) | Reconstruction loss and feature splitting; completeness unclear |
| Representation engineering | Can we read or steer high-level properties? | Zou et al. (2023) | Validity of the property labels |
| Model editing | Where are facts stored, and can they be changed? | Meng et al. (2022) | Localisation is partial; side effects |

## 6. Applications & Concrete Examples
- **Fully reverse-engineered toy algorithm.** Nanda et al. (2023) recovered the algorithm a small transformer learned for modular addition (discrete Fourier transforms and trigonometric identities) and used it to define progress measures splitting "grokking" into memorisation, circuit formation and cleanup. The authors confirmed the algorithm by analysing activations and weights and by ablations in Fourier space, but on a small network trained on a narrow task.
- **Natural behaviour in a small language model.** Wang et al. (2022) explain indirect object identification in GPT-2 small with 26 attention heads in 7 classes, evaluated for faithfulness, completeness and minimality; they report that the criteria support the explanation but reveal remaining gaps.
- **In-context learning.** Olsson et al. (2022) present causal evidence for induction heads in small attention-only models and only correlational evidence for larger models with MLP layers, and describe their broader claim as a hypothesis.
- **Production-scale feature extraction.** Templeton et al. (2024) is the best-known demonstration that dictionary learning can be run on a deployed frontier model (Claude 3 Sonnet); it is a developer report and has not, as far as this draft verified, been independently replicated at that scale.
- **Pre-deployment white-box audit.** The Claude Sonnet 4.5 system card (Anthropic, September 2025; Ref 18) reports a pre-release "white-box" interpretability investigation, which the developer describes as the first such investigation of a frontier language model to its knowledge. It combined model diffing with a targeted study of evaluation awareness, including steering to inhibit internal evaluation-awareness representations, and concluded that the model represented evaluation awareness internally and that this grew over training. This is a developer self-report on one model and is not independently verified.

Apart from the Claude Sonnet 4.5 audit above, deployed uses of interpretability for safety assurance (as opposed to research demonstrations) were not found in the sources reviewed for this draft. In that system card the white-box work is presented as informing the alignment assessment and as a recommended standard component of pre-release evaluations; it does not state that a release decision depended on it. [VERIFICATION NEEDED: whether other developers' system cards or safety cases (including later Anthropic cards) rely on interpretability evidence for a deployment decision; only the Sonnet 4.5 card was read for this audit.]

## 7. Advantages, Limitations & Unresolved Questions
**Potential advantages.** Internal evidence could detect failures that outputs do not show, support targeted editing, and complement behavioural red-teaming.

**Limitations and open problems.**
- *Faithfulness of explanations.* An explanation can be plausible without being causally correct (Adebayo et al., 2018). Mechanistic claims therefore require interventions, and even then depend on the chosen metric and dataset (Conmy et al., 2023).
- *Scale and completeness.* Full circuit-level accounts exist for small models and narrow behaviours (Nanda et al., 2023; Wang et al., 2022). Whether comparable accounts are feasible for frontier models is open (Sharkey et al., 2025).
- *Unit of analysis.* Whether SAE features are the "true" features of a model, or one useful decomposition among several, is unresolved; reconstruction error means some model computation is not captured. Two preprints report negative downstream results: Kantamneni et al. (2025; Ref 19) found that SAE probes did not consistently beat baseline-only methods on probing tasks in four regimes, and Korznikov et al. (2026; Ref 20) report that randomised-feature baselines matched trained SAEs on several metrics. Both are preprints (Tier 3, **Preliminary**) and their authors do not rule out SAE utility on other tasks. [VERIFICATION NEEDED: peer-reviewed status of Refs 19-20, and any positive downstream comparisons after Feb 2026.]
- *Universality.* The claim that similar features and circuits recur across models (Olah et al., 2020) is a hypothesis with partial support.
- *Alternative view.* Rudin (2019) and others argue that post-hoc explanation is the wrong strategy for high-stakes uses; this critique targets explanation of opaque models, not mechanistic work as such, but it affects how far interpretability outputs should be trusted as assurance.
- *Adversarial pressure.* Whether interpretability remains reliable if a model is trained against the tools, or acts to conceal its reasoning, is not established.

## 8. Current State of Knowledge
As of 2026-10-18 (based on the sources below):
- Causal-intervention methods (activation patching) are widely used research tools; **Well Supported** as methods for testing hypotheses about model internals in small and mid-sized models. Dictionary learning (SAEs) is widely used, but its value relative to simple baselines is **Contested** (Refs 19-20, preprints).
- Complete, verified mechanistic explanations are limited to small models or narrow behaviours (**Well Supported** for the specific modular-addition network, a single-group result that this draft did not check for independent replication; **Preliminary** for natural language behaviours).
- Claims that interpretability will provide reliable safety assurance for frontier or superhuman systems are **Hypothetical**; the field's own agenda papers frame this as a set of open problems (Sharkey et al., 2025).
- Rapidly changing results (feature counts, dictionary sizes, tooling) are not recorded here; see the Living Reference collections (`MODEL-001`, `FACT-001`) when available.

## 9. Future Trajectory & Strategic Outlook
The following is an author judgement, not a survey result.

- **Near-term (0-3 years).** Continued work on automated circuit discovery, SAE variants and evaluation of whether features support downstream tasks. Observed trend: growing use of interpretability tools as red-team aids.
- **Medium-term (3-10 years).** Whether interpretability becomes a component of formal safety cases depends on validation against adversarially trained models.
- **Long-term (10+ years).** Complete understanding of superhuman systems is a **Hypothetical** goal; no evidence yet shows it is attainable.

| Field | Content |
| :--- | :--- |
| **Forecast ID** | `FCST-INTERP-001` |
| **Forecast Statement** | A frontier AI developer will publish a deployment safety case in which interpretability-derived evidence is described as a necessary supporting element for a deployment decision. |
| **Operational Definition** | A public system card, safety case or framework report from a frontier developer that explicitly states a deployment or release conclusion depends in part on interpretability evidence (feature, circuit or probe-based analysis), not only behavioural testing. Baseline at 2026-10-18: the Claude Sonnet 4.5 system card (Sept 2025) includes a white-box audit but does not state that the release decision depended on it, so it is not counted as resolving this forecast. |
| **Time Horizon** | By 2029-12-31 (medium-term) |
| **Assumptions** | Continued progress in SAE or circuit methods; developers publish safety cases; interpretability results can be validated against known implanted behaviours. |
| **Confidence Level** | Low-to-medium (30-50%); author's judgement, not derived from a survey |
| **Alternative Scenarios** | Interpretability remains a research aid; safety cases rely on behavioural evaluations and control measures; or tools are found unreliable under adversarial training. |
| **Review Schedule** | Annual review each October; final evaluation January 2030 |
| **Status** | `Active` |

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-NN-001] Artificial Neural Networks](../vol03_technical/TECH-NN-001_artificial_neural_networks.md): The objects being interpreted.
  - [[TECH-ARCH-001] Transformer Architecture](../vol03_technical/TECH-ARCH-001_transformer_architecture.md): Attention heads, MLPs and the residual stream.
- **Related Methods & Technologies**:
  - [[SAFE-ALIGN-001] The AI Alignment Problem](./SAFE-ALIGN-001_ai_alignment.md): Interpretability is proposed as a tool for detecting inner misalignment.
  - [[SAFE-OVERSIGHT-001] Scalable Oversight and Corrigibility](./SAFE-OVERSIGHT-001_scalable_oversight_and_corrigibility.md): Complementary approach based on supervising outputs.
  - [[SAFE-EVAL-001] Dangerous Capability and Alignment Evaluations](./SAFE-EVAL-001_dangerous_capability_and_alignment_evaluations.md): Behavioural counterpart to internal inspection.
  - [[SAFE-ROBUST-001] Technical Safety and Robustness](./SAFE-ROBUST-001_technical_safety_and_robustness.md): Adversarial examples and failure analysis.
  - [[TECH-LLM-001] Large Language Models](../vol03_technical/TECH-LLM-001_large_language_models.md): The principal systems studied.
- **Advanced Implications & Future Trajectories**:
  - [[SAFE-XRISK-001] Existential Risk from AI](./SAFE-XRISK-001_existential_risk.md): Context for why internal assurance is sought.

## 11. References & Foundational Literature
1. Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M., & Carter, S. (2020). *Zoom In: An Introduction to Circuits*. Distill, 5(3). https://doi.org/10.23915/distill.00024.001. [Introduces the features-and-circuits framing and the universality claim for vision models.]
2. Elhage, N., Nanda, N., Olsson, C., et al. (2021). *A Mathematical Framework for Transformer Circuits*. Transformer Circuits Thread, 22 December 2021. https://transformer-circuits.pub/2021/framework/index.html. [Residual-stream view and head decomposition for small attention-only transformers; 25 listed authors.]
3. Elhage, N., Hume, T., Olsson, C., et al. (2022). *Toy Models of Superposition*. arXiv:2209.10652. [Toy-model account of polysemanticity as superposition, with a phase change; 16 authors.]
4. Olsson, C., Elhage, N., Nanda, N., et al. (2022). *In-context Learning and Induction Heads*. arXiv:2209.11895. [Evidence linking induction heads to in-context learning; causal in small models, correlational in larger ones; 26 authors.]
5. Wang, K., Variengien, A., Conmy, A., Shlegeris, B., & Steinhardt, J. (2022). *Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small*. arXiv:2211.00593. [26 heads in 7 classes explain one behaviour; evaluated for faithfulness, completeness, minimality.]
6. Nanda, N., Chan, L., Lieberum, T., Smith, J., & Steinhardt, J. (2023). *Progress measures for grokking via mechanistic interpretability*. arXiv:2301.05217. [Full reverse-engineering of a modular-addition transformer and progress measures.]
7. Meng, K., Bau, D., Andonian, A., & Belinkov, Y. (2022). *Locating and Editing Factual Associations in GPT*. arXiv:2202.05262. [Causal localisation of factual recall to mid-layer feed-forward modules; ROME editing.]
8. Conmy, A., Mavor-Parker, A. N., Lynch, A., Heimersheim, S., & Garriga-Alonso, A. (2023). *Towards Automated Circuit Discovery for Mechanistic Interpretability*. arXiv:2304.14997. [Systematises the interpretability workflow and automates circuit discovery (ACDC).]
9. Cunningham, H., Ewart, A., Riggs, L., Huben, R., & Sharkey, L. (2023). *Sparse Autoencoders Find Highly Interpretable Features in Language Models*. arXiv:2309.08600. [SAE features more interpretable by automated measures than alternatives.]
10. Bricken, T., Templeton, A., Batson, J., et al. (2023). *Towards Monosemanticity: Decomposing Language Models With Dictionary Learning*. Transformer Circuits Thread, 4 October 2023. https://transformer-circuits.pub/2023/monosemantic-features/index.html. [Dictionary learning on a small transformer; lab report.]
11. Templeton, A., Conerly, T., Marcus, J., et al. (2024). *Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet*. Transformer Circuits Thread, 21 May 2024. https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html. [Dictionary learning applied to a production model; developer report.]
12. Zou, A., Phan, L., Chen, S., et al. (2023). *Representation Engineering: A Top-Down Approach to AI Transparency*. arXiv:2310.01405. [Population-level representations for monitoring and steering; 21 authors.]
13. Adebayo, J., Gilmer, J., Muelly, M., Goodfellow, I., Hardt, M., & Kim, B. (2018). *Sanity Checks for Saliency Maps*. arXiv:1810.03292. [Some saliency methods are independent of the model and data.]
14. Rudin, C. (2019). *Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead*. Nature Machine Intelligence, 1(5), 206-215. https://doi.org/10.1038/s42256-019-0048-x. [Argument for inherently interpretable models in high-stakes use.]
15. Lipton, Z. C. (2016). *The Mythos of Model Interpretability*. arXiv:1606.03490. [Conceptual analysis of competing meanings of interpretability.]
16. Sharkey, L., Chughtai, B., Batson, J., et al. (2025). *Open Problems in Mechanistic Interpretability*. arXiv:2501.16496. [Forward-looking review of open conceptual, practical and socio-technical problems; 29 authors.]
17. Bereska, L., & Gavves, E. (2024). *Mechanistic Interpretability for AI Safety -- A Review*. arXiv:2404.14082. [Review of mechanistic interpretability and its relevance to safety.]

18. Anthropic. (2025). *System Card: Claude Sonnet 4.5*. September 2025 (changelog entries dated 10 Oct and 3 Dec 2025). https://assets.anthropic.com/m/12f214efcc2f457a/original/claude-sonnet-4-5-system-card.pdf. [Section 7.6 reports a pre-release white-box interpretability audit with model diffing and an evaluation-awareness investigation; developer report.]
19. Kantamneni, S., Engels, J., Rajamanoharan, S., Tegmark, M., & Nanda, N. (2025). *Are Sparse Autoencoders Useful? A Case Study in Sparse Probing*. arXiv:2502.16681. [SAE probes did not consistently outperform baselines on probing in four regimes.]
20. Korznikov, A., Galichin, A., Dontsov, A., Rogov, O., Oseledets, I., & Tutubalina, E. (2026). *Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?* arXiv:2602.14111. [Reports that random baselines match trained SAEs on several metrics.]

Note: arXiv entries are cited as preprints. Among the cited preprints, arXiv metadata states acceptance at NeurIPS 2022 (Meng et al.), NeurIPS 2023 (Conmy et al.) and TMLR (Bereska & Gavves); venues for the others were not checked.

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Activation patching can test causal roles of components | Well Supported | Wang et al. (2022); Conmy et al. (2023) | Moderate-High |
| Modular-addition network fully reverse-engineered | Well Supported (specific case) | Nanda et al. (2023) | High |
| Some saliency methods are insensitive to the model | Well Supported | Adebayo et al. (2018) | High |
| Superposition explains polysemanticity in toy models | Well Supported (toy setting) | Elhage et al. (2022) | Moderate |
| Superposition is the main cause of polysemanticity in LLMs | Contested | Cunningham et al. (2023); Sharkey et al. (2025) | Moderate-Low |
| SAE features at production scale | Preliminary | Templeton et al. (2024) | Moderate; developer report |
| SAEs improve downstream tasks over simple baselines | Contested | Kantamneni et al. (2025); Korznikov et al. (2026) (Refs 19-20) | Low-Moderate; preprints |
| White-box interpretability used in a pre-deployment audit of a frontier model | Preliminary (developer report) | Anthropic (2025) (Ref 18) | Moderate; single developer, not independently verified |
| Induction heads underlie most in-context learning | Preliminary / Hypothetical | Olsson et al. (2022) | Authors call evidence preliminary and indirect for large models |
| Universality of features across models | Hypothetical | Olah et al. (2020) | Low-Moderate |
| Interpretability will give safety assurance for frontier systems | Hypothetical / Forecast | Sharkey et al. (2025); `FCST-INTERP-001` | Low-Moderate |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.
- **v0.2.0** (2026-10-18): Fact-check audit (see `docs/audits/SAFE-INTERP-001_audit.md`). All 17 original references confirmed to exist with stated authors, year and title; softened "fully verified" and "conflicting"; downgraded modular-addition case from Established to Well Supported; resolved one verification marker using the Claude Sonnet 4.5 system card; added Refs 18-20 on pre-deployment audit and negative SAE results; updated forecast baseline.