---
id: SAFE-ROBUST-001
title: "Technical Safety and Robustness"
alternative_names:
  - "ML Safety"
  - "Adversarial Robustness and Reliability"
  - "Robustness to Distribution Shift"
category: "Safety & Governance → Technical Safety → Robustness"
volume: VIII
chapter: 45
entry_type: "Standard Article"
level: 3
status: "Draft"
last_reviewed: 2026-10-18
version: "0.2.0"
prerequisites:
  - "TECH-ML-001"
related_entries:
  - "SAFE-ALIGN-001"
  - "SAFE-SEC-001"
  - "SAFE-EVAL-001"
  - "SAFE-INTERP-001"
  - "CAP-LIMITS-001"
  - "TECH-DL-001"
---

# Technical Safety and Robustness

## 1. Concise Definition
**Technical safety and robustness** is the body of research and engineering practice concerned with making machine-learning systems behave reliably under conditions that differ from, or are deliberately adversarial to, their training and test conditions. It covers **adversarial robustness** (resistance to worst-case input perturbations), **robustness to distribution shift**, **calibration and monitoring** (knowing when the system is likely wrong), and the broader programme of "ML safety" (Hendrycks et al., 2021; Amodei et al., 2016).

## 2. Overview & Significance
Standard machine-learning evaluation assumes test data are drawn from the same distribution as training data. Deployed systems violate this assumption through natural change (new hospitals, cameras, languages), adversaries, and rare events. Amodei et al. (2016) grouped practical accident risks into wrong objective functions (side effects, reward hacking), expensive objectives (scalable supervision) and undesirable learning-time behaviour (safe exploration, distributional shift). Hendrycks et al. (2021) proposed four research areas: withstanding hazards (**Robustness**), identifying hazards (**Monitoring**), reducing inherent model hazards (**Alignment**) and reducing systemic hazards (**Systemic Safety**).

This article covers the robustness and monitoring strands. Alignment is treated in [SAFE-ALIGN-001], deliberate attacks on deployed LLM systems in [SAFE-SEC-001], and tests of dangerous behaviours in [SAFE-EVAL-001].

## 3. Historical Development
Szegedy et al. (2013) reported that deep networks learn input-output mappings that are fairly discontinuous, so that an imperceptible perturbation found by maximising prediction error can cause misclassification, and that the same perturbation can transfer to a different network trained on a different data subset. Goodfellow et al. (2014) argued that the primary cause is the models' linear behaviour in high dimensions and introduced a fast method of generating adversarial examples. Carlini and Wagner (2017) reported that defensive distillation, which had been reported to cut the success rate of then-current attacks from 95% to 0.5%, did not significantly increase robustness against their three new attacks (which they report succeed with 100% probability on distilled and undistilled networks). Madry et al. (2017) studied robustness through robust optimisation and proposed training and attack methods built around the notion of security against a first-order adversary. Athalye et al. (2018) found that among non-certified white-box defences published at ICLR 2018, 7 of 9 relied on obfuscated gradients, and their new attacks circumvented 6 completely and 1 partially. Cohen et al. (2019) gave a tight certified robustness guarantee for randomized smoothing.

In parallel, researchers studied natural shift and uncertainty: Hendrycks and Gimpel (2016) proposed a maximum-softmax-probability baseline for detecting misclassified and out-of-distribution examples; Guo et al. (2017) showed that modern networks are poorly calibrated; Hendrycks and Dietterich (2019) built corruption and perturbation benchmarks; Koh et al. (2020) assembled WILDS; and D'Amour et al. (2020) identified underspecification as a source of deployment failure. For language models, Wei et al. (2023) and Zou et al. (2023) examined jailbreaks.

## 4. Technical / Conceptual Explanation

### 4.1 Adversarial examples
For a classifier $f_\theta$, loss $\ell$, input $x$ with label $y$, and perturbation budget $\epsilon$ in a norm ball, an adversary solves
$$\max_{\lVert\delta\rVert_p \le \epsilon} \ell\big(f_\theta(x+\delta),\, y\big).$$
The fast gradient sign method takes a single step $x' = x + \epsilon\,\mathrm{sign}(\nabla_x \ell(f_\theta(x),y))$ (Goodfellow et al., 2014); iterated variants use projected gradient steps (Madry et al., 2017).

### 4.2 Robust optimisation and adversarial training
Madry et al. (2017) formulate training as the saddle-point problem
$$\min_\theta\ \mathbb{E}_{(x,y)\sim\mathcal D}\Big[\max_{\lVert\delta\rVert\le\epsilon}\ell\big(f_\theta(x+\delta),y\big)\Big],$$
where the inner maximisation is the attack and the outer minimisation trains the model on the worst case found. The guarantee is relative to the threat model (norm, budget, attacker knowledge).

### 4.3 Certified robustness
Randomized smoothing turns a classifier that works well under Gaussian noise into a new classifier with a provable $\ell_2$ robustness radius (Cohen et al., 2019). Cohen et al. report an ImageNet classifier with 49% certified top-1 accuracy under $\ell_2$ perturbations of norm below 0.5; such figures are specific to that paper's models and date.

### 4.4 Distribution shift and underspecification
Distribution shift means the test distribution differs from the training distribution. Koh et al. (2020) show on 10 datasets (for example shifts across hospitals and cameras) that standard training yields substantially lower out-of-distribution than in-distribution performance. D'Amour et al. (2020) show that an *underspecified* pipeline can return many predictors with equal held-out performance that behave very differently in deployment, a failure mode distinct from structural train/test mismatch.

### 4.5 Calibration and failure detection
A model is *calibrated* if its stated confidence matches the empirical frequency of being correct. Guo et al. (2017) find modern networks poorly calibrated and that temperature scaling is surprisingly effective on most datasets they test. Hendrycks and Gimpel (2016) show that maximum softmax probability is a useful baseline for detecting errors and out-of-distribution inputs, and that it can sometimes be surpassed. For language-model reliability see [CAP-LIMITS-001].

## 5. Types, Components & Classifications
| Category | Threat or condition | Typical defence | Main caveat |
| :--- | :--- | :--- | :--- |
| Adversarial (white-box) | Worst-case perturbation with model access | Adversarial training; certified smoothing | Guarantees are threat-model-specific; evaluation pitfalls (Athalye et al., 2018) |
| Common corruptions | Noise, blur, weather | Data augmentation; benchmarks (Hendrycks & Dietterich, 2019) | Corruption benchmarks evaluate common corruptions, not worst-case adversarial perturbations (Hendrycks & Dietterich, 2019) |
| Distribution shift | New domains, time, location | Domain generalisation methods | Standard training degrades; methods give limited gains (Koh et al., 2020) |
| Underspecification | Many equally good predictors | Stress tests, domain-specific constraints | Not detectable from held-out accuracy (D'Amour et al., 2020) |
| Miscalibration / OOD inputs | Overconfident wrong predictions | Temperature scaling; OOD detectors | Baselines are imperfect (Guo et al., 2017; Hendrycks & Gimpel, 2016) |
| LLM jailbreaks | Prompts that evade safety training | Red-teaming, refusal training | Persistent vulnerabilities (Wei et al., 2023; Zou et al., 2023) |

## 6. Applications & Concrete Examples
- **Image classification.** Adversarial perturbations, adversarial training and certified defences were developed mainly on image benchmarks (Szegedy et al., 2013; Madry et al., 2017; Cohen et al., 2019).
- **Safety-critical deployments.** Distribution-shift benchmarks reflect applications such as tumour identification across hospitals and wildlife monitoring across camera traps (Koh et al., 2020).
- **Language-model safeguards.** Wei et al. (2023) hypothesise two failure modes of safety training, *competing objectives* and *mismatched generalisation*, and report that attacks built on them succeeded on every prompt in a collection of unsafe requests from the models' red-teaming evaluation sets for GPT-4 and Claude v1.3. Zou et al. (2023) report automatically generated adversarial suffixes that transfer to other models, including black-box models. Both findings concern the specific model versions and defences in place at publication; later safeguards are not covered here.

## 7. Advantages, Limitations & Unresolved Questions
- **Evaluation difficulty.** Carlini and Wagner (2017) broke defensive distillation, and Athalye et al. (2018) found that 7 of 9 non-certified white-box defences published at ICLR 2018 relied on obfuscated gradients (6 circumvented completely, 1 partially), so claims of robustness require adaptive evaluation.
- **Threat-model dependence.** Robustness to one perturbation set should not be assumed to transfer to others; Hendrycks and Dietterich (2019) benchmark common corruptions and perturbations explicitly as distinct from worst-case adversarial perturbations.
- **Cost.** Adversarial training and certification typically trade accuracy or compute for robustness. [VERIFICATION NEEDED: quantitative current trade-off figures for large-scale models; not reviewed.]
- **Generalisation to LLMs.** Whether results from vision-style adversarial robustness carry over to language-model safety is an open research question. Wei et al. (2023) and Zou et al. (2023) report jailbreak vulnerabilities in the models they tested despite safety training. Later defence work reports progress: Sharma et al. (2025) report that, in over 3,000 estimated hours of red teaming, no red teamer found a universal jailbreak against an early classifier-guarded LLM at a similar level of detail to an unguarded model across most target queries, with a 0.38% absolute increase in production-traffic refusals and 23.7% inference overhead for their system (single-lab preprint, Preliminary; not independently replicated; figures apply to that system as of 2025).
- **Unspecified failure modes.** Underspecification means held-out accuracy can hide deployment risk (D'Amour et al., 2020).

## 8. Current State of Knowledge
As of 2026-10-18:
- That deep networks are vulnerable to small adversarial perturbations is **Established** (replicated across many studies beginning with Szegedy et al., 2013).
- That adversarial training improves robustness within a threat model and that many non-certified defences fail adaptive attacks is **Well Supported** (Madry et al., 2017; Athalye et al., 2018).
- That standard training suffers on real-world distribution shifts is **Well Supported** (Koh et al., 2020).
- That aligned LLMs remain jailbreakable by automated and hand-crafted methods is **Well Supported** for the models and years studied (Wei et al., 2023; Zou et al., 2023); classifier-based defences have since been reported to resist universal jailbreaks in extended red teaming, a **Preliminary** result (Sharma et al., 2025). The current state of the art tracked in evaluation work ([SAFE-EVAL-001]) and the Living Reference collections. Status of current frontier-model jailbreak robustness was not established in this audit.
- No technique is known to provide general robustness guarantees for large foundation models: this statement is the author's reading of the cited literature and was not confirmed against a recent survey. [VERIFICATION NEEDED: confirm against recent surveys; none located in this audit.]

## 9. Future Trajectory & Strategic Outlook
This section contains no registered forecasts. Observed trends and open questions:
- **Near-term (0-3 years).** Robustness research continues to move toward LLMs and agents, where threats include jailbreaks and tool-mediated attacks (see [SAFE-SEC-001]).
- **Medium-term (3-10 years).** Whether certified or monitoring-based methods scale to foundation models is open.
- **Long-term (10+ years).** Robustness of increasingly capable, autonomous systems is entangled with alignment and oversight (see [SAFE-ALIGN-001]).

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-ML-001] Machine Learning](../vol03_technical/TECH-ML-001_machine_learning.md): Training and generalisation concepts.
  - [[TECH-DL-001] Deep Learning](../vol03_technical/TECH-DL-001_deep_learning.md): The model class most studied for robustness.
- **Related Methods & Technologies**:
  - [[SAFE-SEC-001] AI Security](./SAFE-SEC-001_ai_security_prompt_injection_poisoning_extraction.md): Deliberate attacks on deployed systems.
  - [[SAFE-EVAL-001] Dangerous Capability and Alignment Evaluations](./SAFE-EVAL-001_dangerous_capability_and_alignment_evaluations.md): Testing and red-teaming.
  - [[SAFE-INTERP-001] Interpretability and Mechanistic Interpretability](./SAFE-INTERP-001_interpretability_and_mechanistic_interpretability.md): Understanding failure causes.
  - [[CAP-LIMITS-001] Hallucination, Reliability and Calibration](../vol04_capabilities/CAP-LIMITS-001_hallucination_reliability_and_calibration.md): Reliability of language models.
- **Advanced Implications & Future Trajectories**:
  - [[SAFE-ALIGN-001] The AI Alignment Problem](./SAFE-ALIGN-001_ai_alignment.md): Where robustness meets objective specification.

## 11. References & Foundational Literature
1. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). *Concrete Problems in AI Safety*. arXiv:1606.06565. [Five practical accident-risk problems including distributional shift.]
2. Hendrycks, D., Carlini, N., Schulman, J., & Steinhardt, J. (2021). *Unsolved Problems in ML Safety*. arXiv:2109.13916. [Roadmap: robustness, monitoring, alignment, systemic safety.]
3. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. (2013). *Intriguing properties of neural networks*. arXiv:1312.6199. [First report of imperceptible adversarial perturbations and their transfer.]
4. Goodfellow, I. J., Shlens, J., & Szegedy, C. (2014). *Explaining and Harnessing Adversarial Examples*. arXiv:1412.6572. [Linearity explanation; fast gradient sign method; adversarial training.]
5. Carlini, N., & Wagner, D. (2017). *Towards Evaluating the Robustness of Neural Networks*. 2017 IEEE Symposium on Security and Privacy (SP), 39-57. https://doi.org/10.1109/SP.2017.49; arXiv:1608.04644 (first posted 2016). [Stronger attacks defeat defensive distillation.]
6. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., & Vladu, A. (2017). *Towards Deep Learning Models Resistant to Adversarial Attacks*. arXiv:1706.06083 (ICLR 2018). [Min-max formulation and first-order adversary.]
7. Athalye, A., Carlini, N., & Wagner, D. (2018). *Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples*. arXiv:1802.00420 (ICML 2018). [7 of 9 defences relied on obfuscated gradients.]
8. Cohen, J. M., Rosenfeld, E., & Kolter, J. Z. (2019). *Certified Adversarial Robustness via Randomized Smoothing*. arXiv:1902.02918 (ICML 2019). [Tight $\ell_2$ certified radius; ImageNet certified accuracy figure.]
9. Hendrycks, D., & Dietterich, T. (2019). *Benchmarking Neural Network Robustness to Common Corruptions and Perturbations*. arXiv:1903.12261 (ICLR 2019). [ImageNet-C/ImageNet-P benchmarks.]
10. Hendrycks, D., & Gimpel, K. (2016). *A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks*. arXiv:1610.02136 (ICLR 2017). [Maximum softmax probability baseline.]
11. Guo, C., Pleiss, G., Sun, Y., & Weinberger, K. Q. (2017). *On Calibration of Modern Neural Networks*. arXiv:1706.04599 (ICML 2017). [Modern networks are poorly calibrated; temperature scaling.]
12. Koh, P. W., Sagawa, S., Marklund, H., et al. (2020). *WILDS: A Benchmark of in-the-Wild Distribution Shifts*. arXiv:2012.07421. [10 datasets of natural distribution shift; 23 authors.]
13. D'Amour, A., Heller, K., Moldovan, D., et al. (2020). *Underspecification Presents Challenges for Credibility in Modern Machine Learning*. arXiv:2011.03395. [Underspecified pipelines; 40 authors.]
14. Wei, A., Haghtalab, N., & Steinhardt, J. (2023). *Jailbroken: How Does LLM Safety Training Fail?* arXiv:2307.02483. [Competing objectives and mismatched generalisation.]
15. Zou, A., Wang, Z., Carlini, N., Nasr, M., Kolter, J. Z., & Fredrikson, M. (2023). *Universal and Transferable Adversarial Attacks on Aligned Language Models*. arXiv:2307.15043. [Automated transferable adversarial suffixes.]
16. Sharma, M., Tong, M., Mu, J., Wei, J., et al. (2025). *Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming*. arXiv:2501.18837 (43 authors). [Single-lab preprint reporting a classifier-guarded LLM resisting universal jailbreaks in extended red teaming.]

Note: arXiv entries are cited as preprints. Venues in parentheses and the Carlini and Wagner (2017) DOI were checked against the arXiv record comments and Crossref on 2026-10-18; final venues for Refs 1-4, 12-15 (e.g. Szegedy, Goodfellow, WILDS, D'Amour, Wei, Zou) were not confirmed. [VERIFICATION NEEDED: publication venues for Refs 3, 4, 12, 13, 14, 15.]

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Deep networks have adversarial examples | Established | Szegedy et al. (2013); Goodfellow et al. (2014) | High |
| Adversarial training is effective within a threat model | Well Supported | Madry et al. (2017) | High |
| Many non-certified defences fail adaptive attacks | Well Supported | Carlini & Wagner (2017); Athalye et al. (2018) | High |
| Certified $\ell_2$ robustness via smoothing | Established (theorem); Well Supported (empirics) | Cohen et al. (2019) | High within threat model |
| Natural distribution shift reduces accuracy | Well Supported | Koh et al. (2020) | High |
| Underspecification causes deployment divergence | Well Supported | D'Amour et al. (2020) | Moderate-High |
| Modern networks are miscalibrated | Well Supported | Guo et al. (2017) | Moderate-High |
| Aligned LLMs remain jailbreakable | Well Supported (studied models) | Wei et al. (2023); Zou et al. (2023) | Moderate; version-dependent |
| Classifier-guarded LLM resisted universal jailbreaks in red teaming | Preliminary (single-lab preprint) | Sharma et al. (2025) (Ref 16) | Low-Moderate; system- and date-specific |
| General robustness guarantees for foundation models are unavailable | Contested / author reading | Refs 1-16 | Low-Moderate |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.
- **v0.2.0** (2026-10-18): Fact-check audit (docs/audits/SAFE-ROBUST-001_audit.md). All 15 references confirmed against the arXiv API; Carlini and Wagner (2017) venue/DOI confirmed via Crossref; wording tightened where it overreached the sources; Ref 16 added (appended); venues added where confirmed.