Technical Safety and Robustness

Technical safety and robustness is the body of research and engineering practice concerned with making machine-learning systems behave reliably under conditions that differ from, or are deliberately adversarial to, their training and test conditions. It covers adversarial robustness (resistance to worst-case input perturbations), robustness to distribution shift, calibration and monitoring (knowing when the system is likely wrong), and the broader programme of "ML safety" (Hendrycks et al., 2021; Amodei et al., 2016).

Standard machine-learning evaluation assumes test data are drawn from the same distribution as training data. Deployed systems violate this assumption through natural change (new hospitals, cameras, languages), adversaries, and rare events. Amodei et al. (2016) grouped practical accident risks into wrong objective functions (side effects, reward hacking), expensive objectives (scalable supervision) and undesirable learning-time behaviour (safe exploration, distributional shift). Hendrycks et al. (2021) proposed four research areas: withstanding hazards (Robustness), identifying hazards (Monitoring), reducing inherent model hazards (Alignment) and reducing systemic hazards (Systemic Safety).

This article covers the robustness and monitoring strands. Alignment is treated in [SAFE-ALIGN-001], deliberate attacks on deployed LLM systems in [SAFE-SEC-001], and tests of dangerous behaviours in [SAFE-EVAL-001].

Szegedy et al. (2013) reported that deep networks learn input-output mappings that are fairly discontinuous, so that an imperceptible perturbation found by maximising prediction error can cause misclassification, and that the same perturbation can transfer to a different network trained on a different data subset. Goodfellow et al. (2014) argued that the primary cause is the models' linear behaviour in high dimensions and introduced a fast method of generating adversarial examples. Carlini and Wagner (2017) reported that defensive distillation, which had been reported to cut the success rate of then-current attacks from 95% to 0.5%, did not significantly increase robustness against their three new attacks (which they report succeed with 100% probability on distilled and undistilled networks). Madry et al. (2017) studied robustness through robust optimisation and proposed training and attack methods built around the notion of security against a first-order adversary. Athalye et al. (2018) found that among non-certified white-box defences published at ICLR 2018, 7 of 9 relied on obfuscated gradients, and their new attacks circumvented 6 completely and 1 partially. Cohen et al. (2019) gave a tight certified robustness guarantee for randomized smoothing.

In parallel, researchers studied natural shift and uncertainty: Hendrycks and Gimpel (2016) proposed a maximum-softmax-probability baseline for detecting misclassified and out-of-distribution examples; Guo et al. (2017) showed that modern networks are poorly calibrated; Hendrycks and Dietterich (2019) built corruption and perturbation benchmarks; Koh et al. (2020) assembled WILDS; and D'Amour et al. (2020) identified underspecification as a source of deployment failure. For language models, Wei et al. (2023) and Zou et al. (2023) examined jailbreaks.

#4.1 Adversarial examples

For a classifier $f_\theta$, loss $\ell$, input $x$ with label $y$, and perturbation budget $\epsilon$ in a norm ball, an adversary solves

The fast gradient sign method takes a single step $x' = x + \epsilon\,\mathrm{sign}(\nabla_x \ell(f_\theta(x),y))$ (Goodfellow et al., 2014); iterated variants use projected gradient steps (Madry et al., 2017).

#4.2 Robust optimisation and adversarial training

Madry et al. (2017) formulate training as the saddle-point problem

where the inner maximisation is the attack and the outer minimisation trains the model on the worst case found. The guarantee is relative to the threat model (norm, budget, attacker knowledge).

#4.3 Certified robustness

Randomized smoothing turns a classifier that works well under Gaussian noise into a new classifier with a provable $\ell_2$ robustness radius (Cohen et al., 2019). Cohen et al. report an ImageNet classifier with 49% certified top-1 accuracy under $\ell_2$ perturbations of norm below 0.5; such figures are specific to that paper's models and date.

#4.4 Distribution shift and underspecification

Distribution shift means the test distribution differs from the training distribution. Koh et al. (2020) show on 10 datasets (for example shifts across hospitals and cameras) that standard training yields substantially lower out-of-distribution than in-distribution performance. D'Amour et al. (2020) show that an underspecified pipeline can return many predictors with equal held-out performance that behave very differently in deployment, a failure mode distinct from structural train/test mismatch.

#4.5 Calibration and failure detection

A model is calibrated if its stated confidence matches the empirical frequency of being correct. Guo et al. (2017) find modern networks poorly calibrated and that temperature scaling is surprisingly effective on most datasets they test. Hendrycks and Gimpel (2016) show that maximum softmax probability is a useful baseline for detecting errors and out-of-distribution inputs, and that it can sometimes be surpassed. For language-model reliability see [CAP-LIMITS-001].

• Image classification. Adversarial perturbations, adversarial training and certified defences were developed mainly on image benchmarks (Szegedy et al., 2013; Madry et al., 2017; Cohen et al., 2019).
• Safety-critical deployments. Distribution-shift benchmarks reflect applications such as tumour identification across hospitals and wildlife monitoring across camera traps (Koh et al., 2020).
• Language-model safeguards. Wei et al. (2023) hypothesise two failure modes of safety training, competing objectives and mismatched generalisation, and report that attacks built on them succeeded on every prompt in a collection of unsafe requests from the models' red-teaming evaluation sets for GPT-4 and Claude v1.3. Zou et al. (2023) report automatically generated adversarial suffixes that transfer to other models, including black-box models. Both findings concern the specific model versions and defences in place at publication; later safeguards are not covered here.

• Evaluation difficulty. Carlini and Wagner (2017) broke defensive distillation, and Athalye et al. (2018) found that 7 of 9 non-certified white-box defences published at ICLR 2018 relied on obfuscated gradients (6 circumvented completely, 1 partially), so claims of robustness require adaptive evaluation.
• Threat-model dependence. Robustness to one perturbation set should not be assumed to transfer to others; Hendrycks and Dietterich (2019) benchmark common corruptions and perturbations explicitly as distinct from worst-case adversarial perturbations.
• Generalisation to LLMs. Whether results from vision-style adversarial robustness carry over to language-model safety is an open research question. Wei et al. (2023) and Zou et al. (2023) report jailbreak vulnerabilities in the models they tested despite safety training. Later defence work reports progress: Sharma et al. (2025) report that, in over 3,000 estimated hours of red teaming, no red teamer found a universal jailbreak against an early classifier-guarded LLM at a similar level of detail to an unguarded model across most target queries, with a 0.38% absolute increase in production-traffic refusals and 23.7% inference overhead for their system (single-lab preprint, Preliminary; not independently replicated; figures apply to that system as of 2025).
• Unspecified failure modes. Underspecification means held-out accuracy can hide deployment risk (D'Amour et al., 2020).

As of 2026-10-18:

• That deep networks are vulnerable to small adversarial perturbations is Established (replicated across many studies beginning with Szegedy et al., 2013).
• That adversarial training improves robustness within a threat model and that many non-certified defences fail adaptive attacks is Well Supported (Madry et al., 2017; Athalye et al., 2018).
• That standard training suffers on real-world distribution shifts is Well Supported (Koh et al., 2020).
• That aligned LLMs remain jailbreakable by automated and hand-crafted methods is Well Supported for the models and years studied (Wei et al., 2023; Zou et al., 2023); classifier-based defences have since been reported to resist universal jailbreaks in extended red teaming, a Preliminary result (Sharma et al., 2025). The current state of the art tracked in evaluation work ([SAFE-EVAL-001]) and the Living Reference collections. Status of current frontier-model jailbreak robustness was not established in this audit.
• The cited sources do not report a technique with general robustness guarantees for large foundation models; the certification work cited (Cohen et al., 2019) concerns specified perturbation radii in image classification.

This section contains no registered forecasts. Observed trends and open questions:

• Near-term (0-3 years). Robustness research continues to move toward LLMs and agents, where threats include jailbreaks and tool-mediated attacks (see [SAFE-SEC-001]).
• Medium-term (3-10 years). Whether certified or monitoring-based methods scale to foundation models is open.
• Long-term (10+ years). Robustness of increasingly capable, autonomous systems is entangled with alignment and oversight (see [SAFE-ALIGN-001]).

• Foundational Concepts:
  • TECH-ML-001 Machine Learning: Training and generalisation concepts.
  • TECH-DL-001 Deep Learning: The model class most studied for robustness.
• Related Methods & Technologies:
  • SAFE-SEC-001 AI Security: Deliberate attacks on deployed systems.
  • SAFE-EVAL-001 Dangerous Capability and Alignment Evaluations: Testing and red-teaming.
  • SAFE-INTERP-001 Interpretability and Mechanistic Interpretability: Understanding failure causes.
  • CAP-LIMITS-001 Hallucination, Reliability and Calibration: Reliability of language models.
• Advanced Implications & Future Trajectories:
  • SAFE-ALIGN-001 The AI Alignment Problem: Where robustness meets objective specification.