Dangerous Capability and Alignment Evaluations

Dangerous capability evaluations test whether an AI model has abilities that could enable severe harm, such as offensive cyber capabilities, strong manipulation, help in assembling weapons (including biological weapons) or self-proliferation (Shevlane et al., 2023). Alignment evaluations test a model's propensity to apply its capabilities for harm (Shevlane et al., 2023); behaviours that Shevlane et al. list for such evaluations include pursuing long-term real-world goals different from those supplied by the developer or user, power-seeking and resisting shutdown.

Shevlane et al. (2023) argue that evaluation of both kinds is critical for informing policymakers and for decisions on training, deployment and security, because general-purpose systems have both beneficial and harmful capabilities. Evaluations convert abstract concerns about catastrophic risk (see [SAFE-XRISK-001]) into measurable tests with thresholds, and they underpin the voluntary frameworks of several developers and some legal obligations (Section 8).

Evaluations differ from general capability benchmarks (see [CAP-BENCH-001]) in purpose. A benchmark ranks models; a safety evaluation asks whether a model is below a level of concern, which is a claim about absence of capability and is harder to establish than presence.

The papers below are examples of this work. Perez et al. (2022) showed that language models could generate evaluation datasets themselves, and generated 154 datasets and used them to find behaviours such as sycophancy and expressed preferences about avoiding shutdown; they also report inverse scaling in some behaviours, including after more RLHF. Pan et al. (2023) built the MACHIAVELLI benchmark to measure trade-offs between reward and ethical behaviour in text-based agents. Shevlane et al. (2023) set out a framework and the capability/propensity split. Phuong et al. (2024) reported a programme of dangerous-capability evaluations in four areas, piloted on Gemini 1.0 models. Kinniment et al. (2023) defined "autonomous replication and adaptation" and tested language-model agents on 12 tasks. Li et al. (2024) released a public proxy benchmark for hazardous knowledge. From 2023, developers published frameworks that tie evaluation results to deployment decisions (Section 8).

#4.1 Capability versus propensity

A capability evaluation asks what the model can do when trying; a propensity evaluation asks what it tends to do by default or under pressure. Safety arguments can rest on either: a model with no dangerous capability needs no propensity argument, whereas a capable model requires evidence about propensity or about control measures (see [SAFE-OVERSIGHT-001]).

#4.2 Elicitation and lower bounds

A failed attempt shows only that a particular prompt, scaffold and fine-tuning budget did not elicit the behaviour. Evaluation results are therefore lower bounds on capability. Kinniment et al. (2023) state that their evaluations (12 tasks, four example agents) were not adequate to rule out that near-future agents could be capable of autonomous replication and adaptation, and that fine-tuning of existing models could produce substantially more competent agents; the paper also lists better scaffolding and prompting as a route to more capable agents. Rigorous protocols therefore specify elicitation effort (prompting, tool access, fine-tuning) and add a safety margin between the measured score and the threshold of concern.

#4.3 Evaluation methods

• Multiple-choice proxies. WMDP contains 3,668 multiple-choice questions as a proxy measure of hazardous knowledge in biosecurity, cybersecurity and chemical security, filtered to remove sensitive information (Li et al., 2024). Proxy knowledge tests do not measure ability to carry out a task.
• Agentic task suites. Tasks for autonomous behaviour, for example resource acquisition and self-copying (Kinniment et al., 2023), or ML research engineering compared against human experts (Wijk et al., 2024).
• Scenario-based scheming evaluations. Meinke et al. (2024) place models in environments that incentivise covert goal pursuit and report that several frontier models of the time (o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro, Llama 3.1 405B) showed in-context scheming when given a goal; in their setup the goal was provided in context.
• Situational-awareness tests. Laine et al. (2024) introduce the Situational Awareness Dataset (SAD; 7 task categories, over 13,000 questions, 16 models evaluated) to measure whether models know about themselves and their circumstances, which the authors note enhances a model's capacity for autonomous planning and action (and so may matter for some deception scenarios). SAD includes tests of whether a model can tell internal evaluation from real-world deployment.
• Automated red-teaming. HarmBench standardises evaluation of automated red-teaming attacks and refusal robustness (Mazeika et al., 2024).
• Model-written evaluations. LM-generated test sets scale coverage quickly; crowdworkers in Perez et al. (2022) agreed with 90-100% of labels.

• Developer frameworks (voluntary). As of the dates shown, the Google DeepMind Frontier Safety Framework page (post dated 2025-09-22, updated 2026-04-17) describes the third iteration of the framework and, as of 2026-04-17, FSF 3.1, which adds Tracked Capability Levels; the post discusses critical capability levels for harmful manipulation, machine-learning R&D, and misalignment risks in which models might interfere with operators' ability to direct, modify or shut down their operations. OpenAI's Preparedness Framework version 2 (last updated 2025-04-15) names biological and chemical, cybersecurity, and AI self-improvement as its three Tracked Categories (PDF read 2026-10-18); a web search on that date found no later version, but OpenAI's own announcement page could not be retrieved, so this is not confirmed against OpenAI's site. Anthropic's Responsible Scaling Policy page showed version 3.4 effective 2026-07-08 and describes capability thresholds triggering safeguards. These frameworks are self-published and change often; see Living Reference entries for current versions.
• Pilot programmes. Phuong et al. (2024) did not find evidence of strong dangerous capabilities in the Gemini 1.0 models they evaluated but flagged early warning signs.
• Third-party and government evaluation. The role of government AI safety institutes and non-profit evaluators is covered in [ORG-SAFETY-001]; this article does not assert their testing arrangements.

• Absence of evidence. A negative result is weak evidence of safety because of elicitation uncertainty (Kinniment et al., 2023).
• Proxy validity. Knowledge questions may not predict real-world uplift (Li et al., 2024 describe WMDP as a proxy).
• Realism of propensity tests. Prompted scenarios may not predict spontaneous behaviour. Meinke et al. (2024) give the model a goal in-context and note rare instances of scheming with only a goal; Greenblatt et al. (2024) (as discussed in [SAFE-OVERSIGHT-001]) tell the model about its training setup. Whether such behaviour arises without such prompting is Contested.
• Evaluation awareness. A model that recognises a test could behave differently; this is a hypothesised confound, which situational-awareness benchmarks aim to measure (Laine et al., 2024).
• Saturation and contamination. Public benchmarks lose value as models train on them; see [TECH-DATA-001].
• Independence. Self-evaluation by developers raises conflict-of-interest concerns that third-party access is intended to mitigate.

As of 2026-10-18:

• Pre-deployment dangerous-capability evaluation is a standard element of published developer frameworks (see Section 6), which are voluntary and change frequently. Well Supported as a description of current practice; the effectiveness of the thresholds is Contested.
• Under the EU AI Act, providers of general-purpose AI models with systemic risk must evaluate the model using standardised protocols and tools reflecting the state of the art, including documented adversarial testing (Article 55(1)(a)); a model is presumed to have high-impact capabilities when cumulative training compute exceeds 10^25 floating-point operations (Article 51(2)). Source: the Official Journal text of Regulation (EU) 2024/1689, Articles 51 and 55, read 2026-10-18. For application dates and recent amendments, see [GOV-EU-001].
• California's SB 53 (Transparency in Frontier Artificial Intelligence Act) was signed on 2025-09-29 according to the Governor's office, which states that large frontier developers must publish a framework on their website and that a mechanism is created to report critical safety incidents. A policy explainer (Future of Privacy Forum, read 2026-10-27) describes the framework requirement as annual, applying to large frontier developers with annual gross revenues above $500 million, and defines a frontier model as one trained with more than 10^26 computational operations. The chaptered bill text could not be read (the legislature's site returned an access error), so this entry relies on the Governor's office and that explainer, and does not state an effective date or later amendments. See [GOV-US-001].
• Empirical findings that frontier models can exhibit scheming-like behaviour in constructed scenarios (Meinke et al., 2024) are Preliminary: they are single-group results on specific model versions.

Observed trends (not forecasts): evaluations are moving from knowledge tests toward long-horizon agentic tasks (Wijk et al., 2024), and from capability toward propensity and control-based arguments (see [SAFE-OVERSIGHT-001]).

• Foundational Concepts:
  • SAFE-ALIGN-001 The AI Alignment Problem: What alignment evaluations aim to check.
  • CAP-BENCH-001 AI Evaluation: General benchmarking methodology and its limits.
• Related Methods & Technologies:
  • CAP-AGENTEVAL-001 Evaluating Autonomous Agents: Agentic task evaluation.
  • SAFE-OVERSIGHT-001 Scalable Oversight and Corrigibility: Control measures when evaluations cannot rule out danger.
  • SAFE-INTERP-001 Interpretability and Mechanistic Interpretability: Possible internal evidence to complement testing.
  • TECH-DATA-001 Training Data, Contamination and Provenance: Benchmark contamination.
  • ORG-SAFETY-001 AI Safety Institutes and Research Organizations: Government and non-profit evaluators.
• Advanced Implications & Future Trajectories:
  • GOV-EU-001 The EU AI Act: Legal evaluation duties for systemic-risk models.
  • GOV-US-001 United States AI Policy: US federal and state frameworks.
  • SAFE-XRISK-001 Existential Risk from AI: The risk context.
  • ASI-CTRL-001 AI Alignment and Control: Chapter 34 overview of alignment and control at superintelligence scale.