Testing Claims of General Intelligence

Testing claims of general intelligence is the methodology of deciding what evidence would support or refute the statement that an artificial system is generally intelligent, and of designing, running and interpreting evaluations that supply such evidence. It combines the choice of an operational definition (see AGI-DEF-001) with measurement-validity concerns such as contamination, saturation, elicitation and human baselines.

Claims that a system shows general intelligence are made by developers, researchers and commentators, and they bear on investment, regulation and safety policy. A claim of this kind is a statement about performance across a wide, unspecified range of tasks, so it cannot be verified by any single finite test. What can be done is to assemble a body of evidence, each piece with a stated scope, and to state what the evidence does not cover.

The CAP-BENCH-001 article covers benchmarking and contamination generally. This article is narrower: it concerns the logic of inference from tests to a general-intelligence claim, and the specific instruments proposed for that purpose.

Three features distinguish testing for generality from ordinary capability evaluation.

1. Open task space. A general system is expected to handle tasks its designers did not anticipate, so any fixed test set can at best be a sample.
2. Adversarial incentive. Public benchmarks become training targets, directly or indirectly, so reported scores may not reflect the capability that the benchmark was designed to measure.
3. Definition dependence. The evidence needed depends on whether AGI is defined by economic outcomes, skill acquisition, a human-percentile taxonomy, or a cognitive profile.

Turing (1950) proposed a behavioural criterion, the imitation game, in place of the question "can machines think?". The test has the structure that later proposals share: observable behaviour under constrained conditions is taken as evidence about an unobservable property. Its limitations, including its dependence on the judge and on the conversational format, are discussed in FOUND-005.

For most of the deep-learning era, progress was tracked on fixed benchmarks. As language models improved, scores on widely used benchmarks approached ceilings. The Humanity's Last Exam paper (Phan et al., 2025) states that state-of-the-art language models score over 90% on popular benchmarks such as MMLU, and presents a new closed-ended question set in response. Chollet (2019) argued earlier that skill on a task measures prior knowledge and training exposure as much as intelligence, and proposed a benchmark (the Abstraction and Reasoning Corpus) designed around novel tasks; a successor, ARC-AGI-2, was released in 2025 (Chollet et al., 2025).

A further shift moved from knowledge questions to extended tasks. RE-Bench (Wijk et al., 2024) compares agents with human experts on research-engineering environments, MLE-bench (Chan et al., 2024) uses machine-learning competitions, and the time-horizon approach (Kwa et al., 2025) measures the length of tasks that models complete. In parallel, taxonomy and profile frameworks (Morris et al., 2024; Hendrycks et al., 2025) attempt to organise evidence across many tests.

#4.1 The inference problem

Let $T$ be a test and $s(T)$ a system's score. A claim "the system is generally intelligent" is supported by $s(T)$ only to the extent that $T$ is a valid and representative sample of the target set $\mathcal{G}$ defined by the chosen definition. Writing $P(\text{general}\mid s(T))$ for the support, the following conditions all need to hold for $s(T)$ to be informative:

• Representativeness: tasks in $T$ are drawn so as to cover $\mathcal{G}$ rather than a convenient subset.
• Novelty: the system could not have been prepared for $T$ by training exposure. Otherwise $s(T)$ measures memorisation or familiarity.
• Construct validity: performance on $T$ depends on the capability named, not on incidental features such as answer format.
• Elicitation adequacy: the system was given adequate tools, prompting and time, so that low scores do not reflect poor setup.
• Baseline validity: where comparison with humans is made, the human group is defined and appropriately incentivised and equipped.

Failure of any condition weakens the inference, and none can be established from the score alone.

#4.2 Families of test

Behavioural imitation tests. The Turing test and its variants. Their strength is breadth, since a conversation can probe many subjects. Their weakness is that success may reflect conversational strategy and judge expectations rather than general competence (see FOUND-005).

Static knowledge and reasoning batteries. Large question sets across subjects. Phan et al. (2025) describe Humanity's Last Exam as a multimodal benchmark of 2,500 questions written by subject-matter experts, with known, verifiable solutions that cannot be quickly found by internet search, and report low accuracy and calibration for state-of-the-art models at the time of writing. A closed-ended expert question set tests knowledge and reasoning on questions with unique answers; it does not test open-ended research or sustained action.

Novel-task and skill-acquisition tests. ARC-AGI (Chollet, 2019) and ARC-AGI-2 (Chollet et al., 2025) present input-output pair tasks that are meant to require inferring a rule from few examples. They target requirement 1 in AGI-REQ-001. They cover a narrow format, and results on them are not evidence about tasks that are not grid-based puzzles. Whether a given result reflects novel-task inference or large-scale search and preparation depends on the evaluation conditions, which differ across reports.

Agentic and long-horizon tests. RE-Bench consists of 7 open-ended machine-learning research-engineering environments with data from 71 8-hour attempts by 61 distinct human experts; the authors report that the best AI agents achieve a score 4x higher than human experts when both are given a total time budget of 2 hours per environment, while humans achieve 2x the score of the top AI agent when both are given 32 total hours (Wijk et al., 2024). MLE-bench curates 75 Kaggle competitions and reports that the best-performing setup at the time of writing (OpenAI's o1-preview with AIDE scaffolding) achieved at least a Kaggle bronze medal level in 16.9% of competitions (Chan et al., 2024). Kwa et al. (2025) report that frontier models such as Claude 3.7 Sonnet had a 50% time horizon of around 50 minutes and that the horizon had doubled approximately every seven months since 2019 in their data. These are time-specific values from the cited papers; they are not current figures.

Taxonomies and profiles. Morris et al. (2024) propose a matrix of performance level by generality; its use requires a broad suite of tasks to assign a system to a level. [VERIFICATION NEEDED: whether the paper itself states that an agreed benchmark suite does not yet exist] Hendrycks et al. (2025) propose ten cognitive domains and scoring; their scores are Preliminary.

#4.3 Threats to validity

• Contamination. Test items or close variants appear in training data. Contamination is addressed in CAP-BENCH-001; private held-out sets reduce but do not eliminate the problem, because repeated queries to a private set leak information.
• Saturation. Scores approach the ceiling and the benchmark ceases to discriminate (Phan et al., 2025, on older benchmarks).
• Goodhart effects. Optimising for a benchmark score degrades its validity as a measure.
• Scaffolding and compute asymmetries. Reported scores can depend strongly on the number of attempts, tool access and compute spent per task. The RE-Bench result above illustrates the interaction between time budget and relative performance: the ranking of agents and humans reverses between 2 hours and 32 hours (Wijk et al., 2024).
• Task-length and success-threshold dependence. A 50% success horizon (Kwa et al., 2025) is a statement at a particular reliability level; the horizon at a higher required success rate is shorter, and many practical uses need high reliability.
• Selection of human comparison. "Human-level" depends on whether the comparison group is experts, crowd workers or the general population, and on how much time and tooling they have.

#4.4 Reporting practice

To make an AGI-relevant claim testable, an evaluation report should state: the definition used; the full task list or its sampling procedure; the training-data exposure checks performed; the elicitation protocol, including number of attempts and tools; the human baseline construction; and uncertainty estimates. Few published results satisfy all of these; the papers reviewed here vary in how many they do.

• Research evaluation. The tests above are used by research groups and independent evaluators to report capability.
• Safety frameworks. Capability evaluations determine thresholds for additional safeguards; those are covered in SAFE-EVAL-001. The same validity concerns apply, with the added requirement that negative results (a system cannot do X) are hard to establish, because failure may be an elicitation failure.
• Case: scaffolding sensitivity. In MLE-bench, results depend on the scaffold used with the underlying model (Chan et al., 2024), which shows that a "model score" is a property of a model-plus-scaffold system.
• Case: time budget. In RE-Bench the relative standing of agents and humans changes with time budget (Wijk et al., 2024), so a single comparison number misrepresents the result.

No test reviewed here has been presented by its authors as sufficient to establish AGI. The authors of each frame their instrument as measuring a part of the problem.

Advantages of structured testing. Explicit protocols make claims falsifiable, support comparison across systems and dates, and expose which sub-capabilities lag.

Limitations.

• No finite battery can certify an open-ended property such as generality; negative evidence is also weak when elicitation is imperfect.
• Instrument lifetimes are short: tests are replaced by harder successors as scores rise (for example the move from ARC-AGI to ARC-AGI-2).
• Human baselines are expensive and small. RE-Bench has 71 expert attempts in total; its conclusions are about those environments and participants.
• Developer-run evaluations and independent evaluations can differ, and in general only the former can access internal information such as training data.

Open questions.

1. Is there a principled way to sample from the space of "all economically valuable tasks" or "all human-solvable tasks"?
2. How should reliability be incorporated: a system that succeeds on 50% of long tasks is not equivalent to one that succeeds on 99%.
3. Can evaluation remain informative once systems are used to help design or answer the evaluations?
4. What counts as a pass: a threshold on a profile, a conjunction over domains, or a human-comparison criterion?

As of 2026-10-15, no test or battery is accepted by the research community as a certificate of general intelligence, and the sources reviewed do not claim that one exists. Published evaluations report partial results on specific instruments, with dates and model versions that are superseded quickly; consult the Living Reference collections (MODEL-001, FACT-001) for current figures. Evidence from the instruments discussed is Preliminary: each rests on a single paper or a small group of papers, several are preprints, and independent replication on updated models is not documented in the sources reviewed here. The ARC-AGI-2 paper and the Humanity's Last Exam paper each argue that earlier tests no longer provide adequate discrimination at the frontier, which is consistent with, but does not prove, the saturation concern.

This section is descriptive and registers no forecast.

• Near term (0 to 3 years). Continued release of harder or longer benchmarks, increased use of private held-out sets, and more emphasis on agentic, time-horizon and reliability measures.
• Medium term (3 to 10 years). Possible shift towards evaluations that use real deployment outcomes, such as economic tasks completed, rather than constructed benchmarks. No source reviewed documents such a shift; it is a plausible direction, not an observed trend.
• Long term (10+ years). If systems exceed human performance on most constructed tests, evaluation may rely on comparisons with groups of human experts at scale, or on field outcomes. This is speculative.

Dated forecasts about benchmark milestones are kept in the Forecast Register (FCST-REG-001).

• Foundational Concepts:
  • AGI-001 Artificial General Intelligence: The concept whose attainment the tests assess.
  • AGI-DEF-001 Definitions and Operational Criteria for AGI: The definitions that determine what a test must show.
  • CAP-BENCH-001 AI Evaluation, Benchmarking & Contamination: General benchmarking methodology.
• Related Methods & Technologies:
  • AGI-REQ-001 Requirements for General Intelligence: Capabilities that tests may target.
  • CAP-AGENTEVAL-001 Evaluating Autonomous Agents: Agent-specific evaluation.
  • SAFE-EVAL-001 Dangerous Capability and Alignment Evaluations: Use of evaluations in safety frameworks.
• Advanced Implications & Future Trajectories:
  • AGI-TIME-001 AGI Forecasting & Timeline Methodology: Forecast resolution depends on the test used.
  • ASI-AIRD-001 Automated AI Research: Benchmarks for AI research capability.