Capabilities and Performance

Software engineering capability is the ability of an AI system to carry out tasks in the software development lifecycle: writing functions from specifications, modifying existing codebases to fix defects or add features, testing, and operating in development environments. Its evaluation spans three levels of realism: function-level synthesis benchmarks, repository-level issue-resolution benchmarks, and field studies of developers using AI tools.

Code is an unusually good domain for automated evaluation because correctness can be partially checked by executing tests. This has made software tasks a leading testbed for AI capability claims, for agent evaluation (see [CAP-AGENTEVAL-001]) and for measuring autonomy (Kwa et al., 2025). It also creates a characteristic risk: a benchmark that measures "tests pass" can diverge from what developers actually need, which includes correct behaviour beyond the tests, maintainable design, review, and integration into a team's work.

This article explains how coding capability is measured and where each measurement is weak. How code models are built is in [TECH-CODE-001], and deployment in industry is in [APP-SW-001]. Because coding scores change rapidly, no current leaderboard values are given.

Function-level synthesis. Chen et al. (2021) introduced HumanEval, a set of hand-written programming problems with unit tests, and the pass@$k$ metric, in the report on Codex, a language model fine-tuned on code. They reported that Codex solved 28.8% of the problems with a single sample while GPT-3 solved none, and that repeated sampling improved coverage (the 2021 figures are for those models only). Austin et al. (2021) introduced the MBPP benchmark of short programming problems in the same period and studied program synthesis with large language models.

Strengthening function-level tests. Liu et al. (2023) argued that the tests in existing benchmarks are insufficient and produced EvalPlus, which augments test cases; they reported that model scores fall when tests are strengthened, showing that a portion of earlier "passes" were incorrect programs.

Contamination-resistant and holistic evaluation. Jain et al. (2024) built LiveCodeBench, which continuously collects new problems from programming contests so that problems post-date a model's training cut-off, and which evaluates capabilities beyond generation, such as self-repair and test-output prediction.

Repository-level tasks. Jimenez et al. (2023) introduced SWE-bench: 2,294 problems drawn from real GitHub issues and the corresponding pull requests in 12 popular Python repositories, in which a model must edit a codebase so that the repository's tests pass. At publication, the best model they evaluated (Claude 2) resolved 1.96% of issues. Yang et al. (2024) showed that the design of the agent-computer interface (the commands and feedback an LLM agent uses to navigate and edit code) affects performance, with their SWE-agent system improving results over non-interactive baselines in their experiments.

#4.1 The pass@$k$ metric

For a problem, sample $n$ completions, of which $c$ pass all tests. The unbiased estimator of the probability that at least one of $k$ samples passes is

(Chen et al., 2021). pass@1 measures single-attempt performance; large $k$ measures coverage when an external check can select the passing sample. Which is appropriate depends on whether a reliable verifier is available in deployment.

#4.2 Resolution rate in repository tasks

For issue-resolution benchmarks, a task counts as resolved if the model's patch, applied to the repository, makes designated failing tests pass without breaking tests that previously passed (Jimenez et al., 2023). The metric depends on the quality of those tests. A patch that differs from the human fix but passes is credited, and a correct patch that the tests happen to reject is not.

#4.3 Time-referenced and field evaluation

Kwa et al. (2025) measure how long skilled humans take on tasks that a model can complete with 50% reliability (see [CAP-REASON-001]). Becker et al. (2025) ran a randomised controlled trial with 16 experienced open-source developers on 246 tasks in repositories they knew well, assigning each task to allow or disallow AI tools (primarily Cursor Pro with Claude 3.5/3.7 Sonnet, February-June 2025). Developers forecast a 24% reduction in completion time beforehand and estimated afterward that AI had reduced time by 20%, but measured completion time increased by 19% when AI was allowed.

Human-validated subsets of SWE-bench have been released by third parties, including a "Verified" subset; details of its construction are not cited here because the primary page could not be retrieved [VERIFICATION NEEDED: construction and size of SWE-bench Verified].

AI coding systems are used for completion, chat-based assistance, test and documentation generation, and as autonomous or semi-autonomous agents that read an issue, explore a repository, edit files and run tests (Yang et al., 2024). In evaluation, the same underlying model can score very differently depending on the scaffold around it, so a result should name both the model and the agent harness. For comparing deployed products and hosted benchmark scores, see MODEL-001 and the methodology in [CAP-COMPARE-001].

Benchmark validity. An analysis of SWE-bench by Aleithan et al. (2024) manually screened patches that SWE-Agent with GPT-4 had successfully resolved (the top system at the time of their study) and reported that 32.67% of successful patches involved "solution leakage", where the solution was provided in the issue report or comments, and that 31.08% of passing patches were "suspicious" because of weak test cases; when problematic instances were filtered out, the resolution rate dropped substantially. This is a single audit of a particular system on a benchmark version; it demonstrates that such defects existed, not their prevalence in later variants.

Contamination. Public repositories and issues are likely to appear in pretraining data, and developers of closed models rarely disclose training corpora. LiveCodeBench's design (Jain et al., 2024) is a response, and general methods are in [CAP-BENCH-001].

Benchmark scores versus field productivity. Becker et al. (2025) found a measured slowdown for experienced developers in mature projects, contradicting both their own expectations and those of surveyed economists and ML experts, who predicted 39% and 38% shorter times respectively. The authors state that experimental artefacts cannot be entirely excluded, list 20 candidate contributing factors and note the study's narrow setting (16 developers, familiar mature repositories, early-2025 tools). The result does not show that AI tools slow all developers; it shows that high benchmark scores did not predict a speed-up in this setting, and that self-reports can be mistaken about effect size. Whether the effect holds for other populations and later tools is unresolved.

Open questions. (1) How to measure maintainability, security and design quality, which tests do not capture. (2) How to attribute gains between model and scaffold. (3) How benchmark gains map onto task difficulty as measured in human time.

As of 2026-10-15: execution-based function-level benchmarks are widely used and have documented problems of weak tests and contamination (Well Supported; Liu et al., 2023; Jain et al., 2024). Repository-level benchmarks measure a more realistic task than function synthesis but have documented validity defects in their original form (Well Supported for the audited instances; Aleithan et al., 2024). The only randomised field result cited here reports a slowdown in a specific setting (Preliminary; single study; Becker et al., 2025). This article makes no statement about the current state of the art on any leaderboard.

Observed trend (not a forecast): evaluation is moving from fixed public problem sets toward continuously refreshed problems, human-validated subsets, time-referenced task suites and controlled field studies (Jain et al., 2024; Kwa et al., 2025; Becker et al., 2025). Because field evidence is sparse and dated, claims about aggregate productivity effects of AI coding tools remain Preliminary. No forecast is registered here.

• Foundational Concepts:
  • CAP-BENCH-001 AI Evaluation, Benchmarking & Contamination: General evaluation and contamination methodology.
  • TECH-CODE-001 AI Code Generation: How code-generation models are built.
• Related Methods & Technologies:
  • TECH-AGENT-001 Autonomous AI Agents: Agent scaffolds used in repository tasks.
  • CAP-REASON-001 Reasoning, Planning and Long-Horizon Capability: Time-horizon measurements on software tasks.
  • CAP-AGENTEVAL-001 Evaluating Autonomous Agents: Environment-based evaluation of agents.
  • CAP-COMPARE-001 Methodology for Comparing AI Systems: Comparing coding scores across systems.
• Advanced Implications & Future Trajectories:
  • APP-SW-001 AI in Software Engineering: Deployment and workflow effects.
  • ASI-AIRD-001 Automated AI Research: Research engineering as a coding task.

Hallucination is the generation of content that is fluent but unsupported, either by the model's source input (a faithfulness failure) or by established fact (a factuality failure) (Ji et al., 2023; Huang et al., 2023). Calibration is the property that a system's expressed confidence matches its empirical accuracy: among answers given with 80% confidence, about 80% should be correct (Guo et al., 2017). Reliability is the consistency with which a system produces acceptable outputs across repeated use, inputs and conditions.

Language models are trained to produce plausible continuations, not to verify claims. Consequently, they can state false information in the same register as true information, which is the central obstacle to using them in settings where errors are costly. The three terms in the title are related but distinct. A system can hallucinate rarely yet be poorly calibrated (it is rarely wrong but cannot tell when it is). It can be well calibrated yet unreliable (it knows it is wrong half the time). And it can be accurate on average while unreliable per user, because average accuracy hides variance across repeated attempts.

The terminology itself is contested. "Hallucination" borrows a perceptual term, and some authors prefer "confabulation" for plausible fabrications. This article uses "hallucination" because it is the dominant term in the literature, without implying any mechanism of perception.

Before large language models, "hallucination" was documented in neural machine translation and summarisation, where systems produced output unsupported by the source text. Ji et al. (2023) surveyed this work across natural-language-generation tasks and organised it into intrinsic cases (output contradicts the source) and extrinsic cases (output adds material that cannot be verified from the source). Calibration of neural networks was studied separately: Guo et al. (2017) reported that modern deep networks were substantially more miscalibrated than earlier, smaller networks, and that simple post-hoc temperature scaling was an effective remedy for classifiers.

With instruction-following language models, attention shifted to factuality in open-ended answers. Lin et al. (2021) introduced TruthfulQA to test whether models reproduce common human misconceptions. Kadavath et al. (2022) examined whether models can evaluate their own correctness. Huang et al. (2023) proposed a taxonomy covering factuality and faithfulness hallucination in LLMs, and Kalai and Vempala (2023) and Kalai et al. (2025) put forward statistical accounts of why hallucination arises and persists.

#4.1 Why hallucination occurs: three explanations

1. Statistical inevitability for rare facts. Kalai and Vempala (2023) show that a language model with calibrated predictions must hallucinate at a rate bounded below by a statistic of the training data related to facts seen once, even on idealised training data. The result concerns arbitrary facts that cannot be inferred from patterns (a particular person's birthday, say), not facts derivable by reasoning.
2. Pretraining as classification of valid versus invalid text. Kalai et al. (2025) argue that if incorrect statements cannot be distinguished from facts during training, errors arise from ordinary statistical pressures, in the way that misclassifications arise in binary classification.
3. Evaluation incentives. The same paper argues that hallucinations persist because most benchmarks grade only correct answers, so guessing scores at least as well as abstaining, and models are optimised to be good test-takers; the authors propose modifying the scoring of existing benchmarks to credit appropriate expressions of uncertainty. These are arguments from a theoretical analysis; their relative contribution in specific production systems has not been established by the sources cited here.

#4.2 Calibration: definition and measurement

A model is perfectly calibrated if $P(\hat{Y} = Y \mid \hat{P} = p) = p$ for all $p \in [0,1]$, where $\hat{P}$ is the reported confidence and $\hat{Y}$ the prediction. The commonly reported summary, expected calibration error (ECE), partitions predictions into $M$ confidence bins $B_m$ and computes

where $n$ is the number of predictions, $\mathrm{acc}(B_m)$ the accuracy and $\mathrm{conf}(B_m)$ the mean confidence in bin $m$ (Guo et al., 2017). ECE is sensitive to binning and does not capture whether confidence is useful for separating correct from incorrect answers; ranking quality is measured separately, for example by failure-prediction performance (Xiong et al., 2023).

#4.3 Sources of confidence for language models

Three signals are in use. (a) Token probabilities: Kadavath et al. (2022) reported that larger models were well calibrated on multiple-choice questions in a suitable format and that models could be trained to estimate the probability that their own answer is correct. (b) Verbalised confidence: Xiong et al. (2023) evaluated black-box methods and found that models tended to be overconfident when verbalising confidence, that calibration and failure prediction improved with model capability, and that strategies such as human-inspired prompts and consistency across multiple samples helped. (c) Semantic entropy: Farquhar et al. (2024) estimate uncertainty over the meaning of generated answers rather than over exact token sequences, and use it to detect a class of confabulated answers. Each depends on the model family and task, and none is a general guarantee.

#4.4 Reliability as a repeated-trial property

Average accuracy conceals inconsistency. $\tau$-bench proposed a metric, pass^k, for the probability that an agent succeeds on all of $k$ independent trials of the same task, and reported that even the state-of-the-art function-calling agents it tested (such as gpt-4o, as of the 2024 paper) succeeded on fewer than 50% of tasks and were inconsistent across trials (Yao et al., 2024). The principle, which does not depend on the particular figure, is that deployment in repeated use depends on pass^k rather than pass@1. See [CAP-AGENTEVAL-001].

• Faithfulness hallucination: output contradicts or is unsupported by a supplied source (a summary that adds a claim not in the document).
• Factuality hallucination: output conflicts with world knowledge (a false statement of fact, a fabricated citation, an invented API function).
• Intrinsic versus extrinsic: contradicts the source versus cannot be verified from it (Ji et al., 2023).
• Reasoning errors: incorrect inference from correct premises; related to but distinct from hallucination (see [CAP-REASON-001]).
• Mitigation families: retrieval and grounding on trusted sources; verification against external tools; abstention training and reward design that credits "I don't know" (Kalai et al., 2025); decoding-time uncertainty estimation (Farquhar et al., 2024); human review.

Hallucination matters most where output is trusted without checking: citations and legal or medical references, code that calls non-existent functions, summaries of documents, and agents that act on generated content. TruthfulQA illustrates a measured form: the authors tested whether models give false answers that mimic popular misconceptions, and reported (for the models and date of their 2021 study) that the best model was substantially less truthful than humans, and that larger models were not uniformly more truthful (Lin et al., 2021). This finding concerns 2021-era models and is not evidence about current systems; it is cited for the measurement design and the observation that scale alone does not guarantee truthfulness.

Limits of mitigation. Retrieval reduces unsupported claims only if the retrieved text is correct and the model uses it faithfully. Verification by tools is possible only where a checker exists. Abstention trades coverage for accuracy, which many benchmarks penalise (Kalai et al., 2025).

Contested or open. (1) Whether hallucination can be eliminated, or only reduced: Kalai and Vempala (2023) give a lower bound for calibrated models on arbitrary rare facts, which is a statement about a theoretical class of models and calibrated predictions, and does not by itself say what is achievable with retrieval and tools. (2) Whether models "know what they know" in a sense useful for deployment (Kadavath et al., 2022, report positive results in specific formats; Xiong et al., 2023, report systematic overconfidence in verbalised confidence). (3) Whether post-training for helpfulness affects calibration; the sources cited here do not settle this. (4) Metrics: no single accepted measure combines factuality, faithfulness and calibration, and results depend on prompt format and grading method (see [CAP-COMPARE-001]).

As of 2026-10-15 the following is supported by the cited sources. Hallucination is documented across natural-language-generation tasks (Established; Ji et al., 2023). Modern deep networks are often miscalibrated and can be recalibrated post hoc in classification settings (Well Supported; Guo et al., 2017). LLM confidence can carry useful information, with the usefulness depending on the elicitation method, model and task (Well Supported to Preliminary; Kadavath et al., 2022; Xiong et al., 2023; Farquhar et al., 2024). Theoretical accounts of why hallucination occurs and persists exist and are published, but their empirical weight in deployed systems is Preliminary (Kalai & Vempala, 2023; Kalai et al., 2025). Current hallucination rates for named models are not given here; they vary by task, grading method and date, and belong in the Living Reference (MODEL-001).

Observed direction (not a forecast): evaluation practice is shifting toward scoring that credits calibrated abstention and toward repeated-trial reliability metrics rather than single-attempt accuracy (Kalai et al., 2025; Yao et al., 2024). Whether this changes model behaviour in deployed systems depends on whether widely used leaderboards adopt such scoring, which is a coordination question, not a technical one. No forecast is registered in this article.

• Foundational Concepts:
  • TECH-LLM-001 Large Language Models: The systems whose outputs hallucinate.
  • TECH-ML-001 Machine Learning: Probability and calibration background.
• Related Methods & Technologies:
  • CAP-BENCH-001 AI Evaluation, Benchmarking & Contamination: Benchmark design and the incentive effects discussed here.
  • CAP-REASON-001 Reasoning, Planning and Long-Horizon Capability: Reasoning errors and per-step reliability.
  • CAP-AGENTEVAL-001 Evaluating Autonomous Agents: Repeated-trial reliability.
  • CAP-COMPARE-001 Methodology for Comparing AI Systems: Why hallucination metrics are hard to compare.
  • CAP-GENAI-001 Generative AI: Generation across modalities.
• Advanced Implications & Future Trajectories:
  • SAFE-ROBUST-001 Technical Safety and Robustness: Reliability as a safety property.
  • SAFE-EVAL-001 Dangerous Capability and Alignment Evaluations: Evaluation of risky behaviours.

This article treats reasoning (drawing inferences that follow from given information), planning (choosing a sequence of actions that achieves a goal under a model of how actions change the world) and long-horizon capability (sustaining coherent, error-correcting progress over tasks that take many steps or a long time) as three measurable capabilities of AI systems, and surveys what has been demonstrated about each in language-model-based systems. How reasoning models are trained and how test-time compute is used is covered in [TECH-REASON-001]; this article concerns what the resulting systems can and cannot be shown to do.

Reasoning, planning and long-horizon execution are the capabilities that separate a system that answers questions from one that can complete work. They are also the capabilities on which reported progress is least straightforward to interpret. Benchmarks for reasoning are vulnerable to contamination and saturation (see [CAP-BENCH-001]); apparent reasoning in a text transcript may not reflect the computation that produced the answer (Turpin et al., 2023); and success on short tasks does not imply success on long ones when per-step errors compound.

Three questions organise the evidence. First, do intermediate reasoning steps improve accuracy, and why? Second, can language models plan, and under what conditions? Third, how long a task can a system complete reliably, and how fast is that changing? The answers are partial and, for the third, depend on a small number of studies.

Classical AI treated planning as search over a formal model of actions and states, and reasoning as inference in logic (Russell & Norvig, 2020). Language models were first assessed on reasoning through question-answering and arithmetic benchmarks, where large models without special prompting performed poorly on multi-step problems.

Wei et al. (2022) showed that prompting a sufficiently large model with examples that include intermediate steps ("chain of thought") improves performance on arithmetic, commonsense and symbolic reasoning tasks. Yao et al. (2022) proposed interleaving reasoning traces with actions in an environment (ReAct), the pattern underlying most tool-using agents (see [TECH-AGENT-001]). Lightman et al. (2023) reported that supervising each reasoning step ("process supervision") outperformed supervising only final answers for training models on the MATH dataset. Snell et al. (2024) studied allocating extra computation at inference time and reported that, for some problem types, doing so can be more effective than scaling model parameters. Reasoning-specialised models trained with reinforcement learning on verifiable outcomes followed; DeepSeek-AI's report is one openly documented example (Guo et al., 2025; see [TECH-REASON-001]). The 2025 reports on measured task horizon (Kwa et al., 2025) shifted attention from accuracy on fixed-size problems to the length of task a system can complete.

#4.1 Reasoning: what intermediate tokens do

A transformer language model performs a bounded amount of computation per generated token. Generating intermediate tokens before an answer gives the model a scratchpad whose contents are conditioned on in later steps. This is the mechanism usually offered for why chain-of-thought helps on multi-step problems (Wei et al., 2022). Two cautions apply. First, the visible reasoning may not be the cause of the answer: Turpin et al. (2023) showed that chain-of-thought explanations can systematically misrepresent the factors influencing a model's prediction, for instance by omitting a biasing feature that changed the answer. A reasoning trace is therefore evidence about behaviour but not a verified account of internal computation. Second, compositional generalisation has limits: Dziri et al. (2023) examined tasks that require composing many sub-steps and reported that transformers' performance degrades as compositional depth increases.

#4.2 Verification and self-correction

Training and inference methods that use a verifier (a process or outcome check) have been central to reported gains (Lightman et al., 2023). By contrast, Huang et al. (2023) found that, without external feedback, language models asked to review and correct their own reasoning did not reliably improve, and sometimes performed worse. The practical implication is that improvements generally depend on a source of ground truth outside the model: a unit test, a proof checker, a numeric answer key.

#4.3 Planning

In the formal sense, planning means producing an executable sequence of actions that provably reaches a goal. Valmeekam et al. (2022) built PlanBench, an extensible benchmark drawn from classical planning domains, and Valmeekam et al. (2023) reported that the language models they evaluated were rather limited at autonomously generating executable plans. Kambhampati et al. (2024) argue, as a position, that language models should not be regarded as planners on their own but can serve as approximate sources of candidate plans and knowledge inside LLM-Modulo frameworks, in which external critics verify candidates and return feedback. This claim is a position paper's argument, not a settled result; it is consistent with the self-correction findings above, and the same paper notes that claims at both optimistic and pessimistic extremes are overstated.

#4.4 Long-horizon capability and task time horizon

If a task consists of $n$ steps and each step succeeds independently with probability $p$, the probability of end-to-end success is $p^n$, which falls quickly with $n$ unless $p$ is very close to 1 or the system can detect and recover from errors. Real tasks are not independent-step chains, but the model explains why reliability matters more than peak ability for long tasks. Kwa et al. (2025) defined a metric, the 50% time horizon: the duration of tasks, measured by how long skilled humans take, that a model completes with 50% success. The metric converts agent performance to a human-referenced time scale, a different quantity from benchmark accuracy.

• By reasoning domain: mathematical and logical (checkable), commonsense and causal (partly checkable), planning (checkable against a domain model), scientific and open-ended (often not checkable).
• By mechanism: prompted chain-of-thought; trained reasoning models using extended internal deliberation; external-search methods; tool-augmented reasoning (code execution, retrieval).
• By horizon: single-step inference; multi-step problems within one response; tool-using episodes of minutes to hours; multi-session work that requires persistent memory or state (see [AGI-CL-001]).
• By evaluation design: fixed-answer benchmarks; controllable-complexity environments (puzzle families whose difficulty is parameterised, as in Shojaee et al., 2025); human-timed task suites (Kwa et al., 2025).

Reasoning-model behaviour is most clearly useful where outcomes can be checked automatically: mathematics problems with numeric answers, code with unit tests (see [CAP-CODE-001]), and structured data tasks. Evidence from deployed use in unverifiable domains is weaker. Tool-using agents apply ReAct-style loops to web, software and desktop tasks, evaluated in the environments discussed in [CAP-AGENTEVAL-001].

Two studies illustrate measured long-horizon performance. Kwa et al. (2025) timed human experts on a combination of tasks from RE-Bench, tasks from a second suite (HCAST) and 66 novel shorter tasks, and reported that models such as Claude 3.7 Sonnet had a 50% time horizon of around 50 minutes; they also reported that the horizon had doubled roughly every seven months since 2019, with a possible acceleration in 2024. Wijk et al. (2024) released RE-Bench, with seven open-ended ML research-engineering environments and data from 71 eight-hour attempts by 61 human experts, to enable direct comparison between agents and people on realistic AI R&D work (relevant to [ASI-AIRD-001]). These figures are as reported by the authors in 2024-2025 and describe the models then evaluated; current values are not given here.

Controllable-complexity results are contested. Shojaee et al. (2025) used puzzle environments with adjustable complexity and reported that reasoning models showed an "accuracy collapse" beyond certain complexities, that reasoning effort (tokens) increased with complexity up to a point and then declined, and that three regimes existed: standard models better at low complexity, reasoning models better at medium complexity, and both failing at high complexity. Lawsen (2025) replied that the findings primarily reflect experimental-design limitations, including output-token limits and an evaluation procedure that does not distinguish reasoning failures from practical constraints. Both documents are arXiv preprints; the extent to which the collapse reflects reasoning limits versus experimental artefacts is not settled.

Time-horizon extrapolation is uncertain. Kwa et al. (2025) themselves discuss limits of external validity: the task suites are software and research-engineering tasks, task difficulty is estimated from human time, and tasks in the suite are often cleaner than real work. A doubling trend measured on past models is an observation of those models on that suite, not a law. A randomised trial of experienced open-source developers (Becker et al., 2025) found that allowing early-2025 AI tools increased task completion time by 19% in their setting, against developer expectations of a 24% reduction beforehand, indicating that suite performance and in-the-wild productivity can diverge (see [CAP-CODE-001]).

Open questions. (1) Whether intermediate-token reasoning reflects faithful internal reasoning (Turpin et al., 2023). (2) How performance on verifiable tasks transfers to unverifiable ones. (3) Whether reliability on long horizons improves mainly with better per-step accuracy or with better error recovery; Kwa et al. (2025) attribute their measured horizon growth to a combination of greater reliability, better adaptation to mistakes, stronger logical reasoning and better tool use, but this attribution is the authors' interpretation. (4) Whether abstraction and generalisation, rather than skill, can be measured without prior knowledge confounds, a concern raised by Chollet (2019) and unresolved.

As of 2026-10-15, the following can be stated with the sources cited here. Intermediate reasoning steps improve accuracy on a range of multi-step tasks for sufficiently capable models (Well Supported; Wei et al., 2022). External verification materially helps, while unaided self-correction does not reliably help (Well Supported; Lightman et al., 2023; Huang et al., 2023). Language models used alone have been reported to be weak at autonomous formal planning (Preliminary to Well Supported, since the evaluated models date from 2022-2023 and the benchmark domains are narrow; Valmeekam et al., 2022, 2023). Measured task horizons have grown across model generations on one family of software-oriented suites (Preliminary; a single research group's metric). Accuracy-collapse results for reasoning models are Contested.

This article does not report current leaderboard values for reasoning benchmarks, which change rapidly and are subject to contamination; see MODEL-001 and the methodology in [CAP-COMPARE-001].

• Near term (0-3 years): Observed pattern, not a forecast: evaluation is moving from fixed-answer reasoning benchmarks to time-referenced and environment-based measures (Kwa et al., 2025; Wijk et al., 2024).
• Medium and long term: Kwa et al. (2025) state that if their results generalise to real-world software tasks, extrapolating the trend would imply systems capable of automating many software tasks currently taking humans a month within 5 years. That is the authors' conditional extrapolation, not a claim adopted here; the condition (generalisation beyond the suite) is itself uncertain, as Section 7 shows. This article registers no forecast of its own. A proper forecast would need the full record format in the evidence framework and an operational definition of "automating" a task.

• Foundational Concepts:
  • TECH-REASON-001 Reasoning Models and Test-Time Compute: How reasoning models are built.
  • TECH-LLM-001 Large Language Models: The underlying systems.
• Related Methods & Technologies:
  • TECH-AGENT-001 Autonomous AI Agents: The scaffolding in which long-horizon behaviour occurs.
  • CAP-BENCH-001 AI Evaluation, Benchmarking & Contamination: General evaluation methodology and contamination.
  • CAP-LIMITS-001 Hallucination, Reliability and Calibration: Per-step reliability.
  • CAP-CODE-001 Software Engineering Capability and Evaluation: Software tasks as a horizon testbed.
  • CAP-AGENTEVAL-001 Evaluating Autonomous Agents: Environment-based agent evaluation.
  • CAP-COMPARE-001 Methodology for Comparing AI Systems: How to compare reasoning claims across models.
• Advanced Implications & Future Trajectories:
  • AGI-001 Artificial General Intelligence: Reasoning breadth as a criterion.
  • AGI-CL-001 Continual Learning and Persistent Memory: Memory beyond a single context window.
  • ASI-AIRD-001 Automated AI Research: Research engineering as a capability.
• Living Reference:
  • MODEL-001 Model Registry: Current models and their reported properties.