---
id: CAP-REASON-001
title: "Reasoning, Planning and Long-Horizon Capability"
alternative_names:
  - "LLM Reasoning Capability"
  - "AI Planning Capability"
  - "Task Time Horizon"
category: "Capabilities → Cognitive Capability"
volume: IV
chapter: 23
entry_type: "Technical Article"
level: 4
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "TECH-REASON-001"
  - "TECH-LLM-001"
related_entries:
  - "TECH-AGENT-001"
  - "CAP-BENCH-001"
  - "CAP-LIMITS-001"
  - "CAP-AGENTEVAL-001"
  - "CAP-CODE-001"
  - "AGI-001"
  - "ASI-AIRD-001"
---

# Reasoning, Planning and Long-Horizon Capability

## 1. Concise Definition
This article treats **reasoning** (drawing inferences that follow from given information), **planning** (choosing a sequence of actions that achieves a goal under a model of how actions change the world) and **long-horizon capability** (sustaining coherent, error-correcting progress over tasks that take many steps or a long time) as three measurable capabilities of AI systems, and surveys what has been demonstrated about each in language-model-based systems. How reasoning models are trained and how test-time compute is used is covered in [TECH-REASON-001]; this article concerns what the resulting systems can and cannot be shown to do.

## 2. Overview & Significance
Reasoning, planning and long-horizon execution are the capabilities that separate a system that answers questions from one that can complete work. They are also the capabilities on which reported progress is least straightforward to interpret. Benchmarks for reasoning are vulnerable to contamination and saturation (see [CAP-BENCH-001]); apparent reasoning in a text transcript may not reflect the computation that produced the answer (Turpin et al., 2023); and success on short tasks does not imply success on long ones when per-step errors compound.

Three questions organise the evidence. First, do intermediate reasoning steps improve accuracy, and why? Second, can language models plan, and under what conditions? Third, how long a task can a system complete reliably, and how fast is that changing? The answers are partial and, for the third, depend on a small number of studies.

## 3. Historical Development
Classical AI treated planning as search over a formal model of actions and states, and reasoning as inference in logic (Russell & Norvig, 2020). Language models were first assessed on reasoning through question-answering and arithmetic benchmarks, where large models without special prompting performed poorly on multi-step problems.

Wei et al. (2022) showed that prompting a sufficiently large model with examples that include intermediate steps ("chain of thought") improves performance on arithmetic, commonsense and symbolic reasoning tasks. Yao et al. (2022) proposed interleaving reasoning traces with actions in an environment (ReAct), the pattern underlying most tool-using agents (see [TECH-AGENT-001]). Lightman et al. (2023) reported that supervising each reasoning step ("process supervision") outperformed supervising only final answers for training models on the MATH dataset. Snell et al. (2024) studied allocating extra computation at inference time and reported that, for some problem types, doing so can be more effective than scaling model parameters. Reasoning-specialised models trained with reinforcement learning on verifiable outcomes followed; DeepSeek-AI's report is one openly documented example (Guo et al., 2025; see [TECH-REASON-001]). The 2025 reports on measured task horizon (Kwa et al., 2025) shifted attention from accuracy on fixed-size problems to the length of task a system can complete.

## 4. Technical / Conceptual Explanation

### 4.1 Reasoning: what intermediate tokens do
A transformer language model performs a bounded amount of computation per generated token. Generating intermediate tokens before an answer gives the model a scratchpad whose contents are conditioned on in later steps. This is the mechanism usually offered for why chain-of-thought helps on multi-step problems (Wei et al., 2022). Two cautions apply. First, the visible reasoning may not be the cause of the answer: Turpin et al. (2023) showed that chain-of-thought explanations can systematically misrepresent the factors influencing a model's prediction, for instance by omitting a biasing feature that changed the answer. A reasoning trace is therefore evidence about behaviour but not a verified account of internal computation. Second, compositional generalisation has limits: Dziri et al. (2023) examined tasks that require composing many sub-steps and reported that transformers' performance degrades as compositional depth increases.

### 4.2 Verification and self-correction
Training and inference methods that use a **verifier** (a process or outcome check) have been central to reported gains (Lightman et al., 2023). By contrast, Huang et al. (2023) found that, without external feedback, language models asked to review and correct their own reasoning did not reliably improve, and sometimes performed worse. The practical implication is that improvements generally depend on a source of ground truth outside the model: a unit test, a proof checker, a numeric answer key.

### 4.3 Planning
In the formal sense, planning means producing an executable sequence of actions that provably reaches a goal. Valmeekam et al. (2022) built PlanBench, an extensible benchmark drawn from classical planning domains, and Valmeekam et al. (2023) reported that the language models they evaluated were rather limited at autonomously generating executable plans. Kambhampati et al. (2024) argue, as a position, that language models should not be regarded as planners on their own but can serve as approximate sources of candidate plans and knowledge inside **LLM-Modulo** frameworks, in which external critics verify candidates and return feedback. This claim is a position paper's argument, not a settled result; it is consistent with the self-correction findings above, and the same paper notes that claims at both optimistic and pessimistic extremes are overstated.

### 4.4 Long-horizon capability and task time horizon
If a task consists of $n$ steps and each step succeeds independently with probability $p$, the probability of end-to-end success is $p^n$, which falls quickly with $n$ unless $p$ is very close to 1 or the system can detect and recover from errors. Real tasks are not independent-step chains, but the model explains why reliability matters more than peak ability for long tasks. Kwa et al. (2025) defined a metric, the **50% time horizon**: the duration of tasks, measured by how long skilled humans take, that a model completes with 50% success. The metric converts agent performance to a human-referenced time scale, a different quantity from benchmark accuracy.

## 5. Types, Components & Classifications
- **By reasoning domain**: mathematical and logical (checkable), commonsense and causal (partly checkable), planning (checkable against a domain model), scientific and open-ended (often not checkable).
- **By mechanism**: prompted chain-of-thought; trained reasoning models using extended internal deliberation; external-search methods; tool-augmented reasoning (code execution, retrieval).
- **By horizon**: single-step inference; multi-step problems within one response; tool-using episodes of minutes to hours; multi-session work that requires persistent memory or state (see [AGI-CL-001]).
- **By evaluation design**: fixed-answer benchmarks; controllable-complexity environments (puzzle families whose difficulty is parameterised, as in Shojaee et al., 2025); human-timed task suites (Kwa et al., 2025).

## 6. Applications & Concrete Examples
Reasoning-model behaviour is most clearly useful where outcomes can be checked automatically: mathematics problems with numeric answers, code with unit tests (see [CAP-CODE-001]), and structured data tasks. Evidence from deployed use in unverifiable domains is weaker. Tool-using agents apply ReAct-style loops to web, software and desktop tasks, evaluated in the environments discussed in [CAP-AGENTEVAL-001].

Two studies illustrate measured long-horizon performance. Kwa et al. (2025) timed human experts on a combination of tasks from RE-Bench, tasks from a second suite (HCAST) and 66 novel shorter tasks, and reported that models such as Claude 3.7 Sonnet had a 50% time horizon of around 50 minutes; they also reported that the horizon had doubled roughly every seven months since 2019, with a possible acceleration in 2024. Wijk et al. (2024) released RE-Bench, with seven open-ended ML research-engineering environments and data from 71 eight-hour attempts by 61 human experts, to enable direct comparison between agents and people on realistic AI R&D work (relevant to [ASI-AIRD-001]). These figures are as reported by the authors in 2024-2025 and describe the models then evaluated; current values are not given here.

## 7. Advantages, Limitations & Unresolved Questions

**Controllable-complexity results are contested.** Shojaee et al. (2025) used puzzle environments with adjustable complexity and reported that reasoning models showed an "accuracy collapse" beyond certain complexities, that reasoning effort (tokens) increased with complexity up to a point and then declined, and that three regimes existed: standard models better at low complexity, reasoning models better at medium complexity, and both failing at high complexity. Lawsen (2025) replied that the findings primarily reflect experimental-design limitations, including output-token limits and an evaluation procedure that does not distinguish reasoning failures from practical constraints. Both documents are arXiv preprints; the extent to which the collapse reflects reasoning limits versus experimental artefacts is not settled.

**Time-horizon extrapolation is uncertain.** Kwa et al. (2025) themselves discuss limits of external validity: the task suites are software and research-engineering tasks, task difficulty is estimated from human time, and tasks in the suite are often cleaner than real work. A doubling trend measured on past models is an observation of those models on that suite, not a law. A randomised trial of experienced open-source developers (Becker et al., 2025) found that allowing early-2025 AI tools increased task completion time by 19% in their setting, against developer expectations of a 24% reduction beforehand, indicating that suite performance and in-the-wild productivity can diverge (see [CAP-CODE-001]).

**Open questions.** (1) Whether intermediate-token reasoning reflects faithful internal reasoning (Turpin et al., 2023). (2) How performance on verifiable tasks transfers to unverifiable ones. (3) Whether reliability on long horizons improves mainly with better per-step accuracy or with better error recovery; Kwa et al. (2025) attribute their measured horizon growth to a combination of greater reliability, better adaptation to mistakes, stronger logical reasoning and better tool use, but this attribution is the authors' interpretation. (4) Whether abstraction and generalisation, rather than skill, can be measured without prior knowledge confounds, a concern raised by Chollet (2019) and unresolved.

## 8. Current State of Knowledge
As of 2026-10-15, the following can be stated with the sources cited here. Intermediate reasoning steps improve accuracy on a range of multi-step tasks for sufficiently capable models (Well Supported; Wei et al., 2022). External verification materially helps, while unaided self-correction does not reliably help (Well Supported; Lightman et al., 2023; Huang et al., 2023). Language models used alone have been reported to be weak at autonomous formal planning (Preliminary to Well Supported, since the evaluated models date from 2022-2023 and the benchmark domains are narrow; Valmeekam et al., 2022, 2023). Measured task horizons have grown across model generations on one family of software-oriented suites (Preliminary; a single research group's metric). Accuracy-collapse results for reasoning models are Contested.

This article does not report current leaderboard values for reasoning benchmarks, which change rapidly and are subject to contamination; see `MODEL-001` and the methodology in [CAP-COMPARE-001].

## 9. Future Trajectory & Strategic Outlook
- **Near term (0-3 years)**: Observed pattern, not a forecast: evaluation is moving from fixed-answer reasoning benchmarks to time-referenced and environment-based measures (Kwa et al., 2025; Wijk et al., 2024).
- **Medium and long term**: Kwa et al. (2025) state that if their results generalise to real-world software tasks, extrapolating the trend would imply systems capable of automating many software tasks currently taking humans a month within 5 years. That is the authors' conditional extrapolation, not a claim adopted here; the condition (generalisation beyond the suite) is itself uncertain, as Section 7 shows. This article registers no forecast of its own. A proper forecast would need the full record format in the evidence framework and an operational definition of "automating" a task.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-REASON-001] Reasoning Models and Test-Time Compute](../vol03_technical/TECH-REASON-001_reasoning_models_and_test_time_compute.md): How reasoning models are built.
  - [[TECH-LLM-001] Large Language Models](../vol03_technical/TECH-LLM-001_large_language_models.md): The underlying systems.
- **Related Methods & Technologies**:
  - [[TECH-AGENT-001] Autonomous AI Agents](../vol03_technical/TECH-AGENT-001_autonomous_agents.md): The scaffolding in which long-horizon behaviour occurs.
  - [[CAP-BENCH-001] AI Evaluation, Benchmarking & Contamination](CAP-BENCH-001_ai_evaluation.md): General evaluation methodology and contamination.
  - [[CAP-LIMITS-001] Hallucination, Reliability and Calibration](CAP-LIMITS-001_hallucination_reliability_and_calibration.md): Per-step reliability.
  - [[CAP-CODE-001] Software Engineering Capability and Evaluation](CAP-CODE-001_software_engineering_capability_and_evaluation.md): Software tasks as a horizon testbed.
  - [[CAP-AGENTEVAL-001] Evaluating Autonomous Agents](CAP-AGENTEVAL-001_evaluating_autonomous_agents.md): Environment-based agent evaluation.
  - [[CAP-COMPARE-001] Methodology for Comparing AI Systems](CAP-COMPARE-001_methodology_for_comparing_ai_systems.md): How to compare reasoning claims across models.
- **Advanced Implications & Future Trajectories**:
  - [[AGI-001] Artificial General Intelligence](../vol05_agi/AGI-001_artificial_general_intelligence.md): Reasoning breadth as a criterion.
  - [[AGI-CL-001] Continual Learning and Persistent Memory](../vol05_agi/AGI-CL-001_continual_learning_and_persistent_memory.md): Memory beyond a single context window.
  - [[ASI-AIRD-001] Automated AI Research](../vol06_asi/ASI-AIRD-001_automated_ai_research.md): Research engineering as a capability.
- **Living Reference**:
  - [[MODEL-001] Model Registry](../reference/factbook/MODEL-001_model_registry.md): Current models and their reported properties.

## 11. References & Foundational Literature
1. Becker, J., Rush, N., Barnes, E., & Rein, D. (2025). *Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity*. arXiv:2507.09089. [Randomised trial with 16 developers and 246 tasks reporting a 19% increase in completion time when AI tools were allowed].
2. Chollet, F. (2019). *On the Measure of Intelligence*. arXiv:1911.01547. [Argues that skill at specific tasks does not measure intelligence because priors and training data can be "bought"].
3. Dziri, N., et al. (2023). *Faith and Fate: Limits of Transformers on Compositionality*. arXiv:2305.18654. [Examines transformer performance on compositional tasks as depth increases].
4. Guo, D., et al. (DeepSeek-AI). (2025). *DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning*. Nature, 645, 633–638. https://doi.org/10.1038/s41586-025-09422-z. [Openly documented example of a reasoning model trained with reinforcement learning].
5. Huang, J., et al. (2023). *Large Language Models Cannot Self-Correct Reasoning Yet*. arXiv:2310.01798. [Reports that models do not reliably improve reasoning through self-correction without external feedback].
6. Kambhampati, S., et al. (2024). *LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworks*. ICML 2024 (PMLR 235). arXiv:2402.01817. [Position paper on the role of LLMs with external verifiers in planning].
7. Kwa, T., et al. (2025). *Measuring AI Ability to Complete Long Software Tasks*. NeurIPS 2025. arXiv:2503.14499. [Defines the 50% time-horizon metric and reports its measured growth].
8. Lawsen, A. (2025). *Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity*. arXiv:2506.09250. [Argues that Shojaee et al.'s findings reflect experimental design limitations].
9. Lightman, H., et al. (2023). *Let's Verify Step by Step*. arXiv:2305.20050. [Compares process and outcome supervision for training reasoning models].
10. Russell, S., & Norvig, P. (2020). *Artificial Intelligence: A Modern Approach* (4th ed.). Pearson. ISBN 978-0134610993. [Classical treatment of planning as search over action models].
11. Shojaee, P., et al. (2025). *The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity*. arXiv:2506.06941. [Controllable-puzzle study reporting accuracy collapse and three performance regimes].
12. Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). *Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters*. arXiv:2408.03314. [Studies inference-time compute allocation versus parameter scaling].
13. Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). *Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting*. arXiv:2305.04388. [Shows chain-of-thought explanations can misrepresent the reasons for a prediction].
14. Valmeekam, K., Marquez, M., Olmo, A., Sreedharan, S., & Kambhampati, S. (2022). *PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change*. arXiv:2206.10498. [Planning benchmark from classical domains].
15. Valmeekam, K., Sreedharan, S., Marquez, M., Olmo, A., & Kambhampati, S. (2023). *On the Planning Abilities of Large Language Models (A Critical Investigation with a Proposed Benchmark)*. arXiv:2302.06706. [Reports limited autonomous plan-generation ability of the models evaluated].
16. Wei, J., et al. (2022). *Chain-of-Thought Prompting Elicits Reasoning in Large Language Models*. arXiv:2201.11903. [Shows intermediate-step prompting improves reasoning performance in large models].
17. Wijk, H., et al. (2024). *RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts*. arXiv:2411.15114. [Seven ML research-engineering environments with human-expert baseline data].
18. Yao, S., et al. (2022). *ReAct: Synergizing Reasoning and Acting in Language Models*. arXiv:2210.03629. [Proposes interleaving reasoning traces and actions].

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Chain-of-thought prompting improves multi-step reasoning in large models | Well Supported | Wei et al. (2022) | High |
| Chain-of-thought text can misrepresent the real causes of an answer | Well Supported (demonstrated in controlled settings) | Turpin et al. (2023) | Moderate to High |
| Process supervision outperforms outcome supervision on MATH | Preliminary (single study) | Lightman et al. (2023) | Moderate |
| Unaided self-correction does not reliably improve reasoning | Well Supported | Huang et al. (2023) | Moderate |
| LLMs alone are weak at formal planning; helpful inside verified loops | Preliminary (evaluated models dated 2022-2023); the LLM-Modulo proposal is a position | Valmeekam et al. (2022, 2023); Kambhampati et al. (2024) | Moderate |
| Reasoning models show accuracy collapse at high complexity | Contested | Shojaee et al. (2025); Lawsen (2025) | Low to Moderate |
| 50% time horizon has grown on the measured suite | Preliminary (one research group, one task family) | Kwa et al. (2025) | Moderate |
| Time-horizon trend generalises to real-world software work | Hypothetical / Contested | Kwa et al. (2025); Becker et al. (2025) | Low |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.