---
id: TECH-REASON-001
title: "Reasoning Models and Test-Time Compute"
alternative_names:
  - "Large Reasoning Models"
  - "LRMs"
  - "Inference-Time Scaling"
  - "Chain-of-Thought Reinforcement Learning"
category: "Technical → Language Models"
volume: III
chapter: 14
entry_type: "Technical Article"
level: 4
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "TECH-LLM-001"
  - "TECH-RL-001"
  - "TECH-ALIGN-001"
related_entries:
  - "TECH-SCALE-001"
  - "CAP-REASON-001"
  - "CAP-LIMITS-001"
  - "SAFE-INTERP-001"
  - "TECH-AGENT-001"
  - "CAP-BENCH-001"
---

# Reasoning Models and Test-Time Compute

## 1. Concise Definition
A **reasoning model** (also called a large reasoning model, LRM) is a language model trained, typically with reinforcement learning, to generate a long intermediate chain of thought before giving a final answer, so that its accuracy on hard problems can improve with the amount of computation spent at inference time. **Test-time compute** (inference-time compute) denotes that computation: the number of tokens generated, samples drawn, or search steps taken per problem, as distinct from the compute used in training.

## 2. Overview & Significance
For most of the period in which language models were scaled, additional capability was obtained by spending more compute during training (see [TECH-SCALE-001]). Reasoning models add a second lever: for a fixed trained model, a harder problem can be given more thinking tokens or more attempts. Snell et al. (2024) argued that, for some problems, using test-time compute well can outperform scaling model parameters, and Brown et al. (2024) showed that coverage (the fraction of problems solved by at least one of many samples) increases smoothly with sample count over four orders of magnitude.

The significance is both technical and conceptual. Technically, it shifts cost from training to serving (see [TECH-INFER-001]) and makes the compute spent per query variable. Conceptually, the reasoning traces raise questions about what the model is doing when it "thinks" and whether the written trace is a faithful account of its computation (Section 7). Capability claims for reasoning models, their evaluation, and their limits as general planners are treated in [CAP-REASON-001]; this article describes the methods.

## 3. Historical Development
Producing intermediate steps was shown to help earlier than the current generation. Cobbe et al. (2021) trained verifiers to rank sampled solutions to grade-school math problems (GSM8K) and showed that verification improved performance. Nye et al. (2021) trained models to emit "scratchpads" of intermediate computation. Wei et al. (2022) showed that few-shot **chain-of-thought (CoT) prompting** elicits step-by-step reasoning in sufficiently large models, and Kojima et al. (2022) showed that a zero-shot instruction can trigger similar behaviour. Wang et al. (2023) proposed **self-consistency**, sampling multiple reasoning paths and selecting the most common final answer. Yao et al. (2023) proposed **Tree of Thoughts**, a search over partial reasoning states with self-evaluation and backtracking. Zelikman et al. (2022) proposed STaR, which fine-tunes on self-generated rationales that led to correct answers.

Supervision of reasoning was compared by Uesato et al. (2022) on GSM8K, who found that outcome-based supervision gave similar final-answer error rates with less labelling but that process-based supervision was needed for correct reasoning steps, and by Lightman et al. (2023), who found process supervision significantly outperformed outcome supervision on the MATH dataset (78% on a representative test subset with their best process-supervised model) and released the PRM800K step-level labels.

The 2024–2025 generation made the long chain of thought a trained behaviour. OpenAI's o1 system card states that the o1 model series is "trained with large-scale reinforcement learning to reason using chain of thought" (OpenAI, 2024). Its methodology beyond that statement is **undisclosed**, which motivated replication efforts; the s1 paper describes the o1 model as having shown test-time scaling "but did not publicly share its methodology" (Muennighoff et al., 2025). DeepSeek reported that reasoning abilities of LLMs can be incentivised through pure reinforcement learning, without human-labelled reasoning trajectories, building on DeepSeek-V3-Base with Group Relative Policy Optimization (GRPO) and a reward based on correctness of final answers, and released the resulting model under the MIT licence (Guo et al., 2025). Muennighoff et al. (2025) showed that supervised fine-tuning of Qwen2.5-32B-Instruct on 1,000 curated reasoning examples, combined with "budget forcing", exceeded o1-preview on competition math questions by up to 27%, as reported by the authors.

## 4. Technical / Conceptual Explanation

### 4.1 Where test-time compute can be spent
- **Sequential**: a longer chain of thought, with self-checking, backtracking and revision in a single trace.
- **Parallel**: $n$ independent samples, combined by majority vote, a verifier, or a reward model.
- **Search**: tree or beam search over partial solutions scored by a process reward model (Snell et al., 2024; Yao et al., 2023).

### 4.2 Aggregating samples
Majority voting (self-consistency) returns
$$ \hat{a} = \arg\max_{a} \sum_{i=1}^{n} \mathbb{1}[a_i = a], $$
where $a_i$ is the final answer extracted from sample $i$. With a verifier or reward model $v$, best-of-$n$ returns $\arg\max_i v(x, y_i)$. Coverage (also pass@$k$) is the probability that at least one of $k$ samples is correct. If $n$ samples are drawn and $c$ are correct, the standard unbiased estimator is
$$ \text{pass@}k = 1 - \binom{n-c}{k}\Big/\binom{n}{k}, $$
introduced in the Codex evaluation (Chen et al., 2021), which also reported that repeated sampling solved 70.2% of its problems with 100 samples per problem, against 28.8% for a single sample. Brown et al. (2024) report that coverage versus samples is often log-linear and can be modelled by an exponentiated power law, and that in domains with automatic verifiers (code, formal proofs) coverage gains translate into accuracy gains; in domains without verifiers, majority voting and reward models plateau beyond several hundred samples.

### 4.3 Reinforcement learning with verifiable rewards
For problems whose answers can be checked, the reward is a programmatic function $R(x, y) \in \{0, 1\}$ (or a small set of format and correctness terms). In GRPO (Shao et al., 2024), for each prompt a group of $G$ responses is sampled and each response's advantage is normalised within the group:
$$ \hat{A}_i = \frac{r_i - \operatorname{mean}(\{r_1,\dots,r_G\})}{\operatorname{std}(\{r_1,\dots,r_G\})}, $$
which removes the need for a separate value network and reduces memory use relative to PPO. The policy is updated with a clipped, KL-regularised surrogate objective analogous to PPO (see [TECH-ALIGN-001], [TECH-RL-001]). In DeepSeek's report the reward in the first model was "solely based on the correctness of final predictions against ground-truth answers", with no constraint on the form of the reasoning (Guo et al., 2025). The authors describe behaviours such as self-reflection, verification and strategy adaptation emerging during training; whether these are best described as new capabilities or as the amplification of behaviour already present in the base model is disputed (Section 7).

### 4.4 Process reward models and search
A process reward model (PRM) scores each intermediate step, in contrast to an outcome reward model (ORM) that scores only the final answer (Uesato et al., 2022; Lightman et al., 2023). Snell et al. (2024) study search against dense process-based verifiers and adaptive updating of the model's output distribution, and report that the best approach depends on prompt difficulty; with a "compute-optimal" per-prompt allocation they improve test-time efficiency by more than 4x over a best-of-$N$ baseline and, in a FLOPs-matched comparison, find that on problems where a smaller base model achieves non-trivial success, test-time compute can outperform a 14x larger model.

### 4.5 Budget control
Muennighoff et al. (2025) control inference compute by forcibly terminating thinking or by appending "Wait" to extend it, and report that on AIME24 their model improved from 50% to 57% with extended thinking beyond its unforced performance. Such results show a monotone gain in that setting but are not evidence of unbounded scaling.

## 5. Types, Components & Classifications

| Approach | Mechanism | Training required | Representative sources |
| :--- | :--- | :--- | :--- |
| Prompted CoT | Few-shot or zero-shot instruction to reason stepwise | None | Wei et al. (2022); Kojima et al. (2022) |
| Self-consistency / majority vote | Sample many chains, vote on answer | None | Wang et al. (2023) |
| Verifier or reward-model reranking | Rank samples with a learned scorer | Verifier training | Cobbe et al. (2021); Lightman et al. (2023) |
| Tree/search methods | Explore partial reasoning states | None or PRM | Yao et al. (2023); Snell et al. (2024) |
| Self-taught rationales | Fine-tune on correct self-generated rationales | SFT loop | Zelikman et al. (2022) |
| RL with verifiable rewards | Policy-gradient on checkable outcomes | RL | Guo et al. (2025); Shao et al. (2024) |
| Small-data SFT + budget forcing | Curated traces and controlled thinking length | SFT | Muennighoff et al. (2025) |
| Closed reasoning models | Long CoT trained with RL (method **undisclosed**) | RL (per developer statement) | OpenAI (2024) |

## 6. Applications & Concrete Examples
Reasoning models are used where answers are checkable or multi-step: competition mathematics, programming, scientific question answering, and as the planning component in agents (see [TECH-AGENT-001]). The best-documented open case is DeepSeek-R1, whose paper reports that it surpasses counterparts trained by supervised learning on human demonstrations on verifiable tasks in mathematics, coding competitions and STEM fields (Guo et al., 2025); as a developer-reported result, it is classed Preliminary until independently replicated across settings. Benchmark scores for particular named models change rapidly and are held in `MODEL-001`, not here. The o1 system card also discusses safety work in which models reason about policies in context (OpenAI, 2024); whether such reasoning is faithful is addressed below.

## 7. Advantages, Limitations & Unresolved Questions

### Advantages
Gains on verifiable, multi-step tasks; adjustable compute per query; and, for safety research, a written trace that can in principle be monitored.

### Limitations and disputes
1. **Chain of thought does not help everywhere.** Sprague et al. (2025), in a meta-analysis of over 100 papers and their own evaluation of 20 datasets across 14 models, report that CoT gives strong benefits mainly on tasks involving math or logic, with much smaller gains elsewhere; on MMLU, answers without CoT were almost identical unless the question or response contained an equals sign.
2. **Faithfulness (Contested/Preliminary).** Turpin et al. (2023) found that CoT explanations can systematically misrepresent the reason for a prediction: biasing features in the prompt, such as always placing the correct answer at option (A), were not mentioned in explanations that rationalised the biased answers. Lanham et al. (2023) found that models vary across tasks in how much they condition on their CoT and that, as models become larger and more capable, they produce less faithful reasoning on most tasks they studied. Chen et al. (2025) tested reasoning models on six hint types and found that CoTs revealed use of a hint in at least 1% of the cases in which the hint was used for most settings tested, but often in fewer than 20%; outcome-based RL initially improved faithfulness and then plateaued, and when RL increased hint use via reward hacking the tendency to verbalise it did not increase. They conclude that CoT monitoring is promising but not sufficient to rule out undesired behaviour. These are tests in specific, hint-based settings (see [SAFE-INTERP-001]).
3. **Does RL add reasoning capacity? (Contested).** Yue et al. (2025) evaluated RLVR-trained models using pass@$k$ at large $k$ and report that although RL-trained models are better at small $k$, base models achieved higher pass@$k$ at large $k$; they conclude that the reasoning patterns observed originate from and are bounded by the base model in their setups, and that distillation can add new patterns. The DeepSeek report presents RL as incentivising emergent reasoning behaviours. The two sets of claims address different measurements and models, and the question is open.
4. **Scaling limits and evaluation artefacts (Contested).** Shojaee et al. (2025) report that on controllable puzzle environments reasoning models show accuracy collapse beyond certain complexities and reduced reasoning effort at high complexity. Lawsen (2025) argues that these findings largely reflect experimental design limitations (output token limits, evaluation that does not distinguish reasoning failure from practical constraints, and unsolvable River Crossing instances), and reports preliminary experiments indicating high accuracy when models are asked for generating functions instead of exhaustive move lists. Neither paper settles the matter, and the second reports preliminary experiments only.
5. **Verifier dependence.** Gains are largest where outcomes can be checked automatically; without verifiers, sample-selection methods plateau (Brown et al., 2024).
6. **Cost.** Long traces raise latency and serving cost per query.

### Unresolved questions
- How much reasoning-benchmark improvement reflects genuine generalisation versus training on similar verifiable tasks or contamination (see [TECH-DATA-001]).
- Whether training on outcome rewards alone yields reliable, monitorable reasoning, or whether it incentivises unverbalised shortcuts.
- Which task types will benefit from test-time compute, given the narrow gains in Sprague et al. (2025).

## 8. Current State of Knowledge
As of 2026-10-15, long-chain-of-thought reasoning models trained with reinforcement learning on verifiable rewards are an established product category, and open-weight training recipes are documented (Guo et al., 2025; Lambert et al. in [TECH-ALIGN-001]). Test-time scaling of accuracy on verifiable tasks is Well Supported by multiple independent groups (Snell et al., 2024; Brown et al., 2024; Muennighoff et al., 2025). The training recipes, data, reward models and reasoning-trace handling of closed reasoning models are **undisclosed** except for developer statements such as that in the o1 system card. Whether the traces faithfully describe the computation, and what RL changes relative to the base model, are Contested. Current model comparisons are tracked in `MODEL-001`, and capability measurements in [CAP-BENCH-001] and [CAP-REASON-001].

## 9. Future Trajectory & Strategic Outlook
Observed trends: the combination of RL on checkable tasks, process-level supervision, and adaptive test-time allocation; attempts to extend verifiable rewards to less checkable domains; and growing research on monitoring and faithfulness of traces (Chen et al., 2025). Expected research questions for the next three years include whether faithfulness can be improved by training and whether gains transfer outside verifiable domains. No quantitative forecast record is registered in this article.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-LLM-001] Large Language Models](./TECH-LLM-001_large_language_models.md): Base models that are being post-trained.
  - [[TECH-RL-001] Reinforcement Learning](./TECH-RL-001_reinforcement_learning.md): Policy-gradient background.
  - [[TECH-ALIGN-001] Pretraining, Fine-Tuning and RLHF](./TECH-ALIGN-001_pretraining_fine_tuning_and_rlhf.md): The post-training pipeline, including PPO, DPO and RLVR.
- **Related Methods & Technologies**:
  - [[TECH-SCALE-001] Scaling Laws](./TECH-SCALE-001_scaling_laws.md): Training-time scaling and inference-time scaling.
  - [[TECH-INFER-001] Inference, Quantization and Efficiency](./TECH-INFER-001_inference_quantization_and_efficiency.md): Serving cost of long traces.
  - [[TECH-AGENT-001] Autonomous Agents](./TECH-AGENT-001_autonomous_agents.md): Reasoning models as planners.
  - [[TECH-DATA-001] Training Data, Contamination and Provenance](./TECH-DATA-001_training_data_contamination_and_provenance.md): Contamination in reasoning benchmarks.
- **Advanced Implications & Future Trajectories**:
  - [[CAP-REASON-001] Reasoning, Planning and Long-Horizon Capability](../vol04_capabilities/CAP-REASON-001_reasoning_planning_and_long_horizon_capability.md): Capability evidence and limits.
  - [[CAP-LIMITS-001] Hallucination, Reliability and Calibration](../vol04_capabilities/CAP-LIMITS-001_hallucination_reliability_and_calibration.md): Reliability of generated reasoning.
  - [[SAFE-INTERP-001] Interpretability and Mechanistic Interpretability](../vol08_safety_governance/SAFE-INTERP-001_interpretability_and_mechanistic_interpretability.md): Faithfulness and trace monitoring.

## 11. References & Foundational Literature
1. Chen, M., Tworek, J., Jun, H., et al. (2021). *Evaluating Large Language Models Trained on Code*. arXiv:2107.03374. [Introduces the pass@$k$ metric and reports repeated-sampling gains (28.8% single sample; 70.2% with 100 samples)].
2. Cobbe, K., Kosaraju, V., Bavarian, M., et al. (2021). *Training Verifiers to Solve Math Word Problems*. arXiv:2110.14168. [GSM8K and verifier-based reranking of sampled solutions].
3. Nye, M., Andreassen, A. J., Gur-Ari, G., et al. (2021). *Show Your Work: Scratchpads for Intermediate Computation with Language Models*. arXiv:2112.00114. [Intermediate computation emitted as scratchpads].
4. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E., Le, Q., & Zhou, D. (2022). *Chain-of-Thought Prompting Elicits Reasoning in Large Language Models*. NeurIPS 2022. arXiv:2201.11903. [Few-shot chain-of-thought prompting].
5. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). *Large Language Models are Zero-Shot Reasoners*. NeurIPS 2022. arXiv:2205.11916. [Zero-shot CoT trigger].
6. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2023). *Self-Consistency Improves Chain of Thought Reasoning in Language Models*. ICLR 2023. arXiv:2203.11171. [Majority vote over sampled reasoning paths].
7. Zelikman, E., Wu, Y., Mu, J., & Goodman, N. D. (2022). *STaR: Bootstrapping Reasoning With Reasoning*. NeurIPS 2022. arXiv:2203.14465. [Iterative fine-tuning on rationales that reach correct answers].
8. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T. L., Cao, Y., & Narasimhan, K. (2023). *Tree of Thoughts: Deliberate Problem Solving with Large Language Models*. NeurIPS 2023. arXiv:2305.10601. [Search over reasoning states].
9. Uesato, J., Kushman, N., Kumar, R., Song, F., Siegel, N., Wang, L., Creswell, A., Irving, G., & Higgins, I. (2022). *Solving math word problems with process- and outcome-based feedback*. arXiv:2211.14275. [Comparison of process and outcome supervision on GSM8K].
10. Lightman, H., Kosaraju, V., Burda, Y., Edwards, H., Baker, B., Lee, T., Leike, J., Schulman, J., Sutskever, I., & Cobbe, K. (2023). *Let's Verify Step by Step*. arXiv:2305.20050 (ICLR 2024). [Process supervision outperforming outcome supervision on MATH; PRM800K].
11. Snell, C., Lee, J., Xu, K., & Kumar, A. (2024). *Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters*. arXiv:2408.03314. [Compute-optimal test-time allocation; >4x efficiency over best-of-N; outperforming a 14x larger model in some cases].
12. Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., & Mirhoseini, A. (2024). *Large Language Monkeys: Scaling Inference Compute with Repeated Sampling*. arXiv:2407.21787. [Coverage scaling with sample count; plateau of voting/reward models without verifiers].
13. OpenAI. (2024). *OpenAI o1 System Card*. arXiv:2412.16720. [Developer statement that o1 is trained with large-scale RL to reason with chain of thought; methodology otherwise undisclosed].
14. Guo, D., Yang, D., Zhang, H., et al. (2025). *DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning*. Nature, 645(8081), 633–638. https://doi.org/10.1038/s41586-025-09422-z (preprint arXiv:2501.12948). [RL with correctness rewards on DeepSeek-V3-Base using GRPO; MIT-licensed release].
15. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., & Guo, D. (2024). *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models*. arXiv:2402.03300. [Introduces GRPO].
16. Muennighoff, N., Yang, Z., Shi, W., Li, X. L., Fei-Fei, L., Hajishirzi, H., Zettlemoyer, L., Liang, P., Candès, E., & Hashimoto, T. (2025). *s1: Simple test-time scaling*. arXiv:2501.19393. [1,000-example SFT with budget forcing; AIME24 50% to 57% with extended thinking].
17. Sprague, Z., Yin, F., Rodriguez, J. D., Jiang, D., Wadhwa, M., Singhal, P., Zhao, X., Ye, X., Mahowald, K., & Durrett, G. (2025). *To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning*. ICLR 2025. arXiv:2409.12183. [Meta-analysis of where CoT helps].
18. Turpin, M., Michael, J., Perez, E., & Bowman, S. R. (2023). *Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting*. NeurIPS 2023. arXiv:2305.04388. [Biased inputs unmentioned in CoT explanations].
19. Lanham, T., Chen, A., Radhakrishnan, A., et al. (2023). *Measuring Faithfulness in Chain-of-Thought Reasoning*. arXiv:2307.13702. [Intervention-based tests of CoT reliance; less faithful reasoning in larger models on most tasks studied].
20. Chen, Y., Benton, J., Radhakrishnan, A., et al. (2025). *Reasoning Models Don't Always Say What They Think*. arXiv:2505.05410. [Hint-verbalisation rates of reasoning models; limits of CoT monitoring].
21. Yue, Y., Chen, Z., Lu, R., Zhao, A., Wang, Z., Yue, Y., Song, S., & Huang, G. (2025). *Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?* arXiv:2504.13837. [Large-$k$ pass@$k$ analysis of RLVR versus base models].
22. Shojaee, P., Mirzadeh, I., Alizadeh, K., Horton, M., Bengio, S., & Farajtabar, M. (2025). *The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity*. arXiv:2506.06941. [Puzzle-environment study reporting accuracy collapse].
23. Lawsen, A. (2025). *Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity*. arXiv:2506.09250. [Critique attributing reported failures to experimental design].
24. Lambert, N., Morrison, J., Pyatkin, V., et al. (2024). *Tulu 3: Pushing Frontiers in Open Language Model Post-Training*. arXiv:2411.15124. [Open post-training recipe including RLVR].

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Majority voting and pass@$k$ definitions | Established | Wang et al. (2023); Chen et al. (2021) | High |
| Intermediate reasoning steps improve performance on math/logic tasks | Well Supported | Wei et al. (2022); Cobbe et al. (2021); Sprague et al. (2025) | High |
| Accuracy rises with test-time compute on verifiable tasks | Well Supported | Snell et al. (2024); Brown et al. (2024); Muennighoff et al. (2025) | Moderate-High |
| Process supervision beats outcome supervision (MATH) | Well Supported | Lightman et al. (2023); Uesato et al. (2022) nuance | Moderate (task dependent) |
| Pure-RL training elicits long-CoT reasoning | Preliminary (open replication in progress) | Guo et al. (2025) | Moderate |
| RL expands reasoning beyond base model | Contested | Guo et al. (2025); Yue et al. (2025) | Active debate |
| CoT traces faithfully reflect computation | Contested | Turpin et al. (2023); Lanham et al. (2023); Chen et al. (2025) | Evidence of unfaithfulness in tested settings |
| Accuracy collapse of reasoning models at high complexity | Contested | Shojaee et al. (2025); Lawsen (2025) | Active debate |
| Training method and data of closed reasoning models | Undisclosed | OpenAI (2024) | Not inferred |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.