---
id: ASI-AIRD-001
title: "Automated AI Research"
alternative_names:
  - "AI R&D Automation"
  - "Automated Machine Learning Research"
  - "AI Scientist Systems"
category: "Advanced Intelligence → Superintelligence"
volume: VI
chapter: 32
entry_type: "Technical Article"
level: 4
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "ASI-EXP-001"
  - "TECH-AGENT-001"
related_entries:
  - "ASI-PATH-001"
  - "ASI-001"
  - "AGI-TEST-001"
  - "APP-SCI-001"
  - "SAFE-EVAL-001"
---

# Automated AI Research

## 1. Concise Definition
**Automated AI research** (AI R&D automation) is the use of AI systems to perform some or all of the activities of AI research and development: generating ideas, writing and running experiments, analysing results, engineering software and writing up findings. It is of interest because, if AI systems could improve AI systems, progress might accelerate; whether and how far that could happen is Hypothetical or Contested, and the evidence so far concerns partial automation in constrained settings.

## 2. Overview & Significance
The intelligence-explosion argument ([ASI-EXP-001](../vol06_asi/ASI-EXP-001_intelligence_explosion.md)) rests on a feedback loop in which AI contributes to the research that produces better AI. Automated AI research is the empirical counterpart of that argument: it asks how much of the research process current systems can do, how that compares with human researchers, and what limits remain. The topic is also relevant to safety policy; the RE-Bench authors state that AI R&D automation is a key capability for safety policies to anticipate, while noting that few realistic benchmarks compare AI systems with human experts (Wijk et al., 2024).

This article distinguishes four things that are often run together.

1. **Automating engineering tasks inside AI R&D**, such as writing training code or optimising kernels.
2. **Automating the research loop**, from hypothesis to paper, in a limited domain.
3. **Automating algorithm discovery**, where a system searches for new algorithms against an automatic evaluator.
4. **Closing the loop**, in which the products of automated research improve the system that does the research. No source reviewed shows this at scale.

## 3. Historical Development
Early proposals for self-improving machines (Good, 1966) were theoretical. Automated machine-learning methods, which search over architectures or hyperparameters, preceded the current wave; they automate narrow parts of the process and are not the focus here. Hutter (2012) analysed under what conditions an intelligence explosion would occur, showing that conclusions depend on assumptions about returns to intelligence.

From 2023 onwards, benchmarks and systems appeared that target research tasks directly. MLAgentBench (Huang et al., 2023) evaluates language-model agents on machine-learning experimentation tasks. MLE-bench (Chan et al., 2024) uses Kaggle competitions. RE-Bench (Wijk et al., 2024) compares agents with human experts on research-engineering environments. System-level work includes The AI Scientist (Lu et al., 2024), which the authors describe as a framework in which language models carry out the full research cycle, with a stated cost of under USD 15 per paper in their experiments, and its successor (Yamada et al., 2025). Algorithm-discovery systems include AlphaEvolve (Novikov et al., 2025), and AI co-scientist (Gottweis et al., 2025) targets hypothesis generation in biomedicine.

## 4. Technical / Conceptual Explanation

### 4.1 A model of the feedback loop
A simple stylised model illustrates why assumptions matter. Let $A(t)$ be the level of AI capability and suppose the rate of algorithmic and software progress depends on the research effort supplied by AI systems:
$$\frac{dA}{dt} = f\big(R(A), \; C\big),$$
where $R(A)$ is the effective research labour supplied (increasing in $A$) and $C$ is the other inputs, such as compute and experimental capacity, that research needs. If $C$ is not limiting and returns to research labour are high enough, progress accelerates; if $C$ is limiting, or if ideas become harder to find, the feedback is damped. Bloom et al. (2020) document that research productivity has declined in several fields in the data they study, which is an argument that returns to research effort diminish. Erdil and Besiroglu (2023) review arguments for explosive growth from AI automation and weigh counterarguments including bottlenecks. Ho et al. (2024) estimate, for language models on two benchmarks from 2012 to 2023, that the compute needed to reach a fixed performance halved approximately every 8 months (95% confidence interval roughly 5 to 14 months), and conclude that increased compute contributed more to performance gains than algorithmic innovations did. Together, this suggests that compute is a major input, which matters for how far software-only research automation could go; the question is not settled by these sources.

This model is a schematic, not a measured system; $f$ and $R$ are not known.

### 4.2 What the benchmarks measure
**RE-Bench.** Seven open-ended research-engineering environments with data from 71 8-hour attempts by 61 distinct human experts; 82% of expert attempts achieved a non-zero score and 24% matched or exceeded the strong reference solutions. The best AI agents achieve a score 4x higher than human experts when both are given a total time budget of 2 hours per environment, whereas humans achieve 2x the score of the top AI agent when both are given 32 total hours. The authors also report that agents can generate and test solutions over ten times faster than humans at much lower cost (Wijk et al., 2024). The result describes seven environments and the models tested at that date.

**MLE-bench.** 75 ML-engineering competitions from Kaggle with human baselines from public leaderboards. The best-performing setup at the time, OpenAI's o1-preview with AIDE scaffolding, achieved at least the level of a Kaggle bronze medal in 16.9% of competitions (Chan et al., 2024).

**Time horizons.** Kwa et al. (2025) measure the length of software tasks completed with 50% success. They report a 50% time horizon of around 50 minutes for models such as Claude 3.7 Sonnet and a doubling time of approximately seven months since 2019. The paper includes RE-Bench tasks among its task sources. The metric covers software tasks and a specified success rate.

**Interpretation.** These instruments show that current agents can do parts of research engineering quickly and cheaply and that the advantage is greatest with short time budgets. They do not show that agents can conduct the open-ended, long-horizon work of choosing research directions, which none of the benchmarks measures directly.

### 4.3 End-to-end research systems
The AI Scientist (Lu et al., 2024) generates ideas, writes and runs code, visualises results, drafts a paper and reviews it with a simulated reviewer; the authors tested it in diffusion modelling, transformer language modelling and learning dynamics, and report that their automated reviewer performs close to human reviewers and that some generated papers pass the acceptance threshold of a top machine-learning conference by the automated reviewer's standard. The reviewer is itself an automated system built by the authors, so this is not independent evidence of publishable quality. The AI Scientist-v2 (Yamada et al., 2025) uses agentic tree search and a vision-language feedback loop; the authors submitted three fully autonomous papers to an ICLR workshop, one of which scored above the average human acceptance threshold. A workshop acceptance threshold is lower than that of a main conference, and the result concerns a single submission batch. Both are Preliminary: results come from the developing group and a small number of cases.

### 4.4 Algorithm discovery
AlphaEvolve (Novikov et al., 2025) is an evolutionary coding agent in which language models rewrite code guided by automatic evaluators. The authors apply it to computing infrastructure, open scientific problems and algorithm design, and report a procedure for multiplying two 4x4 complex-valued matrices using 48 scalar multiplications, which they describe as the first improvement after 56 years over Strassen's algorithm in this setting. These systems work where an automatic evaluator can score candidate solutions, which is a narrower condition than most research satisfies. The AI co-scientist (Gottweis et al., 2025) is an example of a multi-agent system for hypothesis generation; its hypotheses are tested separately from the generation system. [VERIFICATION NEEDED: how the paper validates hypotheses was not checked beyond the title and listing] See [APP-SCI-001](../vol07_applications/APP-SCI-001_ai_in_science.md).

## 5. Types, Components & Classifications

| Level of automation | Description | Example | Evidence status |
| :--- | :--- | :--- | :--- |
| Engineering sub-tasks | Code, debugging, experiment setup | RE-Bench; MLE-bench; time-horizon metric | Preliminary |
| Research loop in a narrow setting | Idea, experiment and write-up, with automated review | The AI Scientist; AI Scientist-v2 | Preliminary |
| Search against an automatic evaluator | Evolutionary or tree search for algorithms | AlphaEvolve | Preliminary |
| Hypothesis generation with external validation | Proposals tested by human-run experiments | AI co-scientist | Preliminary |
| Full automation of frontier AI research | Systems do most AI research work | None documented | Hypothetical |
| Recursive self-improvement | Output of research improves the researcher at scale | None documented | Hypothetical / Contested |

The components of a research-automation system are typically a language-model core, an agent scaffold (planning, tool use, code execution), an evaluation harness and a memory or archive of prior attempts. Their general technical background is in [TECH-AGENT-001](../vol03_technical/TECH-AGENT-001_autonomous_agents.md).

## 6. Applications & Concrete Examples
- **Research engineering assistance.** Agents complete portions of software and ML-engineering tasks, as in RE-Bench and MLE-bench, which are evaluation settings and not measures of deployed productivity.
- **Algorithm and kernel discovery.** AlphaEvolve is described by its authors as applied to Google's computing infrastructure and to mathematical problems; the matrix-multiplication result is one reported example (Novikov et al., 2025).
- **Paper-generation pipelines.** The AI Scientist and its successor are laboratory systems (Lu et al., 2024; Yamada et al., 2025).
- **Scientific hypothesis generation.** The AI co-scientist is a laboratory system for biomedical hypotheses (Gottweis et al., 2025).

Deployment inside AI developers is not documented in the sources reviewed, and the extent of internal use of AI in AI development at specific companies is, for the purposes of this article, undisclosed.

## 7. Advantages, Limitations & Unresolved Questions
**Potential advantages.** Speed and cost (agents generate and test solutions more than ten times faster than humans at much lower cost in RE-Bench, per the authors), parallelism and availability.

**Limitations.**
- *Short-horizon strength, long-horizon weakness.* Agents lead at 2-hour budgets and trail at 32 hours in RE-Bench (Wijk et al., 2024).
- *Benchmarks differ from frontier research.* Environments are well specified and scored; frontier research involves choosing problems and judging significance.
- *Evaluator dependence.* Systems such as AlphaEvolve need automatic evaluators; results in domains without them are not documented in the sources reviewed.
- *Self-grading.* End-to-end systems partly rely on automated reviewers built by the same group.
- *Compute and experiment bottlenecks.* Progress depends on compute for experiments (Ho et al., 2024; Erdil & Besiroglu, 2023). A system that proposes ideas faster than they can be tested faces a limit.
- *Diminishing returns.* Bloom et al. (2020) argue ideas are getting harder to find in the data they study; whether this applies to AI research is uncertain.
- *Safety implications.* Rapid capability growth would shorten the time available for evaluation and oversight; see [SAFE-EVAL-001](../vol08_safety_governance/SAFE-EVAL-001_dangerous_capability_and_alignment_evaluations.md) and [SAFE-OVERSIGHT-001](../vol08_safety_governance/SAFE-OVERSIGHT-001_scalable_oversight_and_corrigibility.md).

**Open questions.**
1. How much of the AI research pipeline is bottlenecked by compute rather than by ideas or labour?
2. Can agents do research that requires judgement over weeks to months?
3. Is there a threshold at which the feedback becomes self-sustaining, or does it saturate? Contested.
4. How should automation of AI R&D be measured in ways that are hard to game?

## 8. Current State of Knowledge
As of 2026-10-15, the sources reviewed show partial automation of research engineering and demonstrations of end-to-end research systems in narrow settings. They do not show closed-loop recursive self-improvement, and the claim that recursive self-improvement leads to an intelligence explosion is Hypothetical and Contested (see [ASI-EXP-001](../vol06_asi/ASI-EXP-001_intelligence_explosion.md)). Benchmark values quoted above date from 2024 and 2025 papers and apply to the models tested then; later systems may differ, and current values should be taken from the Living Reference (`MODEL-001`, `FACT-001`) rather than from this article. Evaluations from developers and independent groups use different protocols, so comparisons across them require care.

## 9. Future Trajectory & Strategic Outlook
Observed trends (not forecasts): the benchmark results above show increasing capability on research-engineering tasks between the cited papers' model generations, and the time-horizon metric reports a doubling time of about seven months in its data (Kwa et al., 2025). Extrapolating a trend into the future is a forecast and is registered below.

**Near term (0 to 3 years).** Further benchmarks of AI R&D capability and agent evaluations; continued system-level demonstrations.
**Medium term (3 to 10 years).** The central uncertainty is whether automation of research engineering extends to research direction-setting and whether compute limits it. The sources reviewed do not resolve this.
**Long term (10+ years).** Scenarios in which AI research is largely automated are Hypothetical; see [ASI-PATH-001](../vol06_asi/ASI-PATH-001_pathways_to_superintelligence.md).

### Registered forecast

| Field | Content |
| :--- | :--- |
| **Forecast ID** | `FCST-AIRD-001` |
| **Forecast Statement** | By 2030-12-31, an independent third-party evaluation will report an AI agent system exceeding the median human-expert baseline score at a 32-hour per-environment budget on a RE-Bench-style suite of AI research-engineering environments. |
| **Target Event** | AI agents overtaking human experts at long (32-hour) budgets on AI research-engineering tasks, a point at which the reversal reported by Wijk et al. (2024) no longer holds. |
| **Operational Definition** | The evaluation uses at least 7 open-ended research-engineering environments, a documented human-expert baseline with attempts of 8 hours each or longer, and an agent given a total budget of 32 hours per environment. "Exceeds" means a higher mean normalised score with a stated confidence interval not overlapping the human baseline. The evaluation is published by a party not employed by the developer of the system. |
| **Time Horizon** | By 2030-12-31 |
| **Assumptions** | Continued frontier model development; continued public or independent access for evaluators; evaluation environments are not saturated by contamination; suite remains comparable. |
| **Confidence Level** | Medium (roughly 40 to 60%). This is an editorial judgement, not derived from a quantitative model; it takes into account the trend in Kwa et al. (2025) and the unsolved long-horizon limitation in Wijk et al. (2024). |
| **Alternative Scenarios** | Long-horizon performance stays below experts because of reliability and error-recovery limits (a plateau in the time-horizon trend); or the result is reached but only by a developer-run evaluation, which would not resolve this forecast; or the suite is replaced and no comparable evaluation is published. |
| **Review Schedule** | Annual review each October; final evaluation January 2031. |
| **Status** | `Active` |

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[ASI-EXP-001] The Intelligence Explosion Hypothesis](../vol06_asi/ASI-EXP-001_intelligence_explosion.md): The hypothesis that automated research motivates.
  - [[TECH-AGENT-001] Autonomous AI Agents](../vol03_technical/TECH-AGENT-001_autonomous_agents.md): Agent architectures used in research automation.
  - [[ASI-001] Artificial Superintelligence](../vol06_asi/ASI-001_artificial_superintelligence.md): Overview of ASI.
- **Related Methods & Technologies**:
  - [[AGI-TEST-001] Testing Claims of General Intelligence](../vol05_agi/AGI-TEST-001_testing_claims_of_general_intelligence.md): Validity issues for agentic benchmarks.
  - [[CAP-AGENTEVAL-001] Evaluating Autonomous Agents](../vol04_capabilities/CAP-AGENTEVAL-001_evaluating_autonomous_agents.md): Agent evaluation methods.
  - [[APP-SCI-001] AI in Science](../vol07_applications/APP-SCI-001_ai_in_science.md): Scientific applications.
  - [[ECON-COMP-001] The Economics of Compute & Scaling](../vol09_economy/ECON-COMP-001_compute_economics.md): Compute as a limiting input.
- **Advanced Implications & Future Trajectories**:
  - [[ASI-PATH-001] Pathways to Superintelligence](../vol06_asi/ASI-PATH-001_pathways_to_superintelligence.md): Where automated research sits among pathways.
  - [[SAFE-EVAL-001] Dangerous Capability and Alignment Evaluations](../vol08_safety_governance/SAFE-EVAL-001_dangerous_capability_and_alignment_evaluations.md): Evaluating AI R&D capability as a risk indicator.
  - [[SAFE-OVERSIGHT-001] Scalable Oversight and Corrigibility](../vol08_safety_governance/SAFE-OVERSIGHT-001_scalable_oversight_and_corrigibility.md): Oversight under faster capability growth.

## 11. References & Foundational Literature
1. Bloom, N., Jones, C. I., Van Reenen, J., & Webb, M. (2020). *Are Ideas Getting Harder to Find?* American Economic Review, 110(4), 1104–1144. https://doi.org/10.1257/aer.20180338. [Evidence that research productivity has declined in the fields studied.]
2. Chan, J. S., Chowdhury, N., Jaffe, O., Aung, J., Sherburn, D., Mays, E., Starace, G., Liu, K., Maksin, L., Patwardhan, T., Weng, L., & Mądry, A. (2024). *MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering*. arXiv preprint. arXiv:2410.07095. [75 Kaggle competitions; 16.9% bronze-level figure.]
3. Erdil, E., & Besiroglu, T. (2023). *Explosive growth from AI automation: A review of the arguments*. arXiv preprint. arXiv:2309.11690. [Arguments for and against explosive growth from automation.]
4. Good, I. J. (1966). *Speculations Concerning the First Ultraintelligent Machine*. Advances in Computers, 6, 31–88. https://doi.org/10.1016/S0065-2458(08)60418-0. [Original self-improvement argument.]
5. Gottweis, J., Weng, W.-H., Daryin, A., Tu, T., Sirkovic, P., Myaskovsky, A., et al. (2025). *Accelerating scientific discovery with Co-Scientist* (AI co-scientist). arXiv preprint. arXiv:2502.18864 (51 authors; a Nature DOI, 10.1038/s41586-026-10644-y, is listed on the arXiv record). [Multi-agent system for scientific hypothesis generation.]
6. Ho, A., Besiroglu, T., Erdil, E., Owen, D., Rahman, R., Guo, Z. C., Atkinson, D., Thompson, N., & Sevilla, J. (2024). *Algorithmic progress in language models*. arXiv preprint. arXiv:2403.05812. [Compute-halving time of about 8 months (95% CI roughly 5 to 14 months) on two benchmarks, 2012 to 2023.]
7. Huang, Q., Vora, J., Liang, P., & Leskovec, J. (2023). *MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation*. arXiv preprint. arXiv:2310.03302. [Benchmark of language-model agents on ML experimentation tasks.]
8. Hutter, M. (2012). *Can Intelligence Explode?* Journal of Consciousness Studies, 19(1–2), 143–166. arXiv:1202.6177. [Conditions for and against an intelligence explosion.]
9. Kwa, T., West, B., Becker, J., Deng, A., Garcia, K., Hasin, M., et al. (2025). *Measuring AI Ability to Complete Long Software Tasks*. NeurIPS 2025; arXiv:2503.14499. [50% time horizon around 50 minutes for Claude 3.7 Sonnet; doubling about every seven months since 2019 in their data.]
10. Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., & Ha, D. (2024). *The AI Scientist: Towards Fully Automated Open-Ended Scientific Discovery*. arXiv preprint. arXiv:2408.06292. [End-to-end research framework; under USD 15 per paper in the authors' experiments.]
11. Novikov, A., Vũ, N., Eisenberger, M., Dupont, E., Huang, P.-S., Wagner, A. Z., et al. (2025). *AlphaEvolve: A coding agent for scientific and algorithmic discovery*. arXiv preprint. arXiv:2506.13131 (18 authors). [Evolutionary coding agent and the 4x4 complex matrix multiplication result.]
12. Wijk, H., Lin, T., Becker, J., Jawhar, S., Parikh, N., Broadley, T., et al. (2024). *RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts*. arXiv preprint. arXiv:2411.15114 (23 authors). [Seven environments; human-expert baselines; time-budget crossover.]
13. Yamada, Y., Lange, R. T., Lu, C., Hu, S., Lu, C., Foerster, J., Clune, J., & Ha, D. (2025). *The AI Scientist-v2: Workshop-Level Automated Scientific Discovery via Agentic Tree Search*. arXiv preprint. arXiv:2504.08066. [First entirely AI-generated peer-review-accepted workshop paper, per the authors.]

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Agents lead experts at 2-hour budgets and trail at 32 hours on RE-Bench | Preliminary (single study) | Wijk et al. (2024) | Moderate |
| Best setup reached bronze-level in 16.9% of 75 competitions (MLE-bench) | Preliminary (date-specific) | Chan et al. (2024) | Moderate |
| AI Scientist systems produce papers judged acceptable by an automated reviewer | Preliminary (developer-built reviewer) | Lu et al. (2024) | Low to moderate |
| One AI-generated paper passed workshop-level peer review | Preliminary (n = 1 of 3 submissions) | Yamada et al. (2025) | Low to moderate |
| AlphaEvolve found a 48-multiplication procedure for 4x4 complex matrices | Preliminary (authors' claim) | Novikov et al. (2025) | Moderate |
| Compute needed for fixed performance halved about every 8 months | Preliminary | Ho et al. (2024) | Moderate |
| Research productivity has declined in studied fields | Well Supported (economic data, specific fields) | Bloom et al. (2020) | Moderate |
| Automated research will produce an intelligence explosion | Hypothetical / Contested | Good (1966); Hutter (2012); Erdil & Besiroglu (2023) | Low; assumption-dependent |
| `FCST-AIRD-001` | Forecast | Wijk et al. (2024); Kwa et al. (2025) | Medium (editorial judgement) |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.