---
id: TECH-RL-001
title: "Reinforcement Learning"
alternative_names:
  - "RL"
  - "Deep Reinforcement Learning"
  - "DRL"
category: "Technical → Paradigms"
volume: III
chapter: 17
entry_type: "Technical Reference"
level: 3
status: "Draft"
last_reviewed: 2026-10-15
version: "0.2.0"
prerequisites:
  - "TECH-ML-001"
  - "TECH-DL-001"
  - "Probability Theory"
related_entries:
  - "TECH-LLM-001"
  - "SAFE-ALIGN-001"
  - "AGI-PATH-001"
---

# Reinforcement Learning

## 1. Concise Definition
**Reinforcement Learning (RL)** is a subfield of machine learning concerned with how intelligent agents ought to take actions in an environment in order to maximize the mathematical expectation of cumulative future reward. Unlike supervised learning, which relies on a fixed dataset of labeled input-output pairs, RL requires an agent to discover optimal behavior through active exploration, trial-and-error interactions, and delayed evaluative feedback (Sutton & Barto, 2018).

## 2. Overview & Significance
Reinforcement learning provides the formal mathematical framework for sequential decision-making under uncertainty. It is uniquely suited for tasks where the "correct" action is not known a priori, but a success criterion (a goal) can be defined.

Its scientific and industrial significance stems from its ability to:
1. **Handle Non-Differentiable Environments**: RL can optimize policies in settings whose dynamics or reward are not differentiable with respect to the agent's actions (e.g., board games, robotics, server routing). Policy-gradient methods still use gradient ascent on the policy parameters, via estimators that do not require differentiating through the environment.
2. **Enable Superhuman Discovery**: By learning through self-play, RL agents (like AlphaZero) can discover novel strategies that surpass human intuition, unconstrained by the limits of human demonstration datasets.
3. **Align Foundation Models**: RL forms the algorithmic backbone of Reinforcement Learning from Human Feedback (RLHF), the primary method used to tune raw generative language models into helpful, safe conversational agents (Ouyang et al., 2022).

## 3. Historical Development
The mathematical foundations of RL originate in the 1950s with Richard Bellman's formulation of **Dynamic Programming** and the Bellman equations, which underlie the solution of Markov Decision Processes (MDPs) (see Sutton & Barto, 2018). Parallel developments in animal psychology concerning classical and operant conditioning provided a biological analogue.

In 1988, Richard Sutton formalized **Temporal-Difference (TD) learning**, combining the bootstrapping of dynamic programming with Monte Carlo sampling (Sutton, 1988). **Q-learning**, an off-policy algorithm, was developed by Christopher Watkins and analyzed with Peter Dayan (Watkins & Dayan, 1992).

However, classical RL struggled with high-dimensional state spaces (the "curse of dimensionality"). The modern era of **Deep Reinforcement Learning (DRL)** is commonly dated to the **Deep Q-Network (DQN)**, which combined convolutional neural networks with Q-learning to play Atari 2600 games directly from raw pixel inputs (workshop preprint: Mnih et al., 2013; Nature version: Mnih et al., 2015). In 2016, AlphaGo defeated the European champion Fan Hui 5-0 (Silver et al., 2016) and, in March 2016, the professional Lee Sedol 4-1 (Google DeepMind, AlphaGo page; the Lee Sedol match is not reported in the Nature paper). Policy-gradient work continued with Proximal Policy Optimization (PPO; Schulman et al., 2017).

## 4. Technical / Conceptual Explanation

### 4.1 The Agent-Environment Framework (MDP)
RL is formally modeled as a **Markov Decision Process (MDP)**, defined by a tuple $\langle \mathcal{S}, \mathcal{A}, \mathcal{P}, \mathcal{R}, \gamma \rangle$:
- **States ($\mathcal{S}$)**: The set of environment states $s_t$.
- **Actions ($\mathcal{A}$)**: The set of actions $a_t$ available to the agent.
- **Transition Dynamics ($\mathcal{P}$)**: The probability $P(s_{t+1} \mid s_t, a_t)$ of moving to state $s_{t+1}$ given action $a_t$.
- **Reward ($\mathcal{R}$)**: The immediate scalar reward $R_t = R(s_t, a_t)$.
- **Discount Factor ($\gamma \in [0, 1]$)**: Determines the present value of future rewards.

The objective is to find a **policy** $\pi(a \mid s)$ (a mapping from states to probabilities of selecting actions) that maximizes the expected return $G_t$:
$$G_t = \sum_{k=0}^{\infty} \gamma^k R_{t+k+1}$$

### 4.2 Value Functions and the Bellman Equation
RL algorithms rely on estimating the value of being in a state. The **State-Value Function** $V^\pi(s)$ is the expected return starting from state $s$ and following policy $\pi$. The **Action-Value Function** $Q^\pi(s, a)$ is the expected return starting from state $s$, taking action $a$, and then following $\pi$.
The optimal Q-function satisfies the **Bellman Optimality Equation**:
$$Q^*(s, a) = \mathbb{E} \left[ R_{t+1} + \gamma \max_{a'} Q^*(s_{t+1}, a') \mid s_t = s, a_t = a \right]$$

### 4.3 Exploration vs. Exploitation
A fundamental challenge in RL is balancing **exploitation** (choosing the action known to yield the highest reward) with **exploration** (trying new actions to discover potentially better strategies). Common heuristics include $\epsilon$-greedy exploration and Upper Confidence Bound (UCB).

## 5. Types, Components & Classifications

```
                        ┌─────────────────────────────────────────┐
                        │        Reinforcement Learning           │
                        └───────────────────┬─────────────────────┘
                 ┌──────────────────────────┴──────────────────────────┐
                 ▼                                                     ▼
        ┌──────────────────┐                                  ┌──────────────────┐
        │   Model-Free     │                                  │   Model-Based    │
        │(Learns V or Pi)  │                                  │(Learns P and R)  │
        └────────┬─────────┘                                  └──────────────────┘
       ┌─────────┴─────────┐
       ▼                   ▼
┌─────────────┐     ┌─────────────┐
│ Value-Based │     │Policy-Based │
│(DQN, Q-learn)     │(REINFORCE)  │
└──────┬──────┘     └──────┬──────┘
       └─────────┬─────────┘
                 ▼
         ┌───────────────┐
         │ Actor-Critic  │
         │ (PPO, SAC)    │
         └───────────────┘
```

1. **Model-Free vs. Model-Based**:
   - *Model-Free*: The agent learns directly from experience without attempting to build an internal simulator of the environment's physics or transitions.
   - *Model-Based*: The agent learns a predictive "world model" $\mathcal{P}(s'|s,a)$ and uses it to plan ahead (e.g., via Monte Carlo Tree Search).
2. **Value-Based vs. Policy-Based**:
   - *Value-Based*: Learns the $Q$-function and derives the policy implicitly by selecting the argmax action (e.g., DQN).
   - *Policy-Based*: Directly parameterizes and optimizes the policy $\pi_\theta(a|s)$ via gradient ascent (e.g., Policy Gradients).
3. **Actor-Critic**: Combines both: an *Actor* updates the policy distribution, guided by a *Critic* which estimates the value function to reduce variance. **Proximal Policy Optimization (PPO)** (Schulman et al., 2017) is a widely used actor-critic-style algorithm as of its introduction; Soft Actor-Critic (SAC; Haarnoja et al., 2018) is an off-policy example.
4. **On-Policy vs. Off-Policy**: On-policy methods (PPO, SARSA) learn about the policy currently being executed. Off-policy methods (Q-learning, SAC) can learn from historical data generated by older or exploratory policies.

## 6. Applications & Concrete Examples
- **Large Language Models**: RLHF as described for InstructGPT used PPO to fine-tune the language model (Ouyang et al., 2022); alternatives such as DPO (Rafailov et al., 2023) have since been proposed. A separate reward model scores generated text, and the LLM (the actor) adjusts its generation policy to maximize human approval scores.
- **Complex Board & Video Games**: AlphaZero learned chess, shogi, and Go through self-play with Monte Carlo Tree Search, using the known game rules, without human game data (Silver et al., 2018). OpenAI Five defeated the Dota 2 world champions (Team OG) in April 2019 using large-scale self-play reinforcement learning (Berner et al., 2019).
- **Robotics & Autonomous Control**: RL has been applied to continuous control tasks. Research demonstrations include navigation of stratospheric balloons (Bellemare et al., 2020) and magnetic control of tokamak plasmas (Degrave et al., 2022); the Loon balloon project is reported to have ended in 2021 [VERIFICATION NEEDED: cite]. Legged-robot locomotion is also an RL application [VERIFICATION NEEDED: cite a specific paper].

## 7. Advantages, Limitations & Unresolved Questions

### Demonstrated Advantages
- Can discover novel, non-intuitive solutions outside the scope of human demonstration data.
- Capable of optimizing long-term sequential objectives in stochastic environments.

### Core Limitations
- **Sample Inefficiency**: Deep RL often requires millions or billions of environmental interactions to learn simple tasks, making real-world training on physical robots prohibitively slow and dangerous.
- **Reward Misspecification & Hacking**: If the mathematical reward signal $R$ does not perfectly align with human intent, the agent will find unintended "shortcuts" (specification gaming) to maximize $R$ while violating the spirit of the task.
- **Credit Assignment Problem**: In environments with sparse, delayed rewards (e.g., winning a game of chess after 60 moves), determining which specific early action was responsible for the ultimate success is computationally difficult.
- **Distribution Shift in Offline RL**: Training an RL agent strictly from fixed offline datasets without environmental interaction is challenged by distributional shift between the dataset and the learned policy, which can cause value overestimation on out-of-distribution actions (Levine et al., 2020).

### Unresolved Scientific Questions
1. *Safe Exploration*: How can an agent learn effectively in mission-critical real-world environments (like driving or surgery) without taking catastrophic exploratory actions?
2. *Generalization*: RL policies notoriously overfit to the exact visual and physical dynamics of their training simulation. Under what conditions can zero-shot transfer from simulation to reality (Sim2Real) be achieved reliably? This remains an open problem with no general guarantees.

## 8. Current State of Knowledge
As of 2022, PPO was the algorithm used for RLHF in InstructGPT (Ouyang et al., 2022), and SAC (Haarnoja et al., 2018) is a standard off-policy method for continuous control. Which algorithms dominate current LLM post-training practice is undated here and may differ from this description [VERIFICATION NEEDED: dated survey of current RL post-training practice]. **Offline RL**, where policies are learned from pre-collected datasets without environment interaction, is an active research area (Levine et al., 2020). Methods that learn **world models** (e.g., Dreamer; Hafner et al., 2020) report improved sample efficiency on the benchmarks studied by planning in a learned latent space; results are benchmark-specific.

## 9. Future Trajectory & Strategic Outlook
- **Near-Term (0–3 Years)** [speculative forecast, no forecast record assigned]: Direct Preference Optimization (DPO) and offline methods may substitute for online PPO in some language model post-training; RL may be used more widely for reasoning models.
- **Medium-Term (3–10 Years)** [speculative]: Model-based RL in embodied robotics may increasingly rely on learned world models.
- **Long-Term (10+ Years)** [speculative]: RL algorithms capable of open-ended, continuous lifelong learning, representing the algorithmic engine for general autonomous agents.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-ML-001] Machine Learning](./TECH-ML-001_machine_learning.md): The overarching discipline.
- **Related Methods & Technologies**:
  - [[TECH-DL-001] Deep Learning](./TECH-DL-001_deep_learning.md): The deep neural architectures that enabled DRL.
  - [[TECH-LLM-001] Large Language Models](./TECH-LLM-001_large_language_models.md): Foundation models optimized via RLHF.
- **Advanced Implications & Future Trajectories**:
  - [[SAFE-ALIGN-001] The AI Alignment Problem](../vol08_safety_governance/SAFE-ALIGN-001_ai_alignment.md): The dangers of reward hacking and specification gaming.
  - [[AGI-PATH-001] Scientific Pathways to AGI](../vol05_agi/AGI-PATH-001_pathways_to_agi.md): Self-play and open-ended RL as a primary trajectory toward general intelligence.

## 11. References & Foundational Literature
1. Sutton, R. S., & Barto, A. G. (2018). *Reinforcement Learning: An Introduction* (2nd ed.). MIT Press. http://incompleteideas.net/book/the-book-2nd.html. [Textbook covering MDPs, Bellman equations and temporal-difference learning.]
2. Sutton, R. S. (1988). *Learning to predict by the methods of temporal differences*. Machine Learning, 3(1), 9-44. https://doi.org/10.1007/BF00115009. [Introduces TD learning.]
3. Watkins, C. J. C. H., & Dayan, P. (1992). *Q-learning*. Machine Learning, 8(3-4), 279-292. https://doi.org/10.1007/BF00992698. [Q-learning and its convergence analysis.]
4. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., & Riedmiller, M. (2013). *Playing Atari with Deep Reinforcement Learning*. NIPS Deep Learning Workshop 2013. arXiv:1312.5602. [Workshop preprint of DQN.]
5. Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015). *Human-level control through deep reinforcement learning*. Nature, 518(7540), 529-533. https://doi.org/10.1038/nature14236. [DQN on Atari.]
6. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). *Proximal Policy Optimization Algorithms*. arXiv:1707.06347. [Introduces PPO.]
7. Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T., Leach, M., Kavukcuoglu, K., Graepel, T., & Hassabis, D. (2016). *Mastering the game of Go with deep neural networks and tree search*. Nature, 529(7587), 484-489. https://doi.org/10.1038/nature16961. [AlphaGo; reports a 5-0 win over Fan Hui and a 99.8% win rate against other Go programs; does not report the Lee Sedol match.]
8. Silver, D., Hubert, T., Schrittwieser, J., et al. (2018). *A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play*. Science, 362(6419), 1140-1144. https://doi.org/10.1126/science.aar6404 (13 authors; first three listed). [AlphaZero.]
9. Berner, C., et al. (OpenAI) (2019). *Dota 2 with Large Scale Deep Reinforcement Learning*. arXiv:1912.06680. [OpenAI Five; reports defeat of the Dota 2 world champions.]
10. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., & Lowe, R. (2022). *Training language models to follow instructions with human feedback*. NeurIPS 2022 [venue not independently confirmed]. arXiv:2203.02155. [PPO-based RLHF for InstructGPT.]
11. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). *Direct Preference Optimization: Your Language Model Is Secretly a Reward Model*. arXiv:2305.18290. [Introduces DPO.]
12. Levine, S., Kumar, A., Tucker, G., & Fu, J. (2020). *Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems*. arXiv:2005.01643. [Review of offline RL, including distributional shift.]
13. Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018). *Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor*. ICML 2018. arXiv:1801.01290. [Introduces SAC.]
14. Hafner, D., Lillicrap, T., Norouzi, M., & Ba, J. (2020). *Mastering Atari with Discrete World Models*. ICLR 2021. arXiv:2010.02193. [DreamerV2: world-model-based RL on Atari.]
15. Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J., & Mané, D. (2016). *Concrete Problems in AI Safety*. arXiv:1606.06565. [Discusses reward hacking among other safety problems.]
16. Bellemare, M. G., Candido, S., Castro, P. S., Gong, J., Machado, M. C., Moitra, S., Ponda, S. S., & Wang, Z. (2020). *Autonomous navigation of stratospheric balloons using reinforcement learning*. Nature, 588, 77-82. https://doi.org/10.1038/s41586-020-2939-8. [Loon balloon navigation.]
17. Degrave, J., Felici, F., Buchli, J., et al. (2022). *Magnetic control of tokamak plasmas through deep reinforcement learning*. Nature, 602, 414-419. https://doi.org/10.1038/s41586-021-04301-9 (31 authors; first three listed). [Tokamak control demonstration.]
18. Google DeepMind (n.d.). *AlphaGo*. https://deepmind.google/research/breakthroughs/alphago/ (accessed 2026-10-15). [States that AlphaGo beat Lee Sedol 4-1 in March 2016 and beat Fan Hui 5-0 earlier; Tier 3 organizational page.]

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Bellman Optimality Equations | Established (Mathematical Result) | Sutton & Barto (2018) | High |
| Convergence of Q-Learning (under standard tabular conditions) | Established (Mathematical Proof) [conditions in the paper not rechecked in this revision] | Watkins & Dayan (1992) | High |
| DQN reaches human-level performance on many Atari games | Well Supported (single-lab empirical result on a benchmark) | Mnih et al. (2015) | High on the benchmark |
| PPO used for RLHF in InstructGPT (as of 2022) | Well Supported (dated); current dominance not established | Ouyang et al. (2022) | Moderate; dated |
| Reward Hacking & Specification Gaming | Well Supported (documented cases; conceptual framing) | Amodei et al. (2016) | Moderate to High |

## 13. Revision History
- **v1.0.0** (2026-10-15): Initial draft; citations not independently verified.
- **v0.2.0** (2026-10-15): Corrected following independent fact-check audit (docs/audits/TECH-RL-001_audit.md); 12 findings applied, 3 left marked for verification.