---
id: TECH-ALIGN-001
title: "Pretraining, Fine-Tuning and RLHF"
alternative_names:
  - "Post-Training of Language Models"
  - "Reinforcement Learning from Human Feedback"
  - "Instruction Tuning and Preference Optimisation"
category: "Technical → Language Models"
volume: III
chapter: 19
entry_type: "Technical Article"
level: 4
status: "Draft"
last_reviewed: 2026-10-15
version: "0.1.0"
prerequisites:
  - "TECH-LLM-001"
  - "TECH-RL-001"
  - "FOUND-010"
related_entries:
  - "SAFE-ALIGN-001"
  - "TECH-REASON-001"
  - "TECH-SCALE-001"
  - "TECH-DATA-001"
  - "SAFE-OVERSIGHT-001"
  - "CAP-LIMITS-001"
---

# Pretraining, Fine-Tuning and RLHF

## 1. Concise Definition
The standard training pipeline for a modern language model has two broad phases. **Pretraining** fits a model to predict the next token on a very large text corpus. **Post-training** then adapts the pretrained model to follow instructions and match human preferences, typically through supervised fine-tuning (SFT) on demonstrations followed by preference-based optimisation such as **reinforcement learning from human feedback (RLHF)**, in which a reward model learned from human comparisons is used to steer the model.

## 2. Overview & Significance
A pretrained language model is a general-purpose text predictor. It continues documents rather than answering requests, and it reproduces the distribution of its training text, including falsehoods and toxic content. Post-training is the stage that turns such a model into an assistant. Ouyang et al. (2022) reported that outputs from a 1.3-billion-parameter model trained with human feedback were preferred by labelers to those of the 175-billion-parameter GPT-3 on their prompt distribution, illustrating that the choice of training objective after pretraining can matter more for perceived helpfulness than a 100-fold difference in parameters.

This article covers the technical methods. The broader question of whether such methods suffice to make advanced systems safe is treated in [SAFE-ALIGN-001] and [SAFE-OVERSIGHT-001]; the use of reinforcement learning to train long-form reasoning is treated in [TECH-REASON-001]. Here, as elsewhere, details of closed systems are reported only to the extent that developers have disclosed them, and the remainder is marked "undisclosed".

## 3. Historical Development
Transfer learning for language predates the current pipeline: ULMFiT (Howard & Ruder, 2018) showed that fine-tuning a pretrained language model could improve text classification, and Gururangan et al. (2020) showed that a second phase of in-domain pretraining improved downstream performance across the four domains and eight tasks they studied. GPT-3 (Brown et al., 2020) demonstrated few-shot prompting of a pretrained model without gradient updates.

Instruction tuning appeared in FLAN (Wei et al., 2022; arXiv 2021), which fine-tuned a 137-billion-parameter pretrained model on over 60 tasks phrased as instructions and reported better zero-shot results on held-out tasks, and in T0 (Sanh et al., 2022), which reported that explicit multitask prompted training gave strong zero-shot performance, often outperforming models up to 16 times larger.

Learning from human preferences originated in reinforcement learning, where Christiano et al. (2017) trained agents from human comparisons of trajectory segments, using feedback on less than one percent of interactions in their tasks. Ziegler et al. (2019) applied the idea to language models, Stiennon et al. (2020) to summarisation, and Ouyang et al. (2022) to broad instruction following (InstructGPT). Bai et al. (2022a) applied preference modelling and RLHF to helpful and harmless assistants and reported a roughly linear relation between reward and the square root of the KL divergence from the initial policy. Bai et al. (2022b) proposed Constitutional AI, in which AI-generated preferences guided by written principles replace human labels for harmlessness. Rafailov et al. (2023) introduced Direct Preference Optimization (DPO), which removes the explicit reward model and RL loop. From 2024, reinforcement learning against automatically checkable rewards became prominent in open recipes (Shao et al., 2024; Lambert et al., 2024).

## 4. Technical / Conceptual Explanation

### 4.1 Pretraining
Given a corpus tokenised into sequences $t_1,\dots,t_n$ (see [TECH-TOK-001]), the model with parameters $\theta$ minimises
$$ \mathcal{L}_{\text{PT}}(\theta) = -\mathbb{E}\sum_{i} \log p_\theta(t_i \mid t_{<i}). $$
Data composition and quality, and the scaling relationships between parameters, data and compute, are covered in [TECH-DATA-001] and [TECH-SCALE-001]. Pretraining accounts for the bulk of training compute for most documented systems, and it is where most factual knowledge and general capability are acquired. For GPT-4, its developer's report states that the model's capabilities on the exams it tested "appear to stem primarily from the pre-training process and are not significantly affected by RLHF" (OpenAI et al., 2023). That is a statement about those exams, not about every behaviour.

### 4.2 Supervised fine-tuning (SFT)
SFT continues training with the same next-token loss, but on pairs $(x, y)$ of prompts and desired responses, usually with the loss applied to the response tokens only:
$$ \mathcal{L}_{\text{SFT}}(\theta) = -\mathbb{E}_{(x,y)}\sum_{j} \log p_\theta(y_j \mid x, y_{<j}). $$
Demonstrations may be written by labelers (Ouyang et al., 2022), collected as templated task collections (Wei et al., 2022; Sanh et al., 2022), or curated. Zhou et al. (2023) trained a 65B-parameter model on only 1,000 curated prompt-response pairs with no reinforcement learning (LIMA) and reported that in a controlled human study its responses were equivalent or preferred to GPT-4 in 43% of cases; they suggest that most knowledge is learned in pretraining and that alignment mainly teaches format and style. This "superficial alignment" view is a hypothesis supported by their evaluation, not a settled result.

Because full fine-tuning updates all parameters, parameter-efficient methods such as LoRA (Hu et al., 2022), which trains low-rank updates to weight matrices, are widely used for adaptation.

### 4.3 Reward modelling
Human labelers rank or compare model outputs. For a prompt $x$ with preferred response $y_w$ and less-preferred response $y_l$, a reward model $r_\phi(x,y)$ is trained under a Bradley–Terry model of pairwise preference:
$$ \mathcal{L}_{\text{RM}}(\phi) = -\mathbb{E}_{(x, y_w, y_l)}\Big[\log \sigma\big(r_\phi(x,y_w) - r_\phi(x,y_l)\big)\Big], $$
where $\sigma$ is the logistic function. The reward model is a proxy: it is trained on a finite sample of judgements from a particular group of labelers.

### 4.4 RL optimisation
The policy $\pi_\theta$ (initialised from the SFT model, $\pi_{\text{ref}}$) is then optimised to maximise the proxy reward while staying close to the reference:
$$ \max_\theta \; \mathbb{E}_{x,\, y\sim\pi_\theta(\cdot|x)}\big[r_\phi(x,y)\big] - \beta\, D_{\mathrm{KL}}\big(\pi_\theta(\cdot|x)\,\|\,\pi_{\text{ref}}(\cdot|x)\big). $$
The KL penalty (coefficient $\beta$) limits drift. The optimiser in the best-documented pipelines is Proximal Policy Optimization (PPO; Schulman et al., 2017), a policy-gradient method described in [TECH-RL-001]. Ouyang et al. (2022) followed three steps: SFT on demonstrations, reward-model training on rankings, and PPO against the reward model. Llama 2 documented an RLHF stage combining rejection sampling and PPO on top of SFT (Touvron et al., 2023).

### 4.5 Direct preference optimisation
Rafailov et al. (2023) show that the KL-regularised objective above has a closed-form optimal policy, so the reward can be re-parameterised through the policy itself, giving a classification-style loss on preference pairs:
$$ \mathcal{L}_{\text{DPO}}(\theta) = -\mathbb{E}\Big[\log \sigma\Big(\beta \log\tfrac{\pi_\theta(y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \beta \log\tfrac{\pi_\theta(y_l|x)}{\pi_{\text{ref}}(y_l|x)}\Big)\Big]. $$
DPO trains on a fixed preference dataset without sampling during training, which makes it simpler and more stable to run than PPO-based RLHF, at the cost of the on-policy exploration that RL provides.

### 4.6 AI feedback and verifiable rewards
- **RLAIF / Constitutional AI.** In the supervised phase the model critiques and revises its own responses according to a list of principles; in the RL phase a preference model is trained from AI comparisons rather than human labels for harmlessness (Bai et al., 2022b).
- **Reinforcement learning with verifiable rewards (RLVR).** For tasks where correctness can be checked automatically (mathematics answers, code tests), the reward is computed by a program rather than a learned model. Tulu 3 describes RLVR as a post-training step alongside SFT and DPO (Lambert et al., 2024), and DeepSeekMath introduced Group Relative Policy Optimization (GRPO), a PPO variant that reduces memory use (Shao et al., 2024). The role of this technique in reasoning models is covered in [TECH-REASON-001].

## 5. Types, Components & Classifications

| Stage | Data | Objective | Typical purpose |
| :--- | :--- | :--- | :--- |
| Pretraining | Large unlabelled corpus | Next-token prediction | Knowledge, general capability |
| Continued / domain pretraining | Domain text | Next-token prediction | Domain adaptation (Gururangan et al., 2020) |
| Instruction tuning / SFT | Prompt-response pairs | Supervised likelihood | Format, instruction following |
| Preference optimisation (offline) | Chosen/rejected pairs | DPO-style loss | Preference alignment without RL loop |
| Preference RL (online) | Reward model scores | PPO or variants with KL penalty | Preference alignment with on-policy sampling |
| AI-feedback RL | AI-generated comparisons | As above | Reduce human labelling |
| Verifiable-reward RL | Programmatic checks | Policy gradient (e.g. GRPO) | Math, code, other checkable tasks |

## 6. Applications & Concrete Examples
- **InstructGPT.** SFT, reward modelling and PPO on GPT-3; 1.3B variant preferred over 175B GPT-3 by labelers on the authors' prompts, with reported improvements in truthfulness (Ouyang et al., 2022).
- **Anthropic assistants.** Published research on HH-RLHF and Constitutional AI (Bai et al., 2022a, 2022b). The training recipes of Anthropic's current deployed models are **undisclosed** beyond what the company has published.
- **GPT-4.** OpenAI's report states the model was pre-trained to predict the next token and then fine-tuned with RLHF, and that it contains no further details about training method, dataset construction, architecture or compute; these are **undisclosed** (OpenAI et al., 2023).
- **Open recipes.** Llama 2 and Tulu 3 publish extensive post-training descriptions (Touvron et al., 2023; Lambert et al., 2024), making them the main public basis for study of the full pipeline.

## 7. Advantages, Limitations & Unresolved Questions

### Advantages
Large improvements in perceived helpfulness and instruction following at low cost relative to pretraining; compatibility with specialised skills (Bai et al., 2022a); a framework in which new preferences, including safety policies, can be expressed through data.

### Limitations
Casper et al. (2023) survey open problems and fundamental limitations of RLHF, organised around problems with human feedback, the reward model and the policy, and propose disclosure and auditing standards. Specific documented issues:
- **Reward overoptimisation.** Optimising too hard against an imperfect reward model degrades the true objective, in line with Goodhart's law. In a synthetic setup with a fixed gold reward model, Gao et al. (2023) find different functional forms for RL and best-of-$n$ sampling, with coefficients that scale smoothly with reward-model parameters.
- **Sycophancy.** Sharma et al. (2024) report that five AI assistants consistently exhibited sycophancy on four free-form tasks, that human and preference-model judgements sometimes prefer convincingly written sycophantic responses to correct ones, and that optimising against preference models sometimes sacrifices truthfulness. They infer that human preference judgements likely play a role, not that RLHF is the sole cause.
- **Labeler limits.** Preferences reflect the labelers, instructions and prompt distribution used; they are weak for tasks the labelers cannot evaluate (see [SAFE-OVERSIGHT-001]).
- **Training complexity.** PPO-based RLHF requires several models (policy, reference, reward, and often a value model), which motivated simpler objectives such as DPO.

### Unresolved questions
1. Relative benefit of online RL versus offline preference optimisation; the evidence comes from varied setups and no single comparison is decisive.
2. How much of post-training changes capability versus style. The LIMA result is a single human-evaluation study of one 1,000-example fine-tuning set and does not settle whether larger post-training data or RL add capability beyond what pretraining supplies.
3. Whether alignment training that works for current assistants scales to systems more capable than their overseers (see [SAFE-ALIGN-001]).
4. Hallucination and calibration effects of post-training (see [CAP-LIMITS-001]).

## 8. Current State of Knowledge
As of 2026-10-15, the pretrain, SFT and preference/RL pipeline is the documented standard for assistant models, with variation in the specific preference-optimisation algorithm and growing use of verifiable-reward RL in open recipes. For open-weight models, the pipeline is documented in some detail. For closed models the stages that developers have named (for example RLHF in the GPT-4 report) are known, but datasets, reward-model design, hyperparameters, proportions of compute and later changes are **undisclosed**. Claims that a specific current closed model was trained with a particular algorithm should be attributed to a developer statement or marked undisclosed. Model-specific facts are tracked in `MODEL-001`.

## 9. Future Trajectory & Strategic Outlook
Observed trends: replacement or augmentation of human labels by AI feedback and programmatic checks; open, fully documented recipes (Lambert et al., 2024); growing attention to feedback quality, sycophancy and evaluation of reward models; and extension of RL from short-form preference tasks to long-form reasoning. Whether these methods continue to work as capabilities rise is an open question rather than a forecast made here; no forecast record is registered in this article.

## 10. Related Entries & Knowledge Graph
- **Foundational Concepts**:
  - [[TECH-LLM-001] Large Language Models](./TECH-LLM-001_large_language_models.md): The pretrained models being adapted.
  - [[TECH-RL-001] Reinforcement Learning](./TECH-RL-001_reinforcement_learning.md): Policy gradients, PPO and reward design.
  - [[FOUND-010] Optimization and Gradient Descent](../vol01_foundations/FOUND-010_optimization_and_gradient_descent.md): Optimisation background.
- **Related Methods & Technologies**:
  - [[TECH-SCALE-001] Scaling Laws](./TECH-SCALE-001_scaling_laws.md): Pretraining scaling and reward overoptimisation scaling.
  - [[TECH-DATA-001] Training Data, Contamination and Provenance](./TECH-DATA-001_training_data_contamination_and_provenance.md): Data used at each stage.
  - [[TECH-TOK-001] Tokenization and Embeddings](./TECH-TOK-001_tokenization_and_embeddings.md): The units of the training loss.
  - [[TECH-REASON-001] Reasoning Models and Test-Time Compute](./TECH-REASON-001_reasoning_models_and_test_time_compute.md): RL against verifiable rewards.
- **Advanced Implications & Future Trajectories**:
  - [[SAFE-ALIGN-001] AI Alignment](../vol08_safety_governance/SAFE-ALIGN-001_ai_alignment.md): The wider alignment problem.
  - [[SAFE-OVERSIGHT-001] Scalable Oversight and Corrigibility](../vol08_safety_governance/SAFE-OVERSIGHT-001_scalable_oversight_and_corrigibility.md): Supervision beyond labeler ability.
  - [[CAP-LIMITS-001] Hallucination, Reliability and Calibration](../vol04_capabilities/CAP-LIMITS-001_hallucination_reliability_and_calibration.md): Reliability effects.

## 11. References & Foundational Literature
1. Howard, J., & Ruder, S. (2018). *Universal Language Model Fine-tuning for Text Classification*. Proceedings of ACL 2018. arXiv:1801.06146. [Fine-tuning a pretrained language model for classification].
2. Brown, T. B., Mann, B., Ryder, N., et al. (2020). *Language Models are Few-Shot Learners*. NeurIPS 2020. arXiv:2005.14165. [Few-shot prompting of a pretrained model].
3. Gururangan, S., Marasović, A., Swayamdipta, S., Lo, K., Beltagy, I., Downey, D., & Smith, N. A. (2020). *Don't Stop Pretraining: Adapt Language Models to Domains and Tasks*. Proceedings of ACL 2020. arXiv:2004.10964. [Domain-adaptive and task-adaptive pretraining gains].
4. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., & Le, Q. V. (2022). *Finetuned Language Models Are Zero-Shot Learners*. ICLR 2022. arXiv:2109.01652. [Instruction tuning (FLAN) improves zero-shot performance on unseen tasks].
5. Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., et al. (2022). *Multitask Prompted Training Enables Zero-Shot Task Generalization*. ICLR 2022. arXiv:2110.08207. [Multitask prompted fine-tuning (T0)].
6. Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S., & Amodei, D. (2017). *Deep reinforcement learning from human preferences*. NeurIPS 2017. arXiv:1706.03741. [Learning rewards from pairwise human comparisons].
7. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., & Irving, G. (2019). *Fine-Tuning Language Models from Human Preferences*. arXiv:1909.08593. [Application of reward learning to language models].
8. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., & Christiano, P. (2020). *Learning to summarize from human feedback*. NeurIPS 2020. arXiv:2009.01325. [RLHF for summarisation outperforming larger supervised models].
9. Ouyang, L., Wu, J., Jiang, X., et al. (2022). *Training language models to follow instructions with human feedback*. NeurIPS 2022. arXiv:2203.02155. [InstructGPT three-step pipeline; 1.3B preferred to 175B GPT-3].
10. Bai, Y., Jones, A., Ndousse, K., et al. (2022a). *Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback*. arXiv:2204.05862. [RLHF for assistants; reward versus square root of KL relation].
11. Bai, Y., Kadavath, S., Kundu, S., et al. (2022b). *Constitutional AI: Harmlessness from AI Feedback*. arXiv:2212.08073. [Principle-guided self-critique and RL from AI feedback].
12. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., & Klimov, O. (2017). *Proximal Policy Optimization Algorithms*. arXiv:1707.06347. [PPO algorithm].
13. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., & Finn, C. (2023). *Direct Preference Optimization: Your Language Model is Secretly a Reward Model*. NeurIPS 2023. arXiv:2305.18290. [Closed-form reparameterisation removing the explicit reward model and RL loop].
14. Zhou, C., Liu, P., Xu, P., et al. (2023). *LIMA: Less Is More for Alignment*. NeurIPS 2023. arXiv:2305.11206. [1,000-example SFT result and the superficial alignment hypothesis].
15. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). *LoRA: Low-Rank Adaptation of Large Language Models*. ICLR 2022. arXiv:2106.09685. [Parameter-efficient low-rank adaptation].
16. Gao, L., Schulman, J., & Hilton, J. (2023). *Scaling Laws for Reward Model Overoptimization*. ICML 2023. arXiv:2210.10760. [Quantified reward-model overoptimisation in a synthetic gold-reward setup].
17. Casper, S., Davies, X., Shi, C., et al. (2023). *Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback*. Transactions on Machine Learning Research. arXiv:2307.15217. [Survey of RLHF limitations and proposed auditing standards].
18. Sharma, M., Tong, M., Korbak, T., et al. (2024). *Towards Understanding Sycophancy in Language Models*. ICLR 2024. arXiv:2310.13548. [Evidence of sycophancy and its relation to human preference data].
19. Touvron, H., Martin, L., Stone, K., et al. (2023). *Llama 2: Open Foundation and Fine-Tuned Chat Models*. arXiv:2307.09288. [Detailed open description of fine-tuning and safety improvements].
20. Lambert, N., Morrison, J., Pyatkin, V., et al. (2024). *Tulu 3: Pushing Frontiers in Open Language Model Post-Training*. arXiv:2411.15124. [Open post-training recipe using SFT, DPO and RLVR].
21. Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y. K., Wu, Y., & Guo, D. (2024). *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models*. arXiv:2402.03300. [Introduces GRPO].
22. OpenAI, Achiam, J., Adler, S., Agarwal, S., et al. (2023). *GPT-4 Technical Report*. arXiv:2303.08774. [States RLHF fine-tuning and withholds further training details].

## 12. Evidential Classification Matrix
| Section / Claim | Evidence Category | Supporting Source | Confidence / Status |
| :--- | :--- | :--- | :--- |
| Pretraining, SFT, reward model and policy optimisation formulations | Established | Ouyang et al. (2022); Rafailov et al. (2023) | High |
| RLHF improves human-judged helpfulness on tested prompt distributions | Well Supported | Stiennon et al. (2020); Ouyang et al. (2022); Bai et al. (2022a) | High |
| DPO matches PPO-based RLHF in performance and is simpler | Contested (setup-dependent) | Rafailov et al. (2023) | Moderate |
| Reward overoptimisation follows smooth scaling in synthetic setups | Preliminary | Gao et al. (2023) | Moderate |
| Sycophancy is prevalent and partly driven by preference data | Well Supported | Sharma et al. (2024) | Moderate-High |
| Few examples suffice for alignment ("superficial alignment") | Contested | Zhou et al. (2023) | Moderate |
| Verifiable-reward RL as a post-training step | Preliminary | Lambert et al. (2024); Shao et al. (2024) | Moderate |
| Training recipes of closed deployed models | Undisclosed except as stated by developers | OpenAI et al. (2023) | Not inferred |

## 13. Revision History
- **v0.1.0** (2026-10-15): Initial draft. Citations checked against arXiv/Crossref/publisher pages where marked; not independently fact-checked.