Pretraining, Fine-Tuning and RLHF

The standard training pipeline for a modern language model has two broad phases. Pretraining fits a model to predict the next token on a very large text corpus. Post-training then adapts the pretrained model to follow instructions and match human preferences, typically through supervised fine-tuning (SFT) on demonstrations followed by preference-based optimisation such as reinforcement learning from human feedback (RLHF), in which a reward model learned from human comparisons is used to steer the model.

A pretrained language model is a general-purpose text predictor. It continues documents rather than answering requests, and it reproduces the distribution of its training text, including falsehoods and toxic content. Post-training is the stage that turns such a model into an assistant. Ouyang et al. (2022) reported that outputs from a 1.3-billion-parameter model trained with human feedback were preferred by labelers to those of the 175-billion-parameter GPT-3 on their prompt distribution, which suggests that, on that prompt distribution, the post-pretraining training objective can matter more for perceived helpfulness than a parameter difference of more than 100-fold (about 135-fold). This is a single-study result on the authors' own prompts.

This article covers the technical methods. The broader question of whether such methods suffice to make advanced systems safe is treated in [SAFE-ALIGN-001] and [SAFE-OVERSIGHT-001]; the use of reinforcement learning to train long-form reasoning is treated in [TECH-REASON-001]. Here, as elsewhere, details of closed systems are reported only to the extent that developers have disclosed them, and the remainder is marked "undisclosed".

Transfer learning for language predates the current pipeline: ULMFiT (Howard & Ruder, 2018) showed that fine-tuning a pretrained language model could improve text classification, and Gururangan et al. (2020) showed that a second phase of in-domain pretraining improved downstream performance across the four domains and eight tasks they studied. GPT-3 (Brown et al., 2020) demonstrated few-shot prompting of a pretrained model without gradient updates.

Instruction tuning appeared in FLAN (Wei et al., 2022; arXiv 2021), which fine-tuned a 137-billion-parameter pretrained model on more than 60 NLP datasets verbalised as natural-language instructions and reported better zero-shot results on held-out tasks, and in T0 (Sanh et al., 2022), which reported that explicit multitask prompted training gave strong zero-shot performance, often outperforming models up to 16 times larger.

Learning from human preferences originated in reinforcement learning, where Christiano et al. (2017) trained agents from human comparisons of trajectory segments, using feedback on less than one percent of interactions in their tasks. Ziegler et al. (2019) applied the idea to language models, Stiennon et al. (2020) to summarisation, and Ouyang et al. (2022) to broad instruction following (InstructGPT). Bai et al. (2022a) applied preference modelling and RLHF to helpful and harmless assistants and reported a roughly linear relation between reward and the square root of the KL divergence from the initial policy. Bai et al. (2022b) proposed Constitutional AI, in which AI-generated preferences guided by written principles replace human labels for harmlessness. Rafailov et al. (2023) introduced Direct Preference Optimization (DPO), which removes the explicit reward model and RL loop. In 2024, Shao et al. introduced GRPO (using learned reward models in that paper), and Lambert et al. (2024) named reinforcement learning with verifiable rewards (RLVR) as a post-training stage of an open recipe.

#4.1 Pretraining

Given a corpus tokenised into sequences $t_1,\dots,t_n$ (see [TECH-TOK-001]), the model with parameters $\theta$ minimises

Data composition and quality, and the scaling relationships between parameters, data and compute, are covered in [TECH-DATA-001] and [TECH-SCALE-001]. For one system, Ouyang et al. (2022) report that training their 175B SFT model required 4.9 petaflops/s-days and their 175B PPO-ptx model 60 petaflops/s-days, compared with 3,640 petaflops/s-days for pretraining GPT-3, so post-training was a small fraction of pretraining compute in that case; this is not established for other systems. For GPT-4, its developer's report states that the model's capabilities on the exams it tested "appear to stem primarily from the pre-training process and are not significantly affected by RLHF" (OpenAI et al., 2023). That is a statement about those exams, not about every behaviour.

#4.2 Supervised fine-tuning (SFT)

SFT continues training with the same next-token loss, but on pairs $(x, y)$ of prompts and desired responses, usually with the loss applied to the response tokens only:

Demonstrations may be written by labelers (Ouyang et al., 2022), collected as templated task collections (Wei et al., 2022; Sanh et al., 2022), or curated. Zhou et al. (2023) trained a 65B-parameter model on only 1,000 curated prompt-response pairs with no reinforcement learning (LIMA) and reported that in a controlled human study its responses were equivalent or preferred to GPT-4 in 43% of cases; they suggest that most knowledge is learned in pretraining and that alignment mainly teaches format and style. This "superficial alignment" view is a hypothesis supported by their evaluation, not a settled result.

Because full fine-tuning updates all parameters, parameter-efficient methods such as LoRA (Hu et al., 2022), which trains low-rank updates to weight matrices, are widely used for adaptation.

#4.3 Reward modelling

Human labelers rank or compare model outputs. For a prompt $x$ with preferred response $y_w$ and less-preferred response $y_l$, a reward model $r_\phi(x,y)$ is trained under a Bradley–Terry model of pairwise preference:

where $\sigma$ is the logistic function. The reward model is a proxy: it is trained on a finite sample of judgements from a particular group of labelers.

#4.4 RL optimisation

The policy $\pi_\theta$ (initialised from the SFT model, $\pi_{\text{ref}}$) is then optimised to maximise the proxy reward while staying close to the reference:

The KL penalty (coefficient $\beta$) limits drift. The optimiser in the best-documented pipelines is Proximal Policy Optimization (PPO; Schulman et al., 2017), a policy-gradient method described in [TECH-RL-001]. Ouyang et al. (2022) followed three steps: SFT on demonstrations, reward-model training on rankings, and PPO against the reward model. Llama 2 documented an RLHF stage combining rejection sampling and PPO on top of SFT (Touvron et al., 2023).

#4.5 Direct preference optimisation

Rafailov et al. (2023) show that the KL-regularised objective above has a closed-form optimal policy, so the reward can be re-parameterised through the policy itself, giving a classification-style loss on preference pairs:

DPO trains on a fixed preference dataset without sampling during training, which makes it simpler and more stable to run than PPO-based RLHF, at the cost of the on-policy exploration that RL provides.

#4.6 AI feedback and verifiable rewards

• RLAIF / Constitutional AI. In the supervised phase the model critiques and revises its own responses according to a list of principles; in the RL phase a preference model is trained from AI comparisons rather than human labels for harmlessness (Bai et al., 2022b).
• Reinforcement learning with verifiable rewards (RLVR). For tasks where correctness can be checked automatically (mathematics answers, code tests), the reward is computed by a program rather than a learned model. Tulu 3 describes RLVR as a post-training step alongside SFT and DPO (Lambert et al., 2024), and DeepSeekMath introduced Group Relative Policy Optimization (GRPO), a PPO variant that drops the learned value function to reduce memory use (Shao et al., 2024). In DeepSeekMath, GRPO was applied with learned reward models; its pairing with programmatic rewards is a later usage and is not claimed by that paper. The role of this technique in reasoning models is covered in [TECH-REASON-001].

• InstructGPT. SFT, reward modelling and PPO on GPT-3; 1.3B variant preferred over 175B GPT-3 by labelers on the authors' prompts, with reported improvements in truthfulness (Ouyang et al., 2022).
• Anthropic assistants. Published research on HH-RLHF and Constitutional AI (Bai et al., 2022a, 2022b). The training recipes of Anthropic's current deployed models are undisclosed beyond what the company has published.
• GPT-4. OpenAI's report states the model was pre-trained to predict the next token and then fine-tuned with RLHF, and that it contains no further details about training method, dataset construction, architecture or compute; these are undisclosed (OpenAI et al., 2023).
• Open recipes. Llama 2 and Tulu 3 publish extensive post-training descriptions (Touvron et al., 2023; Lambert et al., 2024), making them the main public basis for study of the full pipeline.

#Advantages

Large improvements in perceived helpfulness and instruction following at low cost relative to pretraining; compatibility with specialised skills (Bai et al., 2022a); a framework in which new preferences, including safety policies, can be expressed through data.

#Limitations

Casper et al. (2023) survey open problems and fundamental limitations of RLHF, organised around problems with human feedback, the reward model and the policy, and propose disclosure and auditing standards. Specific documented issues:

• Reward overoptimisation. Optimising too hard against an imperfect reward model degrades the true objective, in line with Goodhart's law. In a synthetic setup with a fixed gold reward model, Gao et al. (2023) find different functional forms for RL and best-of-$n$ sampling, with coefficients that scale smoothly with reward-model parameters.
• Sycophancy. Sharma et al. (2024) report that five AI assistants consistently exhibited sycophancy on four free-form tasks, that human and preference-model judgements sometimes prefer convincingly written sycophantic responses to correct ones, and that optimising against preference models sometimes sacrifices truthfulness. They infer that human preference judgements likely play a role, not that RLHF is the sole cause.
• Labeler limits. Preferences reflect the labelers, instructions and prompt distribution used; they are weak for tasks the labelers cannot evaluate (see [SAFE-OVERSIGHT-001]).
• Training complexity. PPO-based RLHF requires several models (policy, reference, reward, and often a value model), which motivated simpler objectives such as DPO.

#Unresolved questions

1. Relative benefit of online RL versus offline preference optimisation; the evidence comes from varied setups and no single comparison is decisive.
2. How much of post-training changes capability versus style. The LIMA result is a single human-evaluation study of one 1,000-example fine-tuning set and does not settle whether larger post-training data or RL add capability beyond what pretraining supplies.
3. Whether alignment training that works for current assistants scales to systems more capable than their overseers (see [SAFE-ALIGN-001]).
4. Hallucination and calibration effects of post-training (see [CAP-LIMITS-001]).

As of 2026-10-19, the pretrain, SFT and preference/RL pipeline is the documented standard for assistant models, with variation in the specific preference-optimisation algorithm and growing use of verifiable-reward RL in open recipes. For open-weight models, the pipeline is documented in some detail. For closed models the stages that developers have named (for example RLHF in the GPT-4 report) are known, but datasets, reward-model design, hyperparameters, proportions of compute and later changes are undisclosed. Claims that a specific current closed model was trained with a particular algorithm should be attributed to a developer statement or marked undisclosed. Model-specific facts are tracked in MODEL-001.

Observed trends: replacement or augmentation of human labels by AI feedback and programmatic checks; open, fully documented recipes (Lambert et al., 2024); growing attention to feedback quality, sycophancy and evaluation of reward models; and extension of RL from short-form preference tasks to long-form reasoning. Whether these methods continue to work as capabilities rise is an open question rather than a forecast made here; no forecast record is registered in this article.

• Foundational Concepts:
  • TECH-LLM-001 Large Language Models: The pretrained models being adapted.
  • TECH-RL-001 Reinforcement Learning: Policy gradients, PPO and reward design.
  • FOUND-010 Optimization and Gradient Descent: Optimisation background.
• Related Methods & Technologies:
  • TECH-SCALE-001 Scaling Laws: Pretraining scaling and reward overoptimisation scaling.
  • TECH-DATA-001 Training Data, Contamination and Provenance: Data used at each stage.
  • TECH-TOK-001 Tokenization and Embeddings: The units of the training loss.
  • TECH-REASON-001 Reasoning Models and Test-Time Compute: RL against verifiable rewards.
• Advanced Implications & Future Trajectories:
  • SAFE-ALIGN-001 AI Alignment: The wider alignment problem.
  • SAFE-OVERSIGHT-001 Scalable Oversight and Corrigibility: Supervision beyond labeler ability.
  • CAP-LIMITS-001 Hallucination, Reliability and Calibration: Reliability effects.