Reinforcement Learning for LLMs: RLHF, PPO, GRPO, DPO, RLAIF, and Reasoning

If pretraining teaches a language model how to predict what comes next, reinforcement learning asks a more interesting question: how do we make the model systematically prefer outputs that satisfy a target objective?

That objective might be human preference, instruction following, mathematical correctness, code correctness, safety, or some combination of signals. RLHF, reward modeling, PPO, RLAIF, GRPO, process reward models, DPO, verifiable rewards, and inference-time scaling are not isolated tricks. They are different ways of shaping a language model's behavior through feedback

Given a prompt x, an LLM can be viewed as a policy πθ that samples a completion y token by token. Its probability factorization is:

πθ(y | x) = ∏t πθ(yt | x, y<t)

If a reward function R(x,y) scores the completed trajectory, the high-level objective becomes:

J(θ) = E_(x,y)~πθ [R(x,y)]

The conceptual leap is simple: instead of only increasing the probability of human-written continuations, we can increase the probability of trajectories that produce desirable outcomes

Framing this as a Markov Decision Process makes the structure precise. The state at step t is s_t = (x, y_1, ..., y_{t-1}) — the full prompt concatenated with all tokens generated so far. The action a_t = y_t is a single token drawn from the vocabulary. An episode terminates when the model samples an end-of-sequence (EOS) token or reaches a maximum sequence length. The reward R(x,y) is typically sparse: a single scalar assigned only at the end of the episode, once the full completion is available. This sparsity makes credit assignment over long token sequences a central challenge — gradient signal must propagate backward through potentially thousands of token steps

The action space compounds this difficulty. A typical LLM vocabulary contains 50,000–150,000 tokens, making the branching factor at each step orders of magnitude larger than classic RL environments like Atari (18 actions) or Go (361 board positions). Exploration in this space is not random walk — it is guided by the policy's learned language priors — but the sheer size of the space means that naive exploration strategies are computationally infeasible. This is why RL for LLMs almost always operates on top of a strong SFT policy that already concentrates probability mass on plausible continuations

SFT provides the initial post-training policy. Its objective is the usual autoregressive cross-entropy loss:

L_SFT(θ) = - E_(x,y*) [ Σt log πθ(y*t | x, y*<t) ]

SFT teaches the model what a high-quality response looks like. A strong SFT policy is more than a warm start: it defines the behavior distribution from which later preference optimization operates

3.1 Preference collection

For a prompt, generate multiple responses and ask annotators to compare them. Pairwise or ranked judgments are useful because people are usually better at saying 'A is better than B' than assigning calibrated absolute scores

3.2 Reward modeling

A reward model rφ(x,y) maps a prompt-response pair to a scalar score. A common pairwise formulation is the Bradley-Terry model:

P(yw ≻ yl | x) = exp(rφ(x,yw)) / [exp(rφ(x,yw)) + exp(rφ(x,yl))]

with loss:

L_RM = -log σ(rφ(x,yw) - rφ(x,yl))

In practice, the reward model is typically initialized from the SFT model checkpoint and then fine-tuned on preference data. This initialization gives the reward model strong language understanding before it specializes on preference judgments — a model that already knows what fluent, coherent text looks like is better positioned to distinguish subtle quality differences than one trained from scratch. Reward model calibration matters: overconfident reward models produce spiked preference distributions that are easier for the policy optimizer to exploit, because the optimizer can find inputs that push the reward model into its high-confidence regime without actually improving response quality

3.3 KL-regularized policy optimization

A KL term is commonly used to constrain policy drift:

J_RL(θ) = E[R(x,y)] - β KL(πθ || πref)

In many implementations, the KL penalty is applied at the token level rather than as a single trajectory-level term. The per-token reward signal becomes:

r̂_t(x,y) = rφ(x,y) · 1[t=T] - β · log(πθ(yt|x,y<t) / πref(yt|x,y<t))

This formulation assigns the terminal reward only at the final token (t=T) while subtracting a per-step KL penalty at every token, providing a denser training signal. The KL coefficient β controls the tradeoff between reward maximization and staying close to the reference policy. Goodhart's Law applies directly here: 'When a measure becomes a target, it ceases to be a good measure.' The reward model is a proxy for human preference, and sufficiently strong optimization will find and exploit its weaknesses — this is precisely the reward hacking problem (Ouyang et al., 2022; Stiennon et al., 2020)

Because the reward model is only a proxy, sufficiently strong optimization can discover and exploit weaknesses in it

Proximal Policy Optimization constrains policy updates with a clipped surrogate objective (Schulman et al., 2017). The policy ratio is:

rt(θ) = πθ(at | st) / πθold(at | st)

The clipped objective is commonly written as:

L_CLIP = E[min(rt(θ) Ât, clip(rt(θ), 1-ε, 1+ε) Ât)]

The full PPO training objective for LLMs combines four components:

L(θ) = L_CLIP(θ) - c1 · L_VF(θ) + c2 · S[πθ] - c3 · KL(πθ || πref)

Here L_VF is the value function mean-squared-error loss that trains the critic to accurately predict expected future reward; S[πθ] is an entropy bonus that encourages the policy to maintain exploration rather than collapsing to deterministic outputs; and the KL term penalizes deviation from the reference policy. In LLM practice, the KL penalty is often incorporated into the per-token reward signal rather than the loss directly, as described in section 3.3

4.1 The critic and GAE

PPO is commonly used in an actor-critic setup. Generalized Advantage Estimation is commonly expressed as:

Â_t^GAE = Σ_l (γλ)^l δ_(t+l)

δt = rt + γV(st+1) - V(st)

The value function V_ψ(s_t) is a separate neural network — commonly sharing base transformer weights with the policy but with a distinct scalar output head — that estimates the expected future reward from the current state (the prompt plus all tokens generated so far). Sharing weights reduces total parameter count but couples the actor and critic optimization dynamics. A practical engineering constraint follows directly: actor-critic LLM training requires holding two large model copies in GPU memory simultaneously (actor and critic), plus the frozen reference model for KL computation. For a 70B-parameter model, this can require hundreds of gigabytes of GPU memory, making memory management a primary engineering constraint rather than an afterthought

Reward hacking is often the optimizer faithfully exploiting an imperfect objective rather than an optimizer malfunction

• Length exploitation
• Sycophancy
• Confident but incorrect answers
• Formatting or style gaming

RLAIF replaces or supplements human judgments with AI-generated evaluations, critiques, rankings, or constitutional checks. The upside is scale; the downside is evaluator bias and evaluator-specific failure modes

Constitutional AI (Bai et al., 2022) is a concrete RLAIF implementation developed at Anthropic. Rather than relying on human annotators to label harmful outputs, CAI uses the model itself to generate, critique, and revise responses according to a written 'constitution' — a list of harmlessness and helpfulness principles. The five-step process is:

1. The model generates an initial response to a potentially harmful prompt
2. It critiques its own response using a principle sampled from the constitution
3. It revises the response based on the critique
4. The original and revised responses form preference pairs
5. Those pairs train an AI preference model (APM) that replaces human annotators in the RLHF loop

This creates a self-improvement loop that scales preference data cheaply — the bottleneck shifts from human annotation throughput to model inference compute. Lee et al. (2023) showed that RLAIF can match or approach RLHF quality at scale, though the capability of the evaluator model sets a ceiling on signal quality: an evaluator that cannot reliably distinguish good from bad responses will produce noisy preference labels regardless of how many pairs are generated (Lee et al., 2023)

GRPO extracts a learning signal from relative performance among several responses to the same prompt. A group-normalized advantage can be written as:

Â_i = (r_i - mean(r)) / (std(r) + ε)

The full DeepSeekMath / DeepSeek-R1 formulation (Shao et al., 2024; DeepSeek AI, 2025) makes the group structure explicit. For a prompt x, GRPO samples G responses {y_1, ..., y_G} from the old policy πθ_old. A reward function scores each response: r_i = r(x, y_i). The group-normalized advantage for response i is:

Â_i = (r_i - mean({r_j}_{j=1}^G)) / (std({r_j}_{j=1}^G) + ε)

The GRPO clipped policy objective mirrors PPO's clipped surrogate but uses group-normalized advantages. KL regularization is typically applied directly to the reward rather than the loss: r̂_i = r_i - β · KL(πθ || πref). In practice, G is typically 8–16; DeepSeekMath used G=8. By removing the learned value network entirely, GRPO reduces total GPU memory by approximately 30–40% compared to standard actor-critic PPO implementations, making it attractive for large-scale reasoning model training where memory is the binding constraint. The tradeoff is higher variance in advantage estimates when G is small — with only a few samples, the group mean and standard deviation are noisy estimates of the true baseline

Outcome rewards answer whether a trajectory succeeded. Process rewards evaluate intermediate reasoning steps and can provide denser credit signals for long trajectories

Outcome reward models (ORMs) assign a single reward at the end of a trajectory — fast to train and simple to implement, but the gradient signal is sparse for long reasoning chains. A model that produces a 50-step mathematical derivation receives one reward signal at the end, and that signal must propagate backward through all 50 steps. Process reward models (PRMs) assign a reward at each reasoning step, providing denser gradient signal along the trajectory (Lightman et al., 2023)

Collecting step-level annotations is expensive. OpenAI's PRM800K dataset required human annotators to label each step of thousands of multi-step math solutions, producing 800,000 step-level labels. Automated step verification — for example, checking intermediate calculations with a symbolic solver like SymPy — can partially substitute for human annotation in domains where intermediate steps are machine-verifiable. The credit assignment advantage of PRMs is substantial: for a trajectory of T steps, gradient signal from a PRM has substantially lower variance than an ORM because credit is assigned locally rather than being propagated backward through all T token positions

Verifiable rewards are powerful when correctness can be checked mechanically, such as through software tests, mathematical verification, or proof assistants

R(x,y) = 1 if verifier(y) = correct, else 0

Verifiable rewards apply across several concrete domains:

1. Mathematics: symbolic solvers like SymPy or Mathematica can verify final answers and many intermediate steps, providing exact binary or graded feedback without human involvement
2. Code: unit test suites verify generated code against correctness, edge cases, and performance requirements — the test suite acts as a ground-truth oracle
3. Formal proofs: proof assistants like Lean 4 or Coq verify theorem proofs machine-checkably, providing exact correctness signals used in frontier math reasoning research
4. Logic puzzles and constrained games: environment simulators provide exact binary feedback, making them natural RL training environments

The key advantage of verifiable rewards is objectivity: they cannot be exploited by sycophancy or formatting games because the verifier does not care about tone, length, or style — only correctness. The key limitation is coverage: most real-world tasks lack complete machine verification, and even in verifiable domains, the verifier may not catch all failure modes (a proof assistant verifies logical validity but not whether the theorem is interesting or useful)

DPO directly optimizes a policy from preference pairs without the standard explicit reward-model-plus-online-PPO loop

L_DPO = -log σ(β[log πθ(yw|x) - log πref(yw|x) - log πθ(yl|x) + log πref(yl|x)])

The derivation starts from the KL-constrained RL problem. The optimal policy under KL regularization has a closed-form solution:

π*(y|x) = (1/Z(x)) · πref(y|x) · exp(r(x,y)/β)

where Z(x) = Σ_y πref(y|x) · exp(r(x,y)/β) is an intractable partition function that sums over all possible completions. DPO's key insight is to reparameterize the reward as r(x,y) = β · log(π(y|x)/πref(y|x)) + β · log Z(x). When computing the Bradley-Terry preference probability P(y_w ≻ y_l | x), the log Z(x) terms cancel in the ratio, yielding the tractable DPO loss without ever computing Z(x) (Rafailov et al., 2023). This is what makes DPO practical: the intractable normalization constant disappears algebraically

Several DPO variants address specific limitations of the original formulation:

• IPO (Identity Preference Optimization, Azar et al. 2023): adds a squared identity loss to avoid overconfidence on hard preference pairs, where the standard DPO loss can push log-probability ratios to extreme values
• SimPO (Meng et al. 2024): removes the reference model entirely by normalizing log-probabilities by sequence length, eliminating the need to store and query a reference model during training
• ORPO (Hong et al. 2024): incorporates preference learning directly into the SFT cross-entropy objective in a single training stage, avoiding the need for a separate preference optimization phase

Inference-time scaling spends additional compute on candidate generation, verification, search, or selection. A useful abstraction is:

generate → evaluate → select or expand → repeat

Snell et al. (2024) analyzed optimal test-time compute allocation across strategies, finding that the best approach depends on problem difficulty and the quality of the available verifier. The main strategies are:

1. Best-of-N: generate N completions independently, score each with a reward model or verifier, return the highest-scoring. Compute cost is O(N); accuracy often scales as approximately O(log N), making it effective but compute-intensive at large N
2. Beam search with PRM: maintain K candidate partial solutions; at each step extend and score with a process reward model, prune to the top K. Structured but requires step-level scoring at every generation step
3. Monte Carlo Tree Search (MCTS): tree-structured search where nodes represent reasoning states. Uses UCB (Upper Confidence Bound) to balance exploration and exploitation across branches — effective for combinatorial reasoning tasks where the search space has meaningful branching structure
4. Self-consistency voting: generate N solutions independently and take a majority vote on the final answer — effective for math where many distinct reasoning paths converge to one correct answer, requiring no trained verifier
5. Sequential revision chains: generate a draft, critique it (model or verifier), revise, repeat — effective for code debugging and iterative refinement where each revision can be verified against a test suite

A serious RL stack coordinates rollout generation, reward computation, trajectory storage, advantage calculation, gradient updates, checkpointing, and evaluation. Long reasoning traces make every stage more expensive

• Rollout throughput
• Reward latency
• GPU memory pressure
• Distributed communication
• Variance and reproducibility

Production RL stacks address these constraints with specific engineering solutions:

1. vLLM with PagedAttention for efficient, high-throughput rollout generation using KV cache memory management — paging KV cache blocks like virtual memory pages eliminates fragmentation and enables continuous batching across variable-length sequences
2. Asynchronous reward computation where rollout generation and reward model inference overlap to hide latency — the reward model scores batch N while the policy generates batch N+1
3. Reference model weight sharing — freeze reference weights in float16 and run forward-only passes (no gradients), significantly reducing memory versus maintaining a full duplicate with optimizer states
4. Gradient checkpointing during the actor backward pass, trading recomputation time for memory — activations are recomputed on the backward pass rather than stored, reducing peak memory at the cost of roughly 30% additional compute
5. Ring-allreduce across multi-node setups for distributed gradient aggregation, minimizing communication overhead by passing gradient shards in a ring topology rather than routing everything through a parameter server
6. Multi-epoch PPO updates — reusing a batch of rollouts for several gradient steps (PPO epochs) before discarding, amortizing expensive rollout generation cost. The clipping mechanism limits how far the policy can move per epoch, keeping the reused data approximately on-policy
7. On-policy vs offline distinction: PPO and GRPO require fresh rollouts each iteration (on-policy), making rollout throughput the primary bottleneck; DPO trains on a fixed offline dataset, eliminating rollout generation entirely but requiring a high-quality preference dataset collected in advance

These terms live at different layers rather than being mutually exclusive competitors

• RLHF / RLAIF: feedback source
• PPO / GRPO: policy-optimization mechanisms
• ORM / PRM / verifier: reward or evaluation mechanisms
• DPO: direct offline preference optimization

Before tuning the optimizer, validate the objective and evaluation stack

1. Define the behavior you want to improve
2. Choose the feedback source
3. Keep an independent evaluation set
4. Establish an SFT baseline
5. Monitor KL divergence, reward, entropy, response length, and independent task metrics
6. Red-team the reward model or evaluator
7. Stress-test distribution shift and out-of-distribution behavior
8. Only then tune learning rate, clipping, KL control, sampling, and rollout parameters

• RL for LLMs treats text generation as policy optimization over token sequences
• RLHF combines preference data, reward modeling, and policy optimization
• PPO constrains policy updates with a clipped objective
• GRPO derives relative learning signals from groups of sampled responses
• DPO directly learns from offline preference pairs
• Verifiable rewards are especially powerful where correctness can be machine-checked