Policy Optimization for LLMs:
From One Loss to Scalable Recipes

One Scalar Loss, One Chain Rule

Training = minimize a single scalar $\mathcal{L}(\theta)$, connected to all parameters through a differentiable computational graph.

SFT — Where Differentiability Holds

Gold sequence $y^*$ exists in the training data:

The RL Objective

No gold sequence. The model must sample tokens, then receive a reward. We need to optimize over all possible outputs, weighted by their probability:

• We want $\nabla_\theta J(\theta) = \sum_y r(y) \cdot \nabla_\theta \pi_\theta(y)$ — sum over $|V|^T$ possible sequences. Completely intractable.
• This is why REINFORCE is needed.

REINFORCE — The Log-Derivative Trick

Start from the intractable gradient $\nabla_\theta J = \sum_y r(y) \cdot \nabla_\theta \pi_\theta(y)$:

• Step 1. Multiply-divide by $\pi_\theta(y)$:   $= \sum_y r(y) \cdot \pi_\theta(y) \cdot \frac{\nabla_\theta \pi_\theta(y)}{\pi_\theta(y)}$
• Step 2. Apply $\nabla f / f = \nabla \log f$:   $= \sum_y \pi_\theta(y) \cdot r(y) \cdot \nabla_\theta \log \pi_\theta(y)$
• Step 3. Recognize as expectation:   $\nabla_\theta J = \mathbb{E}_{y \sim \pi_\theta}[r(y) \cdot \nabla_\theta \log \pi_\theta(y)]$

Now estimate the true gradient by sampling $N$ sequences and averaging:

• Surrogate loss (what the optimizer minimizes):   $\mathcal{L}(\theta) = -\frac{1}{N}\sum_i r_i \cdot \log \pi_\theta(y_i)$

REINFORCE — Intuition and Credit Assignment

$\nabla_\theta \log \pi_\theta(a_t)$ = "which direction" (make $a_t$ more likely).   $r$ = "how much" to push.

• REINFORCE baseline: batch mean $\bar{r}$ across all prompts.

PPO — Reuse Rollouts via Importance Sampling

REINFORCE problem: each rollout used once, then discarded. PPO reuses rollouts for multiple optimizer steps. But after the first step, the policy $\pi_\theta$ has changed while the data was generated under the old $\pi_{\theta_\text{old}}$ — the rollouts are now off-policy.

• After the first update, data was generated under $\pi_{\theta_\text{old}}$, not $\pi_\theta$. IS (Importance Sampling) ratio: $\rho_t = \pi_\theta(a_t \mid s_t) / \pi_{\theta_\text{old}}(a_t \mid s_t)$

• Gradient (inside clip): $\nabla_\theta J^\text{PPO} = \mathbb{E}_t[A_t \cdot \rho_t \cdot \nabla_\theta \log \pi_\theta(a_t)]$.   At $\theta = \theta_\text{old}$: $\rho_t = 1$, reduces to REINFORCE.
  • $\nabla_\theta \rho_t = \frac{\nabla_\theta \pi_\theta}{\pi_{\theta_\text{old}}} = \frac{\pi_\theta}{\pi_{\theta_\text{old}}} \cdot \frac{\nabla_\theta \pi_\theta}{\pi_\theta} = \rho_t \cdot \nabla_\theta \log \pi_\theta$   ($\pi_{\theta_\text{old}}$ is a frozen constant.)
• When clip active: the clipped $\rho_t$ is replaced by a constant $(1 \pm \epsilon)$ — no longer a function of $\theta$. Differentiating a constant w.r.t. $\theta$ gives zero. Gradient killed entirely.

PPO — GAE and the Value Model

Value model $V_\phi(s_t)$: a separate neural network that predicts expected future return from position $t$. "How much total reward do I expect from here to the end?" Trained by regression: $\mathcal{L}_V = (V_\phi(s_t) - R)^2$.

TD (Temporal Difference) residual: $\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$. Compares the value model's prediction before and after one token. If $V$ jumped up at token $t$, that token was better than expected ($\delta_t > 0$). If $V$ dropped, that token hurt ($\delta_t < 0$). Per-token credit assignment.

GAE (Generalized Advantage Estimation): $A_t^\text{GAE} = \sum_{l=0}^{\infty}(\gamma\lambda)^l \delta_{t+l}$. Exponentially weighted sum of TD residuals from token $t$ onward. Nearby $\delta$'s are weighted heavily, distant $\delta$'s decay exponentially. $\lambda$ controls how far ahead to look:

• $\lambda = 0$: use only $\delta_t$. Sharp per-token credit, but fully trusts $V$.
• $\lambda = 1$: sum all future $\delta$'s equally, which reduces to Monte Carlo $R - V(s_t)$.
• $\lambda = 0.95$: practical sweet spot. Mostly per-token, with some smoothing.

PPO — Memory and Context

PPO was the standard for RLHF (InstructGPT-style). With learned RM, 5 models on GPU:

RLVR (verifiable rewards, math/code) changes the picture:

• Reward is rule-based and exact → no RM (drop #5). Low reward hacking risk.
• RLVR recipes omit the critic entirely → no $V_\phi$ (drop #4).

GRPO — Drop the Critic

GRPO removes the value model. Sample $G$ completions per prompt, use group statistics as baseline.

• Gradient (inside clip): $\nabla_\theta J^\text{GRPO} = \frac{1}{G}\sum_i A_i \cdot \frac{1}{T_i}\sum_t \rho_{i,t} \cdot \nabla_\theta \log \pi_\theta(a_{i,t})$
  • Same structure as PPO. Gradient dies when clip active.

What's Left to Fix?

• Problem 1: Zero-gradient prompts (all correct or all wrong) → DAPO, ScaleRL
• Problem 2: Trust region issues
  • 2a. Symmetric clip too tight for low-prob tokens → DAPO
  • 2b. Gradient death at clip boundary → CISPO, SAPO
  • 2c. Ratio measures the wrong thing → DPPO
• Problem 3: Per-token gradient magnitude bias
  • 3a. $\sigma$ normalization inflates easy prompts → Dr. GRPO, ScaleRL
  • 3b. Loss aggregation bias (length) → ScaleRL
• Problem 4: MoE token-level ratio instability → GSPO

Problem 1 — Zero-Gradient Prompts

If all $G$ rollouts for a prompt are correct (or all wrong): $r_i - \mu_G = 0$. Zero advantage. Zero gradient. Prompt contributes nothing.

ScaleRL verdict: filtering is preferred. Permanent retirement of easy prompts (≥ 0.9) progressively shifts the training distribution toward harder problems as the model improves — an adaptive curriculum that dynamic sampling lacks.

Problem 2a — Symmetric Clip Too Restrictive

Token at probability 0.01 can only rise to 0.012 before clipping ($\epsilon = 0.2$). Barely changes sampling likelihood. Critical low-probability tokens ("Wait," "However") get suppressed.

• Asymmetric clipping (DAPO): $\epsilon_\text{low} = 0.2$, $\epsilon_\text{high} = 0.28$. Positive-advantage tokens get more room.

• Gradient (inside clip): $\nabla_\theta J^\text{DAPO} = \frac{1}{\sum_i T_i}\sum_i A_i \sum_t \rho_{i,t} \cdot \nabla_\theta \log \pi_\theta(a_{i,t})$
• Wider asymmetric bounds mean fewer tokens hit the boundary. But when they do, same problem: $\nabla_\theta \rho_{i,t} = 0$.

Problem 2b — Gradient Death at Trust Region Boundary

PPO/GRPO/DAPO: gradient flows through $\rho_t$. When clip active, $\rho_t$ becomes constant → $\nabla_\theta \rho_t = 0$. The most informative tokens (largest probability shift) are the first to be clipped.

Problem 2c — The Ratio Measures the Wrong Thing

The ratio measures relative change at one token, not actual distributional shift.

DPPO: replace ratio-based clip with a divergence-based mask. The paper uses efficient Binary and Top-K approximations to estimate policy divergence with negligible overhead. A sampled-action proxy used inside efficient divergence estimation:

• $D_\text{TV}(t) \approx |\pi_\theta(a_t) - \pi_{\theta_\text{old}}(a_t)|$ — tokens exceeding threshold $\tau$ are masked out.

• Gradient: $\nabla_\theta J^\text{DPPO} = \mathbb{E}[M_\text{div}(D_\text{TV}, \tau) \cdot A_t \cdot \rho_t \cdot \nabla_\theta \log \pi_\theta(a_t)]$
• Masking criterion based on actual distributional shift, not ratio value. < 0.5% of updates cause instability.

Problem 3a — Normalization Bias

GRPO normalizes advantage by group standard deviation: $A_i = (r_i - \mu_G)/\sigma_G$.

ScaleRL finding: prompt-level, batch-level, and no $\sigma$ normalization all yield similar performance. Batch-level adopted.

• Prompt-level $\sigma$: each prompt divided by its own $\sigma_G$. Different prompts get different divisors, distorting relative gradient magnitudes across prompts.
• Batch-level $\sigma$: first compute $A_i = r_i - \mu_G$ per prompt (no $\sigma_G$ division), then divide all advantages across the entire batch by one shared std. Same divisor for everyone — just a global scale factor that does not distort relative magnitudes.

Problem 3b — Loss Aggregation Bias

How tokens are weighted across sequences determines which outputs dominate the gradient.

ScaleRL verdict: prompt-level aggregation achieves highest asymptotic performance.

Problem 4 — MoE Token-Level Ratio Instability

MoE models: expert routing changes between $\pi_{\theta_\text{old}}$ and $\pi_\theta$. ~10% of activated experts differ after one gradient update (Qwen3-30B-A3B). Individual token ratios $\rho_t$ fluctuate wildly.

• GSPO (Qwen3): average log ratios across the sequence → one $\rho_i$ per sequence:

• Gradient: $\nabla_\theta J^\text{GSPO} = \frac{1}{G}\sum_i A_i \cdot \rho_i \cdot \frac{1}{T_i}\sum_t \nabla_\theta \log \pi_\theta(a_{i,t})$
  • Token-level noise averages out (law of large numbers). One clip decision per sequence.
  • Substantially stabilized MoE RL training without Routing Replay.
• Tradeoff: outlier tokens can suppress entire sequence gradient, or be masked by normal tokens.

Methods ↔ Problems Summary

Three Structural Axes

ScaleRL — The Scaling Framework

• 400k+ GPU-hours of systematic ablations across RL design choices (Meta / UT Austin, 2025).
• Fits sigmoidal curves to predict the ultimate performance ceiling of an RL recipe from early-stage runs, without full runs.
• Validated by running 100K GPU-hours to convergence, showing a fit on the first 50K accurately predicts the remaining trajectory.

ScaleRL Key Findings

The ScaleRL Recipe

One concrete instantiation combining the study's best practices:

Which Models Used What?

Training Pipeline Context

Policy Optimization is one stage in a multi-stage pipeline. Common pattern:

• Trend: later models expand RL scope. R1 and Qwen3 focus on reasoning RL. GLM-5 adds a dedicated agentic RL stage. MiniMax-M2.5 scales RL across 200k+ real-world environments (code, office, web).
• Small models skip RL entirely. Both DeepSeek-R1-Distill and Qwen3 small use distillation from flagship instead. Qwen3 explicitly claims distillation outperforms RL for smaller models; DeepSeek-R1 reports it for the Qwen2.5-32B case.
  • NVIDIA's AceReason-Nemotron shows distillation → RL works in small models: GRPO on top of DeepSeek-R1-Distill-Qwen-7B yields +14.6pp on AIME 2025 and +6.8pp on LiveCodeBench (7B, math-only RL).

Takeaways

• One equation underlies everything: $\nabla_\theta J = \mathbb{E}[\nabla_\theta \log \pi_\theta \cdot A]$. All methods are strategies for making this gradient more accurate, lower variance, or more stable.
• Critic-free training won for RLVR. Per-token credit (PPO/GAE) lost to simplicity + scale.
• Softer trust regions beat hard clipping. CISPO's "never kill gradient" outperforms PPO-style masking and is most robust to hyperparameters (ScaleRL).
• Not all recipes scale equally. Loss type and FP32 precision shift the ceiling $A$. Most other choices only modulate efficiency $B$.
• $\mathcal{L}(\theta)$: one scalar, one chain rule. Everything since slide 2 is about choosing the right scalar.