RL for LLMs: GRPO, DAPO & R1-Zero Training

Chapter 1: PPO: A Critic For Every Token

Chapter 0 asserted that GRPO deletes a critic network. To see why that deletion is possible — and what it costs — you first need PPO's machinery in full, built from the ground up, not taken on faith.

The problem PPO's clip is solving

Start from the most basic policy-gradient idea: to make good actions more likely, nudge the policy's parameters in the direction of the gradient of log-probability, scaled by how good the action turned out to be (its advantage, written Â_t). Do this update over and over with freshly sampled data, and the policy improves. The trouble is scale: a single large step, taken on noisy, high-variance reward signal, can wreck the policy — push it somewhere so different from where it started that the next batch of sampled data is no longer representative of what the updated policy will actually do, and training becomes unstable or collapses outright.

Proximal Policy Optimization (PPO) fixes this with a clipped surrogate objective:

Unpack it piece by piece. r_t(θ) is the importance ratio — how much more (or less) likely the current policy π_θ is to produce token o_t than the policy that actually generated the data, π_θold:

If r_t=1.5, the new policy is 50% more likely to say this token than the policy that generated the training data did. The clip function then clamps this ratio into the window [1−ε, 1+ε] — typically ε=0.2, so the ratio is not allowed outside [0.8, 1.2] for the purposes of this term. Taking the minimum of the clipped and unclipped versions means: if the advantage is positive (this token was good) and the ratio wants to grow past 1+ε, the clip caps how much credit that step can take for it. If the advantage is negative, clipping caps how much the policy can be dragged down for a token whose probability had already fallen a lot. Either way, the effect is the same — keep every single update inside a "trust region" close to the policy that generated the data, no runaway steps.

A worked ratio, so the clip is not just abstraction

Put a number on it. Suppose token o_t was assigned probability 0.04 by the policy that generated the training data, π_θold(o_t|q,o<t)=0.04, and after one gradient step the current policy assigns it 0.07. The importance ratio is:

With ε=0.2, the clip window is [0.8, 1.2]; 1.75 sits well outside it, so clip(r_t(θ),0.8,1.2) evaluates to 1.2, not 1.75. If this token's advantage was positive (Â_t=+1, a good token), the unclipped term contributes 1.75×1=1.75 while the clipped term contributes 1.2×1=1.2, and PPO's min() selects 1.2 — the smaller of the two. The extra 0.55 of "credit" the raw ratio wanted to claim for this update never enters the loss.

Now flip only the advantage's sign, keeping the same ratio: Â_t=−1 (this token turned out, in hindsight, to be a bad one). Unclipped: 1.75×(−1)=−1.75. Clipped: 1.2×(−1)=−1.2. Taking the min of −1.75 and −1.2 now selects −1.75, the more negative of the two, because min() always picks the smaller number and −1.75 is smaller than −1.2. This asymmetry is worth noticing early: the clip only ever caps how much the objective can reward a ratio moving in the direction the advantage already wants — it never caps how much a ratio can be penalized for moving the wrong way. Chapter 6 returns to exactly this ratio-versus-probability distinction, at much larger scale, to explain a specific training failure.

python
def ppo_clip_loss(logp_new, logp_old, advantages, eps=0.2):
    # logp_new, logp_old: log pi_theta(o_t), log pi_theta_old(o_t), one per token
    ratio = (logp_new - logp_old).exp()          # r_t(theta) = pi_theta / pi_theta_old
    unclipped = ratio * advantages
    clipped   = ratio.clamp(1 - eps, 1 + eps) * advantages
    return -torch.min(unclipped, clipped).mean()  # negative: we ASCEND the objective

Notice the ratio is computed in log-space and exponentiated, not divided directly — standard practice, since subtracting log-probabilities and exponentiating is numerically safer than dividing two very small probabilities against each other. And notice the final negation: ppo_clip_loss returns something meant to be minimized by an optimizer, so the objective this session has been calling J(θ), which is meant to be maximized, gets flipped in sign exactly once, right at the very last line, and nowhere else in the pipeline.

Where the advantage comes from: GAE and the critic

That Â_t in the equation above cannot be pulled out of thin air — it has to be estimated, and PPO's standard estimator is Generalized Advantage Estimation (GAE):

Read δ_l first, because it's the atomic unit everything else is built from: it's a temporal-difference error — the reward you actually got at step l, plus your value function's guess about everything still to come from the next state, minus your value function's guess about everything still to come from the current state. If that number is positive, this particular step did better than your value function expected; if negative, worse. GAE then sums a discounted, exponentially-weighted combination of these one-step surprises into a full advantage estimate for every single token.

Notice what this requires: V, the value function — a prediction, from any partial state, of how much total future reward to expect. In LLM RL, this value function is a second neural network, the critic, typically initialized from the same pretrained model and trained alongside the policy (the actor) at every step, on its own regression loss, predicting the eventual return from every token position in every sampled response.

GAE by hand: a three-step toy trajectory

Make δ_l concrete with a tiny worked example: three consecutive token positions in one response, with toy rewards and value predictions standing in for what a critic's forward pass would actually output. Let γ=0.99, λ=0.95, rewards R=[0, 0, 1] (all the reward arrives on the final token, exactly as it would for an outcome-reward math problem), and suppose the critic predicts values V=[0.6, 0.7, 0.9] at steps 0, 1, 2, with V(s₃)=0 past the end of the response:

Fold these one-step surprises into the actual advantage estimate, working backward from the last step — exactly what the gae() function below does with its reversed loop:

Notice how much of the final estimate leans on the critic's own predictions, not just the raw rewards: two of the three reward entries were exactly zero, yet every one of the three advantage estimates came out positive and non-trivial — entirely because the critic's value predictions were themselves rising (0.6 → 0.7 → 0.9). The critic was already anticipating the reward that eventually arrived, and GAE credits every step along the way for that anticipation being correct. If the critic's predictions had been wrong instead — say it predicted a flat 0.1 at every step, never anticipating the reward at all — these same three δ values, and the resulting  values, would come out completely different, even though the actual rewards R never changed. This is precisely the fragility Chapter 0 hinted at: the advantage estimate PPO trains against is only as good as the critic producing V, and a critic trained on noisy, long-horizon chain-of-thought data can get this systematically wrong long before it ever converges.

Back-of-envelope: what "doubling" means at DAPO's own scale

"Roughly doubling memory footprint" is easy to say and easy to gloss over. Pin it to the actual model this session keeps returning to: Qwen2.5-32B, the base model both DAPO and DeepSeek-R1-Zero-Qwen-32B train from (Chapter 9 works through both results in full). Training with PPO on a 32-billion-parameter policy means also holding a same-sized, 32-billion-parameter critic in memory, together with both networks' own AdamW optimizer states — roughly two extra copies of each network's parameter count, for the running first- and second-moment estimates — on top of the activations from two separate forward-and-backward passes per training step, one for the actor and one for the critic. None of this compute or memory goes toward making the policy itself better at reasoning; all of it goes toward solving a credit-assignment problem the critic exists purely to solve. This is the concrete cost GRPO is about to delete in the next chapter, not a vague inefficiency but a second 32-billion-parameter network's worth of GPU memory and compute, every single step, for the entire training run.

Why the critic starts from the same pretrained weights as the policy

One detail glossed over above: the critic isn't trained from a random initialization. It's typically initialized from the exact same pretrained checkpoint as the policy itself, then fine-tuned into a value function during RL. The reasoning is practical, not decorative: a value function that has to predict "how good is this partial mathematical reasoning trace" benefits enormously from already understanding language and mathematical notation before its very first gradient step, rather than learning both "what tokens mean" and "how to predict reward" from scratch at the same time. This is also exactly why the critic's cost is usually quoted as "roughly doubling" rather than "roughly tripling" or worse — it starts life as a full copy of an already-expensive pretrained model, not as a cheap auxiliary network bolted on the side.

In code, so the two networks are undeniable

python
# PPO for LLM RL needs TWO forward+backward passes per step: policy AND critic
policy_logits = actor(prompt_and_response)          # (B, T, vocab) -- what to say
value_preds   = critic(prompt_and_response)          # (B, T)        -- how much reward is left

# GAE needs value_preds to compute every advantage estimate:
def gae(rewards, values, gamma=0.99, lam=0.95):
    T = len(rewards)
    adv = [0.0] * T
    last_gae = 0.0
    for t in reversed(range(T)):
        next_value = values[t+1] if t+1 < T else 0.0
        delta = rewards[t] + gamma * next_value - values[t]
        last_gae = delta + gamma * lam * last_gae
        adv[t] = last_gae
    return adv

# the critic is trained too, on its own loss, every single step:
critic_loss = ((value_preds - returns) ** 2).mean()

Every one of those tensors on the critic side — value_preds, critic_loss, the critic's own optimizer state — is compute and memory that exists purely to solve the credit-assignment problem GAE needs solved. GRPO's entire pitch, coming up in Chapter 2, is: for a task where you can generate several complete answers to the exact same question and just check which ones were right, you don't need any of this. The group itself tells you what “good” looked like for that specific question.

Two edge cases that clarify what ε is actually doing

Push ε to its two extremes to see what the clip is really controlling. At ε=0, the clip window collapses to a single point, [1,1] — any update where the ratio moves at all away from exactly 1 gets fully clipped, and training would barely move, since virtually every real gradient step budges the ratio away from exactly 1 almost immediately. At ε→∞, the clip window covers the entire real line, clip() never binds no matter how far the ratio strays, and the objective reduces to the plain unclipped policy gradient this chapter opened with — the exact instability that motivated PPO's clip in the first place. ε=0.2 sits deliberately between these two failure modes: tight enough to bound how far a single update can move the policy, loose enough that a typical, well-behaved update isn't clipped away to nothing.

What stays, what goes

Hold two things in mind as GRPO gets built next chapter, because it keeps one piece of PPO whole and discards another. It keeps the clipped surrogate objective exactly — the importance ratio, the min-of-clipped-and-unclipped trick, the trust-region idea. What it discards is how the advantage Â_t is computed: no GAE, no value function, no critic network at all. Everything else about this chapter's equation carries over unchanged into the next one; only the source of  changes.

Chapter 2: GRPO: Deleting the Critic

Chapter 1 ended with a question: for a task where you generate several answers to the same question and just check which ones are right, do you really need a trained network guessing at future reward from every partial sequence? Group Relative Policy Optimization (GRPO), introduced in DeepSeekMath and used as the baseline algorithm in both papers this session studies, answers no.

Where the group's role comes from: the score-function gradient and baseline subtraction

Before writing GRPO's formula down, it's worth deriving why any baseline subtraction is safe, rather than taking Chapter 1's advantage-weighted gradient on faith. The starting point for essentially every policy-gradient method, PPO and GRPO included, is the score-function estimator: the gradient of expected reward can be written as an expectation over the gradient of log-probability, weighted by the return:

This identity keeps holding for any constant b subtracted from R(o), so long as b does not depend on the specific action o being scored:

The second term on the right is exactly zero, for a reason worth internalizing rather than memorizing: E[∇log π_θ(o)] is the expected gradient of a properly normalized probability distribution, and the gradient of a distribution that always integrates to 1 must itself integrate to 0 — there is nowhere else for that probability mass to go. Because that term vanishes identically, subtracting any action-independent b changes nothing about the true gradient, only about the variance of any single sampled estimate of it: a well-chosen b, close to the typical value of R(o), shrinks the spread of (R(o)−b) across samples without touching its expectation. PPO's critic-based value function V(s) is one choice of b. GRPO's group mean, built next, is another. Both are legitimate baselines for exactly this one reason.

The group-relative trick

For one question-answer pair (q, a), sample a group of G complete responses {o₁, …, o_G} from the old policy, all answering the exact same question. Score each one with the rule-based reward from Chapter 0, giving a group of rewards {R₁, …, R_G}. Now, instead of a critic predicting what “average” reward to expect, just compute it directly from the group itself:

Read this the way you'd read a grade curve. Every response in the group gets compared not against some absolute standard, but against how the other attempts at the exact same question did. A response that scored better than the group average gets a positive advantage; worse than average, negative. The std in the denominator rescales this difference into a roughly-standardized unit, so a question where responses varied wildly doesn't automatically produce huge-magnitude advantages just because of that spread. Notice, crucially: every token in response o_i gets the same advantage Â_i,t — it doesn't depend on t at all, because the reward is an outcome reward, one number per whole response, not a per-token signal. No value function anywhere in this formula.

The full objective, assembled

Plug this group-relative advantage into the exact clipped surrogate objective from Chapter 1, average over both the tokens in each response and the responses in the group, and this is GRPO's full training objective:

Two structural details worth naming now, because Chapters 4 and 6 both come back to reopen exactly these two spots. First: notice the double normalization — (1/G) averages across the G samples in the group, and (1/|o_i|) averages across the tokens within each individual sample, before that per-sample average gets folded into the group average. This is called sample-level loss, and Chapter 4 derives a real problem with it. Second: this equation still carries the βD_KL term inherited from RLHF — the next section explains why DAPO removes it entirely.

Hand-working a group of four

Formulas earn their keep by being computable by hand. Take a small, concrete group: G=4 sampled responses to one question, using DAPO's ±1 reward convention (correct = +1, incorrect = −1). Suppose two are correct and two are incorrect: R = [1, 1, −1, −1].

One convention detail matters here and will matter again later: this lesson computes standard deviation with Bessel's correction (dividing the variance by G−1, not G) to match PyTorch's torch.std() default, which is what production GRPO implementations actually call. Divide by G and you'd get slightly different numbers — the qualitative story is identical either way, but for the arithmetic to reproduce what code would output, ddof=1 is the right convention.

Now change the mix, keeping G=4 but making the group less balanced: three correct, one incorrect, R = [1, 1, 1, −1].

Sit with that second result for a moment, because it's the seed of a problem Chapter 4 develops in full: the one incorrect response in the mostly-correct group gets a much larger-magnitude advantage (−1.5) than any individual sample did in the balanced group (±0.866), purely because it was the rare outcome in a group with a smaller spread. Nothing about how wrong that response was changed — only how rare being wrong was within its group.

The degenerate case: what happens at 4 correct, 0 incorrect

Push the same group to the extreme: all four responses correct, R = [1, 1, 1, 1].

Every single advantage in this group is (1−1)÷0 — division by zero. In practice this is implemented as identically zero advantage for the whole group (a small ε is usually added to the denominator to avoid a literal crash, but the effective advantage rounds to nothing). Zero advantage means zero gradient: this entire batch of four generated responses, however much compute it cost to sample and score, does nothing to update the policy. Hold onto this specific failure — Chapter 7 builds an entire technique, Dynamic Sampling, around exactly this observation.

The smallest possible group, and the group size that can't work at all

Push group size to its own smallest edge cases. At G=2, one correct and one incorrect, R=[1,−1]: mean=0, and with Bessel's correction the variance is (1²+1²)÷(2−1)=2, so std=√2≈1.414. Both advantages come out at ±1÷1.414≈±0.707 — GRPO still works at its smallest meaningful group size, it just has the fewest possible samples to estimate a mean and spread from, so any single run's estimate is at its noisiest here. G=1 is a different story entirely: a "group" of one sample has no spread to measure at all, std(·) of a single number is always exactly 0 by definition, and the advantage formula divides by zero on every single sample, correct or not. GRPO's entire mechanism structurally requires G≥2; there's no degenerate-but-usable version of it at G=1, unlike PPO with a critic, which works fine generating even a single response per prompt, because its baseline never depended on a group to begin with.

Removing the KL penalty

Chapter 0 flagged the βD_KL term still sitting in GRPO's objective, inherited straight from RLHF. In the classic RLHF setting, that term exists to keep the policy from drifting so far from its starting point that a learned, imperfect reward model's judgments stop being trustworthy — the model needs to stay “close enough” to the distribution the reward model was validated on.

DAPO's reasoning for dropping it entirely: during long chain-of-thought RL, the model's distribution is supposed to diverge significantly from where it started — that divergence is the whole point, it's how the model learns genuinely new, longer reasoning behaviors. And because the reward here is a fixed, rule-based check rather than a learned model, there is no reward-model-validity region to protect by staying close to the reference policy. The restriction that made sense for RLHF's failure mode simply doesn't apply to this one, so it's removed:

That's DAPO's full objective, minus the βD_KL term — and notice the normalization already looks different from GRPO's: (1/∑|o_i|) instead of (1/G)(1/|o_i|), and ε_low/ε_high instead of one shared ε. Both of those changes are deliberate fixes, not typos — Chapters 6 and 7 derive exactly why each one is there.

The whole family, side by side

It's easy to lose track of exactly what changed between algorithm names as this session moves forward. Before leaving GRPO behind, line up every algorithm this session touches against the same four questions: where does the advantage come from, does it need a critic, does it keep the KL penalty, and how is the loss normalized?

Read down the "advantage source" column and the whole arc of this session is really one throughline: every algorithm in this table is the same score-function estimator from Chapter 2's very first derivation, with a different, increasingly cheap choice of baseline plugged in. PPO's baseline is the most expensive to compute (a full second network) and the most general-purpose (works for any reward, learned or not). GRPO's baseline is nearly free (arithmetic on a handful of sampled rewards) but only well-defined when you can sample several complete responses to literally the same question, which the outcome-reward, verifiable-answer setting this session studies makes trivially available.

Chapter 3: What R1-Zero Actually Rewards

DeepSeek-R1-Zero's headline claim is that pure RL, with no supervised fine-tuning, teaches a base model to reason. Before accepting that claim at face value, Understanding R1-Zero-Like Training: A Critical Perspective asks a more careful question: what, precisely, is the base model contributing before RL ever starts, and what does RL actually add on top of it?

The model roster this chapter's findings are built on

Before getting into any single result, it's worth knowing exactly which models this analysis actually ran, since the findings ahead are specific claims about specific models, not a claim about "base models" as an undifferentiated category:

Two of these six — the Qwen2.5-Math pair — are where the “no template beats every template” finding lives. The other four exist specifically to check that this isn't a universal property of all base models, which the answering-rate results already ruled out: Llama and DeepSeek models both improve with a template, the opposite direction from Qwen2.5-Math.

Templates determine whether a base model even tries to answer

A pretrained base model's actual training objective is sentence completion — predict the next token, period. It was never explicitly taught the shape of “here is a question, now answer it.” A prompt template wraps the raw question in surrounding text that nudges the model into an answering posture. The paper tests three: an R1 template (a full system-style instruction plus a forced <think> tag to open the response), a Qwen-Math template (a shorter chat-formatted instruction), and no template at all — the bare question, nothing else.

A conversation between User and Assistant.
The User asks a question, and the Assistant
solves it. The Assistant first thinks about
the reasoning process in the mind and then
provides the User with the answer. The
reasoning process is enclosed within
<think> </think> and answer is
enclosed within <answer> </answer>
tags.
User: {question}
Assistant: <think>

<|im_start|>system
Please reason step by step, and put your
final answer within \boxed{}.<|im_end|>
<|im_start|>user
{question}<|im_end|>
<|im_start|>assistant

Template 3, no template at all, is simply {question} — the bare text of the problem, with nothing wrapped around it. The gap between the first two is not just length: the R1 template forces a specific <think>…</think> reasoning-then-answer structure onto every response before generation even starts, while the Qwen-Math template only asks for a boxed final answer, with no constraint on how the model gets there.

How "answering rate" is actually measured

"Answers questions far more reliably" is a claim that needs a measurement behind it, not just an impression. The method: generate each base model's responses under No template first, then hand every response to GPT-4o-mini and ask it to judge whether the response is in an answering format — regardless of whether the given answer is correct — versus in a sentence-completion pattern, simply continuing the text as if the question were the start of a paragraph rather than a prompt to respond to. The fraction judged to be in an answering format is the answering rate reported for each model. Separately, each base model's pass@8 — whether at least one of 8 independently sampled responses to a question reaches the correct final answer — is used as an exploration ability metric: it measures whether the base policy's sampling distribution ever reaches a correct trajectory at all, which matters directly for RL, because a policy that never samples a correct response to a given question has nothing for the reward signal to reinforce on that question, no matter how many training steps run.

Across Llama-3.1-8B, DeepSeek-Math-7B, and DeepSeek-V3-Base, the pattern is what you'd predict: these models answer questions far more reliably with a template than without one, and the R1 template works best for them. Without any template, DeepSeek-V3-Base has the lowest answering rate of every model tested — which the authors read as a signal that it is close to a “pure” base model, one that hasn't picked up much conversational structure just from its pretraining mix.

That two-row split is the whole confound in miniature: whichever family a given paper's base model belongs to changes not just its starting accuracy, but which prompting condition even counts as a fair, representative baseline to RL-tune from in the first place.

Qwen2.5 breaks the pattern, and that's the interesting part

Qwen2.5 base models do something different: they answer questions best with no template at all, reaching 100% answering rate that way. Pushed further, on Qwen2.5-Math-7B evaluated on five benchmarks (AIME24, AMC, MATH500, Minerva Math, OlympiadBench), the no-template condition beats every templated condition by a wide margin:

The same pattern shows up at smaller scale too, not just for the 7B model. Qwen2.5-Math-1.5B, evaluated the same way across the same five benchmarks:

Same ordering, at a model roughly a fifth the size: the R1 template is the worst-performing condition (though not quite the total collapse to 0.0 seen at 7B), the bare question is again the strongest, and every conclusion drawn from the 7B numbers holds at 1.5B too. Whatever is happening in Qwen2.5-Math's pretraining data that makes a template counterproductive is a property of the model family's data mix, not a coincidence specific to one model size.

Read the top and bottom rows together: the R1 template — the exact template DeepSeek-R1-Zero itself uses — drives this particular model's average score to zero. And the plain, unadorned question scores higher than every prompted variant, an improvement of roughly 60% over standard 4-shot prompting. The paper's working hypothesis: Qwen2.5-Math was likely pretrained directly on concatenated question-answer text, so the model has effectively already learned “question, then answer” as its native completion pattern — a template doesn't help that model complete a question-answer pair, it actively fights against a format the model already knows.

The “Aha moment” is already there before RL starts

One of DeepSeek-R1-Zero's most-discussed findings is the emergence, during RL training, of what the original report calls an “Aha moment”: the model spontaneously learning to pause, reconsider a step, and say something equivalent to “wait, let me re-examine this.” Prior work had already raised doubts about whether this is genuinely new behavior versus something the base model already knew, because the specific open-source base models tested already contained self-reflection keywords. But none of that prior work had actually tested DeepSeek-V3-Base — the specific base model the real DeepSeek-R1-Zero was RL-tuned from.

This paper closes that gap directly: hosting DeepSeek-V3-Base themselves and running it, with the R1 template, on 500 questions from the MATH dataset. The result — DeepSeek-V3-Base already generates a decent amount of self-reflection, including outputs containing keywords like “Aha,” “wait,” and “verify the problem,” before any RL training has touched it at all. The capability for self-reflective language exists in the base model's pretraining distribution; RL's job, on this evidence, looks more like amplifying and reinforcing a latent behavior than inventing it from nothing.

Why no one had tested this before: the model itself is the obstacle

It's worth pausing on why “no prior study had tested DeepSeek-V3-Base” was even possible as a gap in the literature, rather than something anyone could simply go check on a laptop. DeepSeek-V3-Base is a 685-billion-parameter model — hosting it at all, let alone running a controlled 500-question evaluation against it, requires infrastructure most academic groups don't have sitting around. Closing this gap required going out and hosting DeepSeek-V3-Base-685B specifically, which is worth noting as a methodological point in its own right: a claim as consequential as “the Aha moment might already be latent in the base model” cannot be settled by testing convenient, smaller open-source stand-ins and assuming the finding generalizes upward — it required someone to actually go test the model in question, at its actual scale.

One more honest wrinkle worth carrying forward: comparing DeepSeek-R1-Zero's own outputs to the base model's on the same questions, self-reflection behaviors occur more frequently after RL — but more frequent self-reflection does not reliably track with higher accuracy. The behavior increasing is not the same claim as the behavior helping, and the paper is explicit that these two things need to be measured separately, not conflated. This more-frequent-but-not-more-accurate finding is spelled out in the paper's own dedicated comparison between DeepSeek-V3-Base and DeepSeek-R1-Zero on identical MATH-dataset questions — and it's exactly the finding Chapter 9's closing case study returns to, from the other side, when a real DAPO training transcript is shown mid-training doing precisely this kind of self-correction on a live geometry problem.

So what does RL actually add, if not the raw capability?

Put the base-model findings together and a more precise picture emerges than “RL teaches reasoning from zero.” Base models, at least the ones tested here, already possess meaningful math-solving ability, already generate self-reflective language in some fraction of their outputs, and (for Qwen2.5 specifically) already behave like a question-answering system without any template at all. What RL demonstrably does, and this is the honest, defensible claim, is take a policy that can sometimes stumble onto a correct, well-reasoned trajectory and make that trajectory systematically more probable — reinforcing existing good behavior far more than it is inventing new capability out of nothing.

That reframing matters directly for the next two chapters, because it sharpens exactly what question to ask about response length. If length is growing during training, the honest question isn't “is the model learning to think more, a good thing” by default — it's “is this length growth tracking real accuracy gains, or is something in the optimization itself pushing length up independent of whether the extra tokens are doing any useful work?” Chapter 4 shows that GRPO's own loss function has exactly this kind of independent, non-accuracy-driven pressure baked into it.

Chapter 4: The Bias Hiding in GRPO

Chapter 3 ended on a precise question: is response length growing during R1-Zero-style training because the model is genuinely learning to reason longer, or is something in GRPO's own loss function pushing length up for reasons that have nothing to do with accuracy? This chapter derives the answer directly from the formula Chapter 2 wrote down, term by term.

Bias one: the response-level length bias

Go back to GRPO's double normalization from Chapter 2: (1/G) over samples, (1/|o_i|) over tokens within each sample. Isolate what that inner (1/|o_i|) actually does to any one token's influence on the total gradient. A single token t inside response o_i, with per-token gradient contribution g_t (its own ∇logπ_θ term) and the response's shared advantage Â_i, contributes to the total loss gradient with an effective weight of:

The (1/|o_i|) factor means: the exact same per-token gradient signal gets divided by a bigger number the longer the response is. Work a concrete illustration to feel the size of the effect. Two incorrect responses in a group, both with the same negative advantage Â=−1, but different lengths — response C is 50 tokens, response D is 500. Suppose each contains exactly one clearly-identifiable “bad reasoning step” token with gradient magnitude g=1, with the rest of the tokens contributing roughly zero (filler, connective text):

The identical mistake, penalized identically in principle, receives ten times less corrective pressure simply because it happened to occur inside a longer response. The paper states this precisely: for negative advantages (incorrect responses), longer responses are penalized less per token, due to their larger |o_i|, “causing the policy to prefer lengthier responses among incorrect ones.” And by the same (1/|o_i|) logic run the other direction, for positive advantages (correct responses), shorter responses get a proportionally bigger per-token update — “leading the policy to favor brevity in correct answers.” Put together: the optimization itself, independent of any actual reasoning quality, is structurally biased toward getting longer when wrong and shorter when right.

python
def effective_weight(group_size, response_length):
    # the two GRPO normalization factors this chapter is dissecting, isolated
    return (1 / group_size) * (1 / response_length)

# response C: 50 tokens. response D: 500 tokens. same group size G=4.
w_C = effective_weight(4, 50)    # -> 0.005
w_D = effective_weight(4, 500)   # -> 0.0005, exactly 10x smaller
assert round(w_C / w_D) == 10   # the tenfold dilution, made checkable in one line

Writing it as a function makes the bias inspectable rather than just described: effective_weight takes only group_size and response_length as inputs — nothing about correctness, reasoning quality, or how good any individual token's contribution actually was. The bias is entirely a property of two bookkeeping numbers, which is exactly why Chapter 5's fix can remove it without touching anything about how rewards or correctness are computed.

Bias two: the question-level difficulty bias

The second bias lives in the other normalization term: dividing by std({R₁,…,R_G}). Questions where the group's rewards are close to uniform — nearly all correct, or nearly all incorrect — have a small standard deviation, and dividing by a small number amplifies whatever advantage comes out. Chapter 2's hand-worked example already hinted at this (the 3-correct-1-incorrect group's outlier got a bigger advantage than the balanced group's); now push it to a scale where the effect is unmistakable, using a group of 16 and the R1-Zero paper's own reward convention for this specific analysis, R ∈ {0, 1} rather than DAPO's ±1.

Case A — a balanced, “medium” question: 8 of 16 responses correct.

Case B — an “easy” question: 15 of 16 correct, only 1 incorrect.

Compare the two “incorrect” advantages directly: −0.968 in the balanced group versus −3.75 in the easy group — the lone failure on the easy question gets roughly 3.9× the gradient magnitude of an equally-wrong response in the balanced group, purely because it was rare inside a low-variance group. Meanwhile every correct response in the easy group gets shrunk to a fifth of its balanced-group counterpart (0.25 versus 0.968). Nothing about how correct or incorrect any of these individual responses actually is changed between the two cases — only how lopsided their group happened to be. A handful of easy or hard questions in a training batch can end up dominating the gradient, disproportionate to how informative they actually are about the policy's general reasoning ability.

The general formula: what happens as group size G grows

The two group sizes worked by hand above are specific cases of a formula worth deriving once, in general, because it reveals something the two examples alone don't: what happens to the lone-outlier advantage magnitude as G itself grows. Take the specific pattern this chapter keeps returning to — exactly one incorrect response in an otherwise all-correct group of size G, using R∈{0,1}. The mean is (G−1)/G, and working through the same variance calculation as before (with Bessel's correction) gives a standard deviation of exactly 1/√G. Plugging both into the advantage formula and simplifying:

Check it: at G=16, this gives −15÷4=−3.75, exactly matching Case B above. Now evaluate the same formula at group sizes DAPO's own real configuration (Chapter 7 covers this: G=16) sits alongside:

The magnitude doesn't level off — it keeps growing as G grows, roughly proportional to √G once G is reasonably large (since (G−1)/√G ≈ √G for large G). This is a genuinely counterintuitive implication: sampling more responses per question, which normally makes a statistical estimate more reliable, makes this specific bias worse, not better, for the rare-outlier case. DAPO's real G=16 configuration sits partway up this curve, not at some safely small end of it — one more reason Chapter 5's fix matters at production scale, not just in a toy G=4 illustration.

Contrast: the balanced case's advantage magnitude is bounded, not growing

It's worth contrasting the lone-outlier formula just derived against the balanced case from earlier in this chapter. For a group split exactly in half (G/2 correct, G/2 incorrect, R∈{0,1}), the same style of derivation gives mean=0.5 and, after Bessel's correction, std=√(G÷(4(G−1))), so the advantage magnitude simplifies to:

Unlike the lone-outlier formula, which grows without bound as G increases, this expression is capped: it starts at √(3/4)≈0.866 for G=4, reaches √(15/16)≈0.968 for G=16 (matching Case A exactly), and approaches, but never exceeds, exactly 1.0 as G grows arbitrarily large. A perfectly balanced group's advantage magnitude is bounded and converges; a lopsided group's rare-outlier advantage magnitude is unbounded and diverges. The difficulty bias is entirely a property of how lopsided a group is, not of group size on its own — group size only determines how severe the bias becomes once a group happens to be lopsided.

When both biases pull the same way at once

The two biases derived above don't always fight each other — sometimes they compound. Difficulty bias under-penalizes an incorrect response specifically when incorrectness is common in its group (a “hard” question most attempts fail); length bias under-penalizes an incorrect response specifically when it's long. A response that is both typical-wrong in a hard group and long gets hit by both under-corrections at once. Take the mirror image of Case B: a 16-sample “hard” group with only 1 correct and 15 incorrect — by the same arithmetic as before, with the roles of correct and incorrect reversed:

Already, a typical wrong response in this hard group gets an advantage of only −0.25 in magnitude — smaller than the balanced group's −0.968, let alone the easy group's −3.75. Now let that same typical, already-under-weighted-by-difficulty response also be long, 800 tokens, with the same single identifiable bad-reasoning-step token of gradient magnitude g=1:

Set this next to this chapter's very first worked number: response D, 500 tokens, in a perfectly balanced group, contributed −0.002. The doubly-biased response here contributes roughly six times less correction still, despite containing the exact same single bad-reasoning-step token. Both biases push in the same direction — toward silently forgiving long, typical mistakes on hard questions — and they don't just add together, they multiply.

Two biases, one root cause

Both biases share the same shape: a normalization term that was defensible in isolation (average per response so long responses don't dominate; normalize by spread so noisy-reward questions don't dominate) turns into an unintended source of systematic distortion once you notice it interacts with something correlated with the very thing you're trying to measure — response length in one case, question difficulty in the other. Chapter 5 shows the fix is almost insultingly simple once you see it this way: stop normalizing by anything that isn't the same constant for every sample.

One last generalization worth naming before moving on: nothing about either bias derived in this chapter is specific to math problems, DeepSeek, or Qwen. Any RL setup using GRPO's exact double-normalization structure — group sampling, outcome rewards, per-sample and per-group division — inherits both biases automatically, regardless of the domain. Code generation scored by pass/fail unit tests, or any other verifiable-reward task sampled in groups: the same arithmetic applies unchanged. The fix built in Chapter 5 is exactly as general as the bias it fixes.

Chapter 5: Dr. GRPO: Doing It Right

Chapter 4 traced both of GRPO's biases to the same kind of move: normalizing by a quantity that varies from sample to sample or question to question, when a variance-reduction baseline should not depend on the thing you're measuring. The fix — Dr. GRPO, short for “GRPO Done Right” — is almost anticlimactic in how small it is on the page, and that's exactly the point.

The fix, in one sentence

Remove the 1/|o_i| term and the std({R_i}) term. That's the entire modification. In place of 1/|o_i| — which varied per sample, tying the per-token weight to that specific response's length — substitute a single constant, the same for every sample in every batch: a fixed generation budget, e.g. MAX_TOKENS, chosen once ahead of time and never adjusted per response. And simply drop the std(·) division from the advantage formula, leaving the centered reward on its own:

The paper's own account of why this recovers something principled, not just a heuristic patch: with these two terms removed, the objective is mathematically equivalent to the original, textbook PPO objective from Chapter 1, with the advantage estimated as a Monte Carlo return using an unbiased baseline — exactly the kind of advantage estimator standard RL theory already endorses, just without a critic network computing it. Dr. GRPO isn't a new trick bolted onto GRPO; it's what falls out when you stop introducing normalization that PPO's own formulation never asked for.

What actually changes in the implementation

In code, this fix is a one-line diff, not a rewrite. Sample-level loss is typically implemented with a masked_mean helper that divides by each sample's own token count:

python
# before (GRPO): divide by this sample's own token count
def masked_mean(values, mask, axis=None):
    return (values * mask).sum(axis=axis) / mask.sum(axis=axis)  # mask.sum() = |o_i|, varies per sample

# after (Dr. GRPO): divide by a fixed constant, the same for every sample
def masked_mean(values, mask, axis=None, constant=MAX_TOKENS):
    return (values * mask).sum(axis=axis) / constant          # MAX_TOKENS, never varies

The entire fix is replacing the sample's own token count in the denominator with a fixed generation-budget constant that never changes from sample to sample or batch to batch. Every other line of a GRPO implementation — the clip, the ratio, the group-relative advantage computation itself — is untouched. This is worth sitting with as a general lesson about optimization bugs: the fix for a bias that took an entire paper to diagnose correctly can be, once diagnosed, a single denominator swapped out.

Re-running the worked example, fixed

Return to Chapter 4's two incorrect responses — response C at 50 tokens, response D at 500 tokens — with the same single “bad token” of gradient magnitude g=1 in each, same advantage Â=−1. Under Dr. GRPO, the per-token weight is a fixed constant regardless of response length. Take MAX_TOKENS=1000 as an illustrative fixed budget:

Identical. The same mistake now gets penalized identically no matter how long the response containing it turns out to be. The tenfold dilution from Chapter 4 is gone, not by cleverly re-weighting anything, but by refusing to let the weight depend on length in the first place.

What actually happens to training when you fix this

The paper's controlled comparison runs both algorithms from the same Qwen2.5-1.5B base model, same R1 template, same reward rule (1 if the response contains the correct final answer, 0 otherwise), same MATH training questions — the only difference is GRPO versus Dr. GRPO. Both show the DeepSeek-R1-Zero-style pattern of response length increasing alongside training reward early on. But GRPO keeps generating progressively longer responses even after reward improvement has largely flattened out — length still climbing with nothing left to show for it. Dr. GRPO's length curve, under the identical setup, stops growing once reward plateaus. And on evaluation benchmarks specifically, the length of incorrect responses is substantially shorter under Dr. GRPO than under vanilla GRPO — direct evidence against the “prefer lengthier responses among incorrect ones” bias derived in Chapter 4, measured after the fact, not just predicted from the formula.

This reframes what “emergent long-CoT reasoning” means for a reader of any R1-Zero-style paper: some portion of any reported length increase, on some non-trivial fraction of training runs, could be nothing more than this specific accounting artifact rather than genuinely more effective reasoning. It doesn't mean length growth is never real reasoning improvement — both papers show real accuracy gains too — it means length alone is not, on its own, trustworthy evidence of it.

Does the recipe survive different templates and different question sets?

Before trusting a single "minimalist recipe" number, it's worth stress-testing it along two axes that could plausibly matter just as much as the algorithm itself: which template wraps the question, and how much of the problem space the training question set actually covers. Starting from Qwen2.5-Math-1.5B (recall from Chapter 3: exactly the model family that performs best with no template at all), the analysis runs Dr. GRPO under all three templates from Chapter 3 — R1, Qwen-Math, and none — each paired with four training question sets of very different size and difficulty:

Two results stand out. First: whichever template a run starts from, RL training can push nearly every combination up to a comparable final accuracy, around 40%, provided the question set is a reasonable match — templates mostly set where a policy starts, not where it ends up, once RL has enough steps and a decent question set to work with. Second, and more surprising: when using the Qwen-Math template, the best final performance actually comes from training on GSM, the simplest, smallest, most out-of-distribution question set of the four — nearly doubling test accuracy on the harder held-out benchmarks, despite never training on anything close to that difficulty. The reading: when a base model and template are already a good match (as Qwen2.5-Math and its own template are), RL's job is mostly to reinforce reasoning behaviors the model already has, rather than infuse genuinely new knowledge — and reinforcing existing behavior can be done even with a small, easy, mismatched question set, because the policy already knows how to solve harder problems, it just needs its correct trajectories made more probable.

Does math pretraining set a ceiling on what RL can reach?

Everything so far in this chapter uses Qwen2.5 models, which are already strong math solvers before RL ever starts. A fair objection: maybe Dr. GRPO's clean results are specific to base models that already know most of what they need to know. The analysis tests the opposite case directly — starting from Llama-3.2-3B, a base model with comparatively weak native math ability, using Dr. GRPO with the R1 template throughout.

Plain Llama-3.2-3B does improve under RL, but only modestly. To test whether pretraining domain sets a ceiling on how far RL can push a weak base model, the comparison continues pretraining Llama-3.2-3B on FineMath (a math-focused pretraining dataset) to get Llama-3.2-3B-FineMath, and separately continues pretraining that checkpoint for two more epochs (learning rate 1e-5) on a concatenated version of NuminaMath-1.5 — question and response text joined together, mirroring the pretraining pattern Chapter 3 hypothesized for Qwen2.5-Math — to produce Llama-3.2-3B-NuminaQA. RL-tuning each of these three checkpoints with an identical recipe produces a clean ordering: plain Llama shows the smallest gain, FineMath continual pretraining meaningfully raises the ceiling RL can reach, and the concatenated NuminaQA pretraining raises it further still.

The minimalist recipe this analysis produces

Putting Chapters 3 through 5's findings together — pick a base model whose pretraining already lines up well with the target domain, choose a template that doesn't fight the base model's native behavior, and fix the optimization bias — the paper assembles what it calls a minimalist R1-Zero recipe: RL-tune Qwen2.5-Math-7B with Dr. GRPO on MATH level 3–5 questions, using the Qwen-Math template. The result: 43.3% accuracy on AIME 2024, a new state-of-the-art result for a 7B-scale model at time of publication, reached with only 27 hours of compute on 8×A100 GPUs — a genuinely modest compute budget compared to the large-scale systems built around it. Chapter 9 returns to this number directly, next to DAPO's own headline result, to work out what can and cannot be honestly compared between the two.

Putting "27 hours on 8×A100" in perspective

216 total GPU-hours (8 GPUs × 27 hours) is a genuinely small number by large-model RL standards — well within reach of a single well-equipped academic lab, not a frontier-lab-scale run spanning thousands of GPUs for weeks. That's the actual point of calling this recipe "minimalist": the result isn't state-of-the-art because of raw compute thrown at the problem, it's state-of-the-art because Chapters 3 through 5's diagnosis — the right base model, the right template, the unbiased optimizer — lets a comparatively small amount of compute reach a ceiling that a naively-configured run, even with far more compute behind it, might never reach at all. Compute and correct configuration aren't substitutes for each other in this story: DAPO's own 30-point naive-GRPO baseline from Chapter 0 was run on a full-scale system too, and configuration, not raw scale, was what needed fixing first.

What MAX_TOKENS actually needs to be, as Dr. GRPO's fixed constant

Dr. GRPO's fix replaces a sample-dependent denominator with "a fixed generation budget, e.g. MAX_TOKENS" — worth being precise about what that constant needs to satisfy. It doesn't need to equal any specific response's actual length; it only needs to be the same number for every sample in every batch, which is the entire property the earlier derivation relies on. In practice this is naturally the same maximum-generation-length setting used elsewhere in the training pipeline (Chapter 8 covers DAPO's own choice of this number directly, in the context of truncation). Pick it too small relative to typical response lengths, and the per-token weight 1/MAX_TOKENS becomes uncomfortably large, amplifying every gradient signal; pick it far larger than any response actually needs, and the per-token weight becomes uncomfortably small, diluting every gradient signal uniformly. Either extreme is a global scaling issue, correctable by adjusting the learning rate — unlike GRPO's original 1/|o_i|, which varied per sample and could not be fixed by any single global learning-rate adjustment at all.

The paper's own takeaways, mapped to this session's chapters

Understanding R1-Zero-Like Training closes its introduction with a compact list of its own headline findings. Now that Chapters 3 through 5 have derived each one from scratch, it's worth seeing them laid out together, next to exactly where each was built in this session:

Six findings, and not one of them required taking the paper's word for it — every row above was either derived symbolically (the bias itself, worked by hand in Chapter 4) or reconstructed from a specific, named experiment (the templates, the Llama pretraining comparison) earlier in this session.

Chapter 6: Clip-Higher

Switch tracks back to DAPO's own four techniques, and the 30-point naive GRPO baseline from Chapter 0. The first failure the DAPO team diagnosed, watching that baseline train, was entropy collapse: the policy's output distribution getting sharper and sharper, sampled responses within a group becoming nearly identical to one another, exploration shutting down early. A policy that has stopped exploring cannot discover new, better reasoning strategies — it can only exploit whatever it already knows, which is precisely the opposite of what long, effective chain-of-thought reasoning training needs.

Where the collapse comes from: the clip range is not symmetric in its effect

Go back to PPO's clip range from Chapter 1: [1−ε, 1+ε], with the default ε=0.2 used by most algorithms including naive GRPO. This bounds the importance ratio symmetrically — but a symmetric bound on a ratio is not a symmetric bound on the underlying probability, and that asymmetry is the whole story.

Take the paper's own worked comparison: two tokens, one the policy currently assigns probability π_θold=0.01 (a rare, “exploration” token), the other probability π_θold=0.9 (an already-likely, “exploitation” token). With ε=0.2, the maximum the clip allows either probability to grow to in a single update is:

Read the gap between those two headroom numbers. The already-likely token's clip ceiling (1.08) sits entirely past the maximum possible probability (1.0), meaning the clip essentially never binds for it — it's free to keep growing toward certainty. The rare token's ceiling allows only a tiny absolute increase, 0.002, no matter how much the policy has learned that this token deserves to be much more likely. The DAPO team observed this pattern directly in their own training runs, tracked across every token whose ratio actually hit the upper clip bound: the mean probability of these up-clipped tokens stayed low, under 0.2, across training — exactly the signature of the clip actively suppressing the growth of low-probability tokens on average, confirming the arithmetic above isn't just a hypothetical edge case.

The same asymmetry, across the full probability range

Two tokens make the point, but it's worth seeing the pattern hold everywhere, not just at the two extremes just used for illustration. Absolute headroom under a symmetric ε=0.2 clip, for a spread of starting probabilities:

The absolute headroom scales with the starting probability itself — a direct consequence of clipping a ratio rather than a probability. Every row's headroom is exactly 0.2×p, so the tokens that most need room to grow (the rare, low-probability exploration candidates near the top of the table) get the least of it in absolute terms, and the tokens that already dominate the distribution (the bottom of the table) hit the 1.0 probability ceiling and stop being meaningfully constrained by the clip at all.

The general formula behind that table

The table above isn't five isolated calculations — every row is the same formula, headroom(p) = p·ε, evaluated at a different starting probability. Derive it once: the clip ceiling is p×(1+ε), so the absolute headroom is p×(1+ε)−p = p×ε. Headroom is linear in the starting probability p, which is exactly why rare tokens near p=0 get vanishingly small absolute headroom while common tokens near p=1 get comparatively large headroom, using the identical multiplier ε in both cases. This is the algebraic root of everything else in this chapter: entropy collapse is what happens when a linear-in-p headroom function structurally favors already-likely tokens growing over unlikely ones, compounded across thousands of update steps.

Why removing the KL penalty (Chapter 2) makes this problem more urgent, not less

It's worth connecting this chapter back to a decision made four chapters ago. Chapter 2 removed the βD_KL term entirely, on the grounds that a rule-based reward doesn't need a leash keeping the policy close to a reference model. That reasoning was sound on its own terms — but the KL term, as a side effect in classic RLHF, also happened to keep the policy's distribution from drifting too far from a reasonably-entropic starting point. With no KL term pulling the policy back toward its reference distribution, and a naive symmetric clip actively suppressing the growth of exactly the low-probability tokens that carry exploration, there is nothing left in naive GRPO's objective counteracting entropy collapse at all. Clip-Higher isn't an unrelated bug fix bolted onto a separate problem; it's effectively the replacement for a stabilizing force Chapter 2 deliberately removed, once you see exactly why that removal was safe in the reward-hacking sense but not free in the exploration sense.

Three qualitative shapes for the same monitored entropy curve make the distinction concrete — illustrative numbers, not DAPO's own logged values, but useful for recognizing each pattern on a real training dashboard:

Healthy training nudges entropy up slowly and keeps climbing gently. Collapsed training slides toward the narrow-distribution regime this chapter derives from the arithmetic above. Over-exploring training's entropy runs away in the other direction entirely, correlating with the gibberish-and-repetition failure mode the earlier callout in this chapter already named. Clip-Higher is tuned to sit inside the first column, not to chase the third one by raising ε_high without limit.

The fix: decouple the two sides of the clip

DAPO's fix, already previewed in Chapter 2's objective, is to stop sharing one ε between both directions of the clip and use two independent values, ε_low and ε_high:

DAPO's chosen values: ε_low=0.2 (left unchanged from the standard default), and ε_high=0.28 (raised, giving low-probability tokens more room to grow). Recompute the low-probability token's headroom under the new, raised ceiling:

python
def decoupled_clip(ratio, eps_low=0.2, eps_high=0.28):
    return ratio.clamp(1 - eps_low, 1 + eps_high)   # asymmetric window, not [1-eps, 1+eps]

One clamp call, two independent bounds instead of one shared ε — the entire Clip-Higher implementation is this small. Everything else in PPO's clipped objective from Chapter 1 (the min of clipped and unclipped, the multiplication by the advantage) stays exactly as written; only the two numbers passed into clamp change.

On this single token, the visible change looks small — but this one update compounds across every low-probability token, every step, over the entire training run, and the aggregate effect is what the DAPO team actually measured: applying Clip-Higher visibly raises the policy's entropy compared to the unmodified baseline and produces more diverse sampled responses within a group, exactly the property entropy collapse was destroying.

Why ε_low is left alone, not raised too

It would be tempting to raise both sides symmetrically. DAPO specifically keeps ε_low small, and the reasoning matters: increasing ε_low would let the clip range extend lower, allowing already-unlikely tokens' probabilities to be pushed down further and faster in a single step. Push a token's probability down aggressively enough, and it can collapse toward zero — at which point that token is functionally removed from the model's sampling space entirely, which is the opposite failure mode from entropy collapse but just as damaging to exploration: a sampling space that has permanently lost options is not meaningfully different from one that's collapsed onto a narrow peak. Raising only the ceiling, leaving the floor where it was, targets the specific asymmetry Chapter 6 opened with without introducing a new one.

What "entropy" concretely measures: a three-token toy example

Before talking about entropy rising or falling, it's worth computing one by hand, since "the policy's entropy" can otherwise stay an abstract phrase. For a token position with three possible continuations, entropy is H = −∑p·log₂p, summed over the distribution's probabilities. Take a policy early in training, still exploring, that assigns roughly even probability to all three options:

Now take the same three-way choice after entropy collapse has set in, the policy having narrowed sharply onto one option:

Entropy dropped by roughly 86% between the two distributions, at just one token position — and this is the quantity DAPO tracks, averaged across every token position in every sampled response, as training progresses. “Entropy collapse” means this number sliding toward the second regime, at scale, across the whole policy, not a metaphor: it is literally this same −∑p·log₂p computation converging toward zero as the distribution sharpens onto fewer and fewer live options.

More entropy is not automatically better

It would be easy to leave this chapter thinking "raise ε_high as much as possible, get as much entropy as possible." DAPO's own monitoring rules that out directly: entropy needs to sit inside an appropriate range, not simply be maximized. Too low, and the distribution is overly sharp — the entropy collapse this chapter derives. But too high is its own distinct failure, associated with over-exploration: gibberish and repetitive generation, a policy so undecided about what to say next that its outputs stop being coherent reasoning at all. In practice, a slow, gradual upward trend in entropy over the course of training — not a spike, not a flat line near zero — is what actually correlates with improving performance. This is why ε_high=0.28 is a specific, measured choice rather than "as large as possible": it's tuned to sit inside the healthy range, nudging entropy upward gently rather than blowing the distribution open.

What would happen at ε_high=1.0, hypothetically

Push the thought experiment to a concrete number. At ε_high=1.0, the low-probability token's headroom becomes 0.01×(1+1.0)=0.02, ten times the headroom DAPO's actual ε_high=0.28 allows. That sounds like it would help exploration even more — and locally, on this one token, it would. But recall the earlier finding in this chapter: high-probability tokens already hit the 1.0 ceiling and become unconstrained well before ε=0.28 is even reached. Pushing ε_high further doesn't meaningfully change what happens to those tokens; it only discards more of the trust-region protection Chapter 1 built this entire clipping mechanism to provide, specifically for large positive-advantage updates (the clip's other job, protecting negative-advantage updates via ε_low, is untouched by raising ε_high at all). DAPO's 0.28 is a compromise point: enough room to meaningfully help the low-probability tail, without discarding so much of the trust region that a single large update becomes as risky as the unclipped objective this whole session started from.

Measured effect on the actual metric that matters

Applying just this one technique on top of naive GRPO's 30-point baseline — recorded in DAPO's own progressive ablation, which Chapter 9 walks through in full — moves the score to 38, a real, measured gain, though notably smaller than two of the other three techniques still to come. Clip-Higher fixes a specific, diagnosable failure (entropy collapse); it is not, on its own, the single largest contributor to DAPO's final 50-point result. Keep that in mind as a preview — Chapter 9's honest reading of the full table is exactly about which techniques earned which points, not assuming they contributed equally just because they're all described as important.

It's worth being clear about what this measured gain does and doesn't tell you on its own. Reaching 38 from naive GRPO's 30 confirms Clip-Higher helps; it doesn't, by itself, prove entropy collapse was the training run's single biggest obstacle, since Overlong Filtering is already stacked on top of naive GRPO by the time this row of the table is reached in Chapter 9's full account — the +8 from 30 to 38 is Overlong Filtering's own +6 plus Clip-Higher's +2, not Clip-Higher's contribution alone. Reading one row of a progressive ablation table in isolation, without the rows around it, is exactly the kind of shortcut this session keeps warning against.

Chapter 7: Dynamic Sampling & Token-Level Loss

Two more of DAPO's four techniques, and both trace back to observations already made earlier in this session — one to Chapter 2's degenerate all-correct group, one to Chapter 4's length bias, but arrived at independently by the DAPO team, watching their own training runs rather than deriving it from Dr. GRPO's paper.

Dynamic Sampling: don't waste compute on zero-gradient groups

Chapter 2 already showed the mechanism: when every response in a group receives the identical reward — all correct, or all incorrect — the group's standard deviation is zero, the advantage collapses to nothing, and that batch contributes no gradient at all, despite costing exactly as much compute to sample and score as any other batch. DAPO's own training observed this happening more and more as training progressed: the fraction of questions where every sampled response in the group was already correct kept climbing, which means the effective number of prompts actually contributing useful gradient in each batch kept shrinking, even though the nominal batch size stayed fixed. Fewer effective prompts per batch means noisier, higher-variance gradients and weaker training signal, exactly when you'd want it to be getting cleaner as the model improves.

Put a hypothetical number on "effective prompts shrinking," just to feel the size of the problem: suppose a rollout batch samples groups for 512 prompts, and by some point in training 30% of those groups have already collapsed to all-correct — a plausible, if illustrative, scenario for a model that has gotten reliably good at the easier end of its training distribution. Even though the batch nominally still contains 512 prompts, only about 358 of them are actually contributing nonzero gradient: the training step's effective sample size has silently shrunk by nearly a third, with no change to the code, the batch-size setting, or the compute budget spent sampling it.

The fix, called Dynamic Sampling, is to over-sample and filter: before finalizing a training batch, keep sampling additional groups and discard any group whose accuracy came out exactly 0 or exactly 1, continuing until the batch is entirely filled with groups that have a non-degenerate mix of correct and incorrect responses. Written as an explicit constraint on which groups are admissible:

In words: the count of correct responses in the group must be strictly between 0 and G — not all wrong, not all right. Every group that makes it into the batch is now guaranteed to produce a nonzero gradient.

Why filtering happens before the advantage is computed, not after

This ordering isn't arbitrary. Computing a group-relative advantage for a degenerate group and only discarding it afterward would still cost the same forward pass, and would still hit the same division-by-near-zero numerical awkwardness Chapter 2 already flagged. Filtering at the point where a group's correct-count is already known — immediately after scoring, before the advantage formula ever runs — means degenerate groups never touch the more expensive parts of the pipeline at all, and the advantage computation downstream can simply assume every group it ever sees is already guaranteed non-degenerate, simplifying that code path too.

Why Dynamic Sampling is affordable specifically because the reward is a rule

Oversampling groups until enough are non-degenerate only stays cheap because checking each response's reward is nearly free — is_equivalent(ŷ, y) from Chapter 0 is a fixed comparison, not a forward pass through a second neural network. If this session's reward were instead a learned reward model, every extra sampled group needed to satisfy Dynamic Sampling's filter would also mean an extra forward pass through that reward model, adding real GPU cost on top of the extra generation cost. Dynamic Sampling's near-free oversampling and Chapter 0's rule-based reward aren't two separate design choices that happen to coexist inside DAPO — the second one is part of what makes the first one this cheap to run at all.

How much oversampling, as a function of how often groups degenerate

The "doesn't oversampling cost more compute" question above has a clean quantitative answer. If p is the probability that a freshly-sampled group turns out degenerate (all-correct or all-incorrect), then — treating each group's outcome as an independent coin flip — the expected number of groups you need to sample before getting one non-degenerate group is 1/(1−p), a standard property of a geometric distribution (the same logic behind "how many times do I expect to flip a coin before it lands heads?").

This is exactly why the observation that degenerate rates climb over training matters: as the model gets reliably good at easier questions, p climbs, and the oversampling factor needed to fill a clean batch grows with it — not linearly, but in a way that accelerates as p approaches 1. A training run that starts out needing barely any extra sampling (p near 0) can, by late training, need several times its nominal batch size in raw samples just to fill one clean batch. The wall-clock argument from the callout above — generation time dominated by the slowest response regardless — is precisely what keeps this growing sampling cost from translating into a proportional growth in training wall-clock time.

What the buffer actually contains, once assembled

It's worth being precise about what ends up in the training buffer once Dynamic Sampling has finished filtering. Every group inside it has a correct-count strictly between 0 and G, but that says nothing about how lopsided each surviving group is — a group with 1 correct out of 16 and a group with 8 correct out of 16 both pass the filter equally, even though Chapter 4's difficulty bias treats them very differently once the group-relative advantage is computed. Dynamic Sampling solves a narrower problem than Chapter 5's difficulty-bias fix: it guarantees every surviving group contributes some nonzero gradient; it says nothing about whether that gradient ends up disproportionately weighted by how lopsided the surviving group happens to be. DAPO's own advantage formula, notably, keeps the std(·) division from GRPO's original Chapter 2 formula rather than adopting Dr. GRPO's fix for it — Dynamic Sampling addresses the zero-gradient failure mode directly, while leaving the difficulty-bias question to whichever advantage formula a given implementation chooses to pair it with.

Token-Level Loss: rebalancing what Dr. GRPO already flagged

The second fix in this chapter targets the exact response-level length bias Chapter 4 derived by hand — reached independently by the DAPO team, watching a different symptom: they observed that excessively long responses often contained low-quality patterns, gibberish and repetitive phrases, and that sample-level loss's inability to penalize those patterns effectively led to an unhealthy, ongoing increase in both entropy and response length during training — the same growth pattern Dr. GRPO's paper diagnosed as a bias, described here as an observed training pathology.

DAPO's fix has the identical mathematical shape as Dr. GRPO's: replace the double sum-then-average (1/G)∑_i(1/|o_i|)∑_t with a single flat sum normalized by the total token count across the whole batch, ∑_i|o_i|:

This is called token-level loss, because every individual token across the entire batch now gets an equal share of the total gradient weight, rather than every sample getting an equal share regardless of how many tokens it contains. Under this scheme, a longer sequence naturally has more total influence on the overall gradient than a shorter one — proportional to how many tokens it actually contains, not artificially deflated. And from the perspective of any single token: if a particular pattern tends to raise or lower reward, it gets equally reinforced or suppressed no matter which response, long or short, it happens to appear in — precisely undoing the length-dependent dilution Chapter 4 derived.

A worked comparison: 2 samples, very different lengths

Make the difference between sample-level and token-level loss concrete with a small batch: two responses in a group, response E at 100 tokens and response F at 900 tokens, both with advantage magnitude 1 for simplicity. Under sample-level loss, each response gets an equal 1/G=1/2 share of the batch regardless of length (1÷2=0.5 each), then that 0.5 is divided by its own token count: E's per-token weight is 0.5÷100=0.005, F's is 0.5÷900≈0.00056. Under token-level loss, the shared denominator is the batch's total token count, 100+900=1,000: E's total weight is 100÷1,000=0.1, F's is 900÷1,000=0.9, and every individual token in either response carries the identical 1÷1,000=0.001.

Under sample-level loss, the two responses split the total gradient budget evenly regardless of length — which sounds fair at the sample level, but per token it means E's tokens each carry roughly nine times the weight of F's. Under token-level loss, the total weight each response receives is proportional to its length (F, being nine times longer, receives nine times E's total influence) — but the crucial property is in the last column: every individual token, regardless of which response it's in, now carries the identical per-token weight. That's what “equally reinforced or suppressed, regardless of length” means, made numeric.

Measured effect

In DAPO's own progressive ablation table, Token-Level Loss on its own contributes a comparatively modest gain — from 41 to 42 points, one point. But the paper is explicit that raw accuracy points understate its value: this technique's real contribution is training stability and healthier length dynamics, not squeezing out the largest possible score bump on its own. Chapter 9 puts this number in context next to Dynamic Sampling's own contribution, which turns out to be dramatically larger — and asks what that gap is actually telling you about where DAPO's real gains came from.

The full training loop, assembled

Every piece built across Chapters 2, 6, and 7 fits into one training loop, and seeing it as a whole loop — rather than four separate equations — is worth doing once before Chapter 8 adds the final piece. DAPO's own pseudocode, condensed:

DAPO training loop
for step in range(M):
    D_b = sample_batch(D)                       # a batch of questions
    pi_old = pi_theta.copy()                     # freeze the old policy
    buffer = []
    while len(buffer) < N:                    # Dynamic Sampling (Ch 7)
        for q in D_b:
            group = sample_G_outputs(pi_old, q)   # G=16 responses to the same q
            rewards = [reward_fn(o) for o in group]
            n_correct = sum(r == 1 for r in rewards)
            if 0 < n_correct < G:
                buffer.append((q, group, rewards)) # keep only non-degenerate groups
    advantages = compute_grpo_advantage(buffer)   # mean/std per group (Ch 2)
    for _ in range(mu):                         # mu gradient updates per rollout step
        loss = dapo_objective(pi_theta, pi_old, advantages,
                               eps_low=0.2, eps_high=0.28,  # Clip-Higher (Ch 6)
                               normalize='token_level')       # Token-Level Loss (Ch 7)
        pi_theta.step(loss)

Nothing here is new physics — it's a checklist of everything already derived, wired into the order it actually executes: sample a group per question, throw away degenerate groups before they ever reach the loss, compute the group-relative advantage on what's left, and take the gradient step with the decoupled clip and token-level normalization instead of GRPO's originals. DAPO's real configuration fills in the constants: a rollout batch of 512 prompts, G=16 sampled responses per prompt, and a training mini-batch of 512 — meaning 16 separate gradient updates (μ=16 in the loop above) happen per rollout step before the policy generates fresh data again. Chapter 9 uses these same numbers when it reconstructs the full picture of what DAPO actually trained.

Choosing the buffer target N

The assembled loop's while len(buffer) < N condition has one more parameter worth naming: N, the target number of non-degenerate groups needed to fill a training batch. Set N too small, and the mini-batch used for each gradient update is noisier than intended — fewer independent groups means a higher-variance estimate of the true gradient, exactly the problem Dynamic Sampling exists to reduce in the first place. Set N unnecessarily large, and every rollout step pays for more oversampling than the degenerate-rate table above actually requires, without a proportional benefit to gradient quality. In practice, N is chosen to match the intended training batch size — DAPO's own mini-batch of 512 — so that Dynamic Sampling's filtering changes which groups make it into the batch, not how large the batch is once it's assembled.

Why 16 gradient updates on the same rollout data still needs the clip

One detail in the assembled loop is worth flagging explicitly: μ=16 separate gradient updates all reuse the exact same batch of sampled responses, generated once by π_θold at the start of the rollout step. By the second, third, and especially the sixteenth of those updates, π_θ has already moved away from π_θold — the very policy that generated the data being trained on. This is exactly the staleness problem Chapter 1's trust region was built to guard against, and it's why the importance ratio r_i,t(θ) and the clip are doing real work on every single one of those 16 updates, not just the first: without them, later updates inside the same rollout step would be applying gradient steps computed as if the data were still on-policy, when by that point it increasingly isn't.

Chapter 8: Overlong Reward Shaping

The last of DAPO's four techniques starts from an operational fact every RL-for-LLM system has to confront: generation cannot run forever. A maximum length has to be set, and any response that hits that ceiling gets truncated — cut off mid-generation, whatever it was in the middle of saying.

Detecting truncation is mechanical, not a judgment call

Before assigning any reward at all, the training loop needs to know whether a given response actually finished on its own or got cut off. This is a simple, purely mechanical check, not a judgment call: every response either ends because the model generated a designated end-of-sequence token on its own, or because the generation loop hit its hard token-count ceiling first and stopped it forcibly. A response in the first category made its own decision about when it was done; a response in the second category was interrupted mid-thought, with no say in the matter. This distinction is exactly what the naive default gets wrong — it treats both categories of “no verifiable correct answer” identically, when only one of them actually reflects something the model chose to do.

One edge case worth naming explicitly: a response that finishes naturally (generates its own end-of-sequence token) at, say, 17,000 tokens — comfortably inside where Soft Overlong Punishment's soft-cache zone will later sit, but past whatever penalty-free threshold gets set. This response was never truncated at all; it simply chose to be long. Soft Overlong Punishment, built later in this chapter, doesn't distinguish "chose to be long and finished on its own" from "was truncated partway through the soft-cache zone" — length alone determines the length-based penalty, regardless of which category the response falls into. That's a deliberate simplification, not an oversight: separating "long but complete" from "long and truncated" would require the reward function to reason about the response's internal structure beyond its raw token count, reintroducing exactly the kind of judgment-call complexity Chapter 0's rule-based reward was designed to avoid in the first place.

The naive default is quietly wrong

The obvious thing to do with a truncated response is assign it a punitive reward, the same as any other wrong answer — after all, it never produced a verifiable final answer at all. DAPO's diagnosis: this default introduces real reward noise, because truncation is not evidence that the reasoning inside the response was bad. A response can be pursuing a genuinely sound, on-track chain of reasoning and simply run out of token budget before reaching the final boxed answer — punishing that response the same as a response that reasoned itself into total nonsense sends a confusing, inconsistent signal about what the model's reasoning process actually did wrong.

The first thing DAPO tries, as a diagnostic more than a final answer, is Overlong Filtering: simply mask the loss for every truncated sample entirely, contributing neither reward nor gradient from it. This alone, the paper reports, significantly stabilizes training and improves performance — strong evidence that the noise really was coming from truncation-as-punishment, not from some other source.

The stabilization shows up on two separate metrics, not just accuracy: the paper's own before-and-after comparison tracks both AIME accuracy and the entropy of the policy's generation probabilities, and both improve once truncated samples stop injecting punitive, uncorrelated-with-quality reward into training. That two-metric agreement is a useful sanity check in general when diagnosing a training pathology: if masking one specific, well-understood source of noise (here, truncation) improves multiple independent metrics at once, that's much stronger evidence the diagnosis was correct than either metric improving alone would have been.

Soft Overlong Punishment: a length-aware ramp instead of a cliff

Filtering entirely discards a signal (whether a response even respects the length budget is itself useful information, in principle). DAPO's actual solution is a middle ground, Soft Overlong Punishment — a length-aware penalty that ramps up gradually as a response gets close to and then exceeds the length budget, rather than switching abruptly from zero penalty to maximum penalty the instant the hard cutoff is crossed:

Read it as three zones. Below a threshold, no penalty at all — the response fit comfortably inside the budget. Above the hard maximum L_max, a full −1 penalty, same as the naive default. In between — a “soft cache” zone of width L_cache — the penalty ramps linearly from 0 down to −1 as the response creeps further into the buffer. This penalty gets added to the base correctness reward, not substituted for it, so a genuinely correct answer that runs a little long is still credited for being correct, just partially discounted for the length overrun.

Worked example, with DAPO's real hyperparameters

DAPO's training configuration sets the expected maximum generation length at 16,384 tokens, with an additional 4,096 tokens allocated as the soft-punishment cache — making the true hard ceiling for generation 20,480 tokens total:

Take a response that runs to exactly 18,432 tokens — 2,048 tokens into the 4,096-token soft-cache zone, precisely halfway through the buffer:

If this particular response's underlying reasoning was actually correct (base reward +1), the combined reward is 1 + (−0.5) = +0.5 — still net positive, still reinforced, just discounted for running unnecessarily long.

Push the same response 1,024 tokens further into the buffer, to 19,456 tokens (three-quarters of the way through the cache):

Combined with a correct base reward: 1 − 0.75 = +0.25, still positive, but the model is now feeling meaningfully more pressure to wrap up sooner. And at the hard boundary, 20,480 tokens exactly, the piecewise formula's soft zone ends and the flat −1 penalty applies in full — even a correct answer delivered at exactly the ceiling nets to 1 − 1 = 0, no reward at all for correctness that arrived too late to be worth crediting.

What happens when a truncated response was also going to be wrong anyway

One more case worth working through: a response that both gets truncated and, had it been allowed to finish, would likely have reasoned its way to an incorrect answer anyway. Soft Overlong Punishment doesn't need to know this in advance — it only ever sees length, never the correctness of an unfinished chain. A truncated response gets exactly the length-based penalty from the table above regardless of whether the reasoning inside it was sound, because there simply is no base correctness reward to add it to: Chapter 0's is_equivalent(ŷ, y) has nothing to compare against a response that never reached a final boxed answer at all. In practice this means a truncated response's total reward is just the length penalty, floored at −1 — the same floor as a response that finished cleanly and got the wrong answer outright. That's a deliberate design choice: it treats "ran out of budget" as no worse than "reasoned to the wrong conclusion," without ever needing to guess which one actually happened.

Overlong Filtering versus Soft Overlong Punishment, side by side

Why is_equivalent needs to be exact, not approximate

It's worth being precise about what is_equivalent, from Chapter 0's reward rule, is actually allowed to do. A "close enough" fuzzy match — textual similarity, or a small numeric tolerance — would reopen exactly the reward-hacking door Chapter 0 closed by moving to a rule-based reward in the first place: any threshold has an edge, and an optimizer pursuing reward hard enough eventually learns to land just inside that edge without actually being right, the same way a learned reward model's blind spots get found and exploited. This is why the data-transformation step from Chapter 0 (rewriting answers into a single checkable integer) matters as much as it does: it isn't just a convenience, it's what makes an exact, zero-tolerance equivalence check possible at all for problems whose natural answer format wouldn't otherwise support one.

python
def soft_overlong_penalty(length, L_max=20480, L_cache=4096):
    L_free = L_max - L_cache                       # 16,384 -- penalty-free zone ends here
    if length <= L_free:
        return 0.0
    elif length <= L_max:
        return (L_free - length) / L_cache       # linear ramp, 0 down to -1
    else:
        return -1.0                             # hard ceiling, flat penalty

total_reward = base_correctness_reward + soft_overlong_penalty(response_length)

Three branches, one if/elif/else, added directly onto whatever the base correctness reward already was. Nothing about Chapter 0's is_equivalent(ŷ, y) rule changes; this function only ever subtracts from it, and only once a response has actually crept into or past the buffer zone.

Why 4,096 tokens specifically, and what tuning it would trade off

DAPO's paper reports L_cache=4,096 as the chosen width of the soft-punishment buffer without deriving it from first principles — it's a tuned hyperparameter, not a value with a closed-form justification, and this session won't pretend otherwise. What is derivable is the general tradeoff any choice of L_cache makes: a narrower cache makes the ramp steeper, more like the original all-or-nothing cliff this chapter opened by rejecting; a wider cache spreads the same −1 total penalty across a longer stretch of tokens, giving the model more graduated warning before the hard ceiling, but also diluting the penalty's per-token urgency near the free zone's edge. L_cache=4,096 against L_max=20,480 sits at 20% of the hard ceiling — wide enough to meaningfully grade the penalty, without eating so much of the budget that the penalty-free zone becomes a small fraction of the total generation length.

L_max is also an inference-cost decision, not just a training one

It's worth naming what L_max actually trades off, beyond the reward-shaping mechanics this chapter builds. Every token of generation budget costs real compute and real wall-clock time, both during RL training's rollout phase and, later, at actual inference time once the trained model is deployed. Setting L_max too low starves the model of room to reason through genuinely hard problems; setting it too high means paying for a long tail of rarely-used capacity, and — per this chapter's own diagnosis — creates more opportunities for truncation-related reward noise if the reward shaping around it isn't handled carefully. DAPO's 20,480-token ceiling is a specific answer to this tradeoff for math reasoning at the scale this session studies; a different task with typically shorter or longer solutions would reasonably choose a different number, using the exact same underlying reasoning.

Why this interacts with everything Chapters 4 through 7 already built

Overlong Reward Shaping isn't an isolated bolt-on; it closes a loop the earlier chapters opened. Chapter 4 showed GRPO's own loss function structurally favors longer incorrect responses. Chapter 5's Dr. GRPO fix removes that structural bias from the loss itself, but does nothing about a response that grows long enough to blow through the token budget entirely — a separate, purely operational failure mode with a separate fix. Left unaddressed, the two problems would compound in the worst possible direction: a policy nudged toward generating longer responses (whether from genuine reasoning or a lingering bias) eventually starts producing responses that get truncated, and a naive truncation penalty would then punish exactly the kind of sound-but-verbose reasoning that a lingering bias was already tempting the policy toward in the first place. Soft Overlong Punishment breaks that particular feedback loop at its most direct point of contact: the reward function itself, right where a truncated response's fate gets decided.

Measured effect

In DAPO's progressive ablation, adding Soft Overlong Punishment on top of Overlong Filtering plus Clip-Higher moves the score from 38 to 41 — a three-point gain, on top of Overlong Filtering's own six-point gain earlier in the same table (from naive GRPO's 30 up to 36). Both numbers belong to the same underlying insight — don't let truncation inject noise into the reward — approached first crudely (discard truncated samples' gradient entirely) and then more precisely (grade the penalty by how far over the line the response actually went).

Chapter 9: Reading the AIME Numbers Honestly

Every technique in this session has a citation and a mechanism. This closing chapter puts the numbers side by side and asks the question a careful engineer should always ask of an ablation table: which of these techniques actually earned the points, and which ones matter for reasons an accuracy score doesn't fully capture?

The full training configuration, for the first time in one place

Every chapter in this session has surfaced one piece of DAPO's actual training configuration at a time — worth collecting into one place now that the whole algorithm has been assembled. DAPO trains on Qwen2.5-32B with the verl framework, using AdamW at a constant learning rate of 1×10⁻⁶, with a linear warm-up over the first 20 rollout steps. Each rollout step samples a batch of 512 prompts, drawing G=16 responses per prompt (Chapter 7's assembled training loop), for a mini-batch size of 512 — meaning 16 gradient updates happen per rollout step before the policy generates fresh data again. Evaluation on AIME uses avg@32 (Chapter 0) at temperature 1.0 and top-p 0.7. Every number this session has quoted from DAPO — ε_low=0.2, ε_high=0.28 (Chapter 6), L_max=20,480 with a 4,096-token soft cache (Chapter 8) — was measured inside exactly this configuration, not some idealized or simplified version of it.

DAPO's own ablation, in full

DAPO reports a progressive ablation on Qwen2.5-32B, adding one technique at a time on top of the naive GRPO baseline, each row's AIME24 avg@32 score built cumulatively on everything above it:

Read the gain column, not just the final number. Dynamic Sampling — the technique framed in Chapter 7 as essentially housekeeping, filtering out zero-gradient groups rather than any change to the loss function's mathematical form — contributes the single largest jump in the entire table, +8 points, more than Clip-Higher, Soft Overlong Punishment, and Token-Level Loss combined (+6 together). And Overlong Filtering, the crude, throw-it-away diagnostic version tried before the more refined Soft Overlong Punishment, is itself worth more points (+6) than the refined version that superseded it (+3 on top). The headline story — “four sophisticated algorithmic techniques” — undersells how much of the real gain came from two comparatively unglamorous fixes: don't waste compute on degenerate batches, and don't let truncation poison the reward signal. This is precisely the kind of result an ablation table is for: it tells you where the engineering effort should have concentrated, which the paper's own prose narrative, read on its own, would not have made obvious.

One more way to read the same table: percentage of the total gain

Framed this way, two techniques — Dynamic Sampling and Overlong Filtering — together account for 70% of the entire gap this session opened with in Chapter 0. The other three techniques, combined, account for the remaining 30%. None of this diminishes what Clip-Higher, Soft Overlong Punishment, or Token-Level Loss actually fix — each addresses a real, distinct failure mode, and the caveat below about not treating one ablation run as universal still applies — but it does mean that if you had to prioritize which two fixes to implement first on a limited compute or engineering-time budget, this specific run's numbers say clearly where to start.

What DAPO's final number is, and isn't, comparable to

DAPO's 50-point result comes with two comparisons worth separating carefully, because conflating them would be a mistake. First, the paper's own framing: 50 points on Qwen2.5-32B, beating DeepSeek-R1-Zero-Qwen-32B's 47 points, using only about 50% of DeepSeek's training steps — a genuine, apples-to-apples improvement, same model size, same task, less compute spent reaching a higher score.

Second, and this is where care matters: Dr. GRPO's minimalist recipe, from Chapter 5, reaches 43.3% on AIME24 using Qwen2.5-Math-7B — a model roughly one-quarter the parameter count of DAPO's Qwen2.5-32B — in 27 hours on 8×A100 GPUs, an unambiguously smaller compute budget than a full-scale system like DAPO's. It would be a mistake to read this as “Dr. GRPO alone gets 86% of the way to DAPO's result for a fraction of the cost,” because the two numbers are not measuring the same thing: different base model, different parameter count, different training data, different template, and DAPO's full system includes techniques (Clip-Higher, Dynamic Sampling, Overlong Reward Shaping) that the minimalist recipe does not use at all. What the two results together do honestly establish is that meaningful R1-Zero-style gains are reachable from two largely independent directions — fixing GRPO's own mathematical bias (Dr. GRPO's route) and engineering around GRPO's observed training-time failure modes at scale (DAPO's route) — and that neither paper's authors claim these are the same fix wearing two names.

What the training curves actually looked like, monitored live

Numbers in a final results table hide what training actually looked like along the way. DAPO's own account tracks four curves throughout every run as monitoring indicators: response length, reward, generation entropy, and mean token probability. Each carries a specific, sometimes counterintuitive lesson for anyone running a similar system.

Length correlates with training stability, but not in a simple "always up" way — over considerable stretches of training, length can stagnate or even decline, a pattern also documented in the DeepSeek-R1 report. The practical upshot: length has to be read together with validation accuracy to tell whether an experiment is deteriorating, not read alone as a proxy for "is the model still improving." Reward on the training set tends to increase in a smooth, stable trend across most runs — a sign the model can reliably fit whatever distribution the training data represents — but there's an important gap: final training-set reward often shows little correlation with validation-set accuracy, a direct symptom of overfitting to the training questions specifically, not to reasoning ability in general. Entropy and mean generation probability are the two sides of the exploration-collapse story Chapter 6 already built in full: entropy too low means a distribution too sharp to explore with, too high means gibberish and repetition, and a slow, gradual upward entropy trend, rather than a spike or a flat line, is what actually correlates with improving performance.

A live transcript: self-correction caught mid-training

Chapter 3 asked whether self-reflection is genuinely new behavior invented by RL, or an amplification of something already latent in the base model. DAPO's own training case study offers one more piece of direct evidence, from the other side of training. Partway through a run, on a geometry question about the volume of a tetrahedron given a dihedral angle and an orthocenter condition, the model's sampled response includes this moment mid-solution:

That's a live, in-context example of exactly the self-correcting behavior Chapter 3 discussed in the abstract: the model catches itself mid-derivation, names what's wrong with its own approach, and switches strategy without being told to. The broader observation across many such transcripts: in the earliest stages of training, this kind of checking-and-reflecting-on-previous-steps behavior was virtually absent; as training progresses, it becomes a distinct, recognizable pattern. Read next to Chapter 3's finding that this capability already existed in the base model before any RL, the fuller picture is neither "RL invents this from nothing" nor "RL does nothing here": the behavior exists early, is rare early, and becomes common as training reinforces the trajectories where it shows up and helps.

Connections: where this sits in the RL-for-LLMs landscape

This session builds directly on two topics elsewhere in this course. Policy gradients covers the REINFORCE score-function trick and baseline subtraction that both PPO's advantage estimator and GRPO's group-relative advantage are specific instances of — if the log-probability gradient and variance-reduction argument in Chapter 2 felt unfamiliar, that lesson builds it from first principles. RL algorithms places PPO, GRPO, and their relatives inside the broader taxonomy of on-policy versus off-policy, model-free methods — useful context for seeing why an on-policy method with a trust region (PPO's clip) was the starting point these papers all inherited, rather than something with a replay buffer.

Session 06 of this course, on DPO, is worth revisiting from the other direction now: DPO sidesteps RL entirely, turning a preference-comparison objective into a closed-form supervised loss with no sampling loop at all. Everything in this session assumes you specifically need an RL loop — a setting where a verifiable reward exists and repeated online sampling from the current policy is affordable. When that assumption doesn't hold (open-ended preferences without a rule-based check, or a compute budget that can't support a sampling-and-scoring loop), DPO-style methods are the more natural tool, not a worse version of the same idea.

What this session did not cover, honestly

Neither paper studied here claims to have solved RL-for-reasoning in general. Both are scoped specifically to math, with rule-based, exactly-checkable rewards — code generation, open-ended writing, and multi-turn agentic tasks all have messier, harder-to-verify reward signals where the entire “rule-based reward has no hacking surface” argument from Chapter 0 weakens or breaks down. And R1-Zero's own base-model analysis in Chapter 3 is itself scoped to a handful of specific model families; whether its findings about templates and pretraining bias generalize to every future base model is an open empirical question, not something either paper resolves once and for all.

One more scope boundary worth stating plainly: this session studied outcome-reward RL, where a single scalar reward arrives once, at the end of a complete response. Process-reward approaches, which reward individual reasoning steps along the way rather than only the final answer, are a related but distinct research direction this session did not build from scratch. GRPO's own advantage formula is noted, in the literature it's drawn from, to apply to process-reward cases as well as outcome-reward ones — but working through exactly how the per-step reward changes the group-relative arithmetic Chapter 2 derived is left as an exercise, not covered chapter by chapter the way the outcome-reward case was here.

The whole session, one chapter per row

Ten chapters, one throughline: start from PPO's full machinery, delete what a verifiable reward makes unnecessary, find the bias that deletion quietly introduces, fix it, then layer on the four engineering fixes that took one team's naive reproduction from 30 points back up past the number they were trying to beat.

A practical checklist, if you're about to run one of these yourself

Collapsing ten chapters into a pre-flight list: verify your reward function has no learnable component before trusting it against reward hacking (Chapter 0). Confirm whether you actually need a critic at all, or whether your setting affords the same-question-many-samples structure GRPO's baseline needs instead (Chapters 1–2). If using GRPO's group-relative advantage, prefer Dr. GRPO's fixed-constant normalization over the original 1/|o_i| and std(R) terms, unless you have a specific reason to want the bias they introduce (Chapters 4–5). Watch entropy as a live metric during training, not just final accuracy, and consider a decoupled clip if it starts dropping (Chapter 6). Filter degenerate groups before they ever reach the loss, and normalize per-token rather than per-sample if response lengths vary widely across your data (Chapter 7). And decide, deliberately, what a truncated response's reward should be — not by silent default, but because a flat punitive value was actually checked against the alternative and found wanting (Chapter 8).

Back to where this session started

Chapter 0 opened with DAPO's own blunt framing: the actual algorithm and key recipe for scalable RL training “remained a myth,” hidden from the technical reports of every major reasoning model at the time. Ten chapters later, that myth has a name for every piece of it: a rule-based reward instead of a reward model, a group-relative baseline instead of a critic, two specific mathematical biases and their fix, and four concrete engineering techniques with individually measured contributions. None of it required access to a frontier lab's internal infrastructure to understand — every equation, every worked number, and every code snippet in this session came from openly published papers, derived and checked by hand. That's the entire premise this session was built on: a “myth” is often just an explanation nobody has bothered to write down carefully yet.

Both papers this session is built on remain the primary source for anything not fully reproduced here: DAPO: An Open-Source LLM Reinforcement Learning System at Scale for the four engineering techniques and the full training configuration, and Understanding R1-Zero-Like Training: A Critical Perspective for the base-model analysis and Dr. GRPO. Every number, quote, and table in this session traces back to one of these two papers directly, not to a secondary summary of them.