Interactive explainer · Phase 4 — Post-training & eval

RLHF, DPO, GRPO

SFT shows the model what good looks like; preference methods teach it what better looks like. Three recipes dominate, and they share one idea: turn a comparison into a gradient. Every panel below computes the real loss functions on numbers small enough to read — no training run required to see how each method thinks.

$L_{\text{DPO}} = -\log \sigma\!\left(\beta\left[\log\tfrac{\pi(y_w)}{\pi_{\text{ref}}(y_w)} - \log\tfrac{\pi(y_l)}{\pi_{\text{ref}}(y_l)}\right]\right)$ — one pair, one sigmoid, no reward model.

§ 1 · The atom

A preference is a probability

Nobody can write down "this answer is worth 7.3 points" — but anyone can say which of two answers they prefer. The Bradley–Terry model turns those comparisons into a scale: the bigger the reward gap, the more confidently the comparison should go one way. Drag the gap and watch the probability.

prompt · Explain overfitting in one sentence.

chosen · reward r_w

The model memorizes quirks of the training set instead of the underlying pattern, so it aces training data and flops on anything new.

rejected · reward r_l

Overfitting is when a model overfits the training data too much.

reward gap1.50

P(chosen ≻ rejected) = σ(gap) = 81.8%

This one function is the load-bearing assumption under all of RLHF: labelers only say which answer is better, and Bradley–Terry turns those comparisons into a scale. A reward model is just a machine trained to produce these gaps — and everything downstream inherits whatever it gets wrong.

§ 2 · DPO

Skip the reward model

DPO's trick: the policy itself is the reward model. Define each response's implicit reward as β times how far the policy has drifted from the reference on that response — then push the chosen response's reward above the rejected one's, through the same sigmoid as §1. Every term is live below.

log-ratio, chosen1.50log-ratio, rejected-0.50β — the leash0.10

log-ratios are log π(y)/π_ref(y) — how far the policy has moved from the reference on each response. reference = 0 by definition.

implicit reward, chosen: 0.15
implicit reward, rejected: -0.05
margin: 2.00
loss: 0.598
gradient weight σ(−β·margin): 0.450

Two things to feel here. The gradient weight falls as the margin grows — DPO stops pushing on pairs it already ranks correctly and spends its gradient where it's still wrong. And β sets the leash: drag it down and the loss curve flattens — the policy can drift far from the reference before the loss cares; crank it up and tiny drifts saturate the loss immediately.

fig. 2 — the dashed line is ln 2, the loss on a pair the policy has no opinion about. Whatever β you pick, the curve passes through it at margin 0.

§ 3 · GRPO

The group is the baseline

GRPO samples a group of attempts at the same prompt, scores each with a programmatic checker (right answer = 1, wrong = 0), and normalizes within the group: advantage = (reward − group mean) / group std. No value network, no reward model — the other attempts are the baseline. Toggle the checkmarks and watch the advantages rebalance; then make them all correct.

prompt · Compute 17 × 24. — click an attempt to flip what the checker returns

4/8 correct · group mean 0.50 · std 0.50

Above-average attempts get pushed up, below-average pushed down — the group mean is the baseline, so no learned critic is needed. And the reward is a checker (did it print 408?), not a learned model: that's RLVR, and it's the entire small-reasoning-models frontier — verifiable rewards can't be flattered, only gamed.

§ 4 · The family

Three recipes, honestly compared

RLHF (PPO)

what it eats: human comparisons → train a reward model → online rollouts scored by it
extra models it needs: reward model + value network + frozen reference — four models in memory
what goes wrong: reward hacking: the policy finds the gaps in the learned proxy

DPO

what it eats: offline preference pairs, straight into the loss — no reward model step
extra models it needs: just the frozen reference model
what goes wrong: overfits its fixed dataset; pairs go stale as the policy drifts from whoever wrote them

GRPO / RLVR

what it eats: online rollouts scored by a programmatic checker — no humans in the loop
extra models it needs: just the frozen reference — the group mean replaces the critic
what goes wrong: uniform groups (all right or all wrong) give zero gradient; KL/entropy collapse when the leash slips

The roadmap's ordering advice stands: SFT + DPO is the safe first post-training project — offline data, one frozen reference, failures you can debug. GRPO is the exciting frontier and a debugging swamp (KL control, reward hacking, vLLM colocation, wall-clock ≫ GPU-hours) — earn it second. The RLHF Book is the spine for all of this.

§ 5 · The whole trick

Comparisons all the way down

a comparison is a gradient

P(y_w \succ y_l) = \sigma(r_w - r_l)

Bradley–Terry turns "this one is better" into a differentiable target. Everything else is plumbing.

DPO's reward is implicit

r(y) = \beta \log \tfrac{\pi(y)}{\pi_{\text{ref}}(y)}

Distance from the reference IS the reward. β sets how long the leash is.

uniform groups teach nothing

A_i = \tfrac{r_i - \bar{r}}{\text{std}(r)}

GRPO's baseline is the group mean — all-right or all-wrong prompts produce zero gradient.

The Phase 4 deliverable applies this page: SFT a small model, then run DPO on a focused preference set — and put your real effort into the evaluation, because the reward you optimize is never quite the quality you meant.

phase 4 — post-train & measure honestly →sibling lab — reward hacking DPO paper ↗DeepSeek-R1 ↗