Interactive explainer · Phase 4 — Post-training & eval
Sampling & inference
Training set the probabilities; decoding decides what you actually see. Temperature, top-k, top-p, greedy-vs-sampled — same weights, different knobs, and the text (and the benchmark score) changes. That's why decoding parameters belong in every eval report, and why "the model said" always means "the model, decoded this way, said."
— one divisor between deterministic and delirious.
§ 1 · The knobs
Temperature reshapes, truncation deletes
One real next-token distribution — ten candidates after "The bird ate the …" — pushed through the standard pipeline: divide logits by T, softmax, apply top-k and top-p, renormalize. Ghost bars show the mass each excluded token had before truncation deleted it. Drag T to 0.1, then to 2; watch "car" die and resurrect.
the pipeline · logits ÷ T → softmax → top-k ∩ top-p → renormalize
entropy 2.43 bits · 10/10 tokens kept
10 of 10 candidates in the running — mass from the excluded tail was renormalized onto them.
fig. 1 — the distribution is hand-crafted; the pipeline arithmetic is the real thing.
§ 2 · The failure you've seen
Greedy loops, sampling wanders
A twelve-word Markov chain stands in for a language model — small enough that you can verify the loop by hand. Greedy decoding takes the argmax every step, and the argmax path has a cycle: the bird ate the bird ate… Sampling follows the same probabilities and escapes. Every "why does my model repeat itself" bug report is this demo at scale.
Every step picks the single most likely next word, and the chain has a most-likely loop — so greedy walks it forever. Repetition is what maximum-likelihood decoding does to a language model; the fix is a decoding choice, not a training one.
§ 3 · The eval tie-in
Decoding changes the measured number
Give a toy solver a success rate that depends on temperature, then score it two ways with the exact pass@k formula: . At T→0 all eight samples are the same attempt, so pass@8 collapses onto pass@1; heat restores the independence that multiple attempts pay for. The curves peak at different temperatures — pick your knob for the metric you report.
pass@1 18.0%
pass@8 55.3%
The two curves peak at different temperatures: cold decoding maximizes one-shot accuracy; heat buys the diversity that pass@8 pays for. Two papers "evaluating the same model" at different T and k are measuring different systems — which is why decoding params belong next to every number you publish.
fig. 2 — the success-vs-T curve is hand-shaped for legibility; the pass@k arithmetic and the duplicate-samples-at-low-T mechanism are real. The evals lab picks this thread up.
§ 4 · Free tokens, exactly
Speculative decoding
A small draft model proposes γ tokens; the big model checks them all in one forward pass and keeps the agreeing prefix (plus one token of its own). Expected yield per pass: where α is how often the draft guesses right. Below, the curve, the strip, and the memory bill that explains why verification is nearly free.
KV cache = 2 · 16 layers · 2048 dims · 2 B = 128 KB/token → 4.0 GB at this context — memory the GPU re-reads for every single generated token.
Two honest facts: the output distribution is exactly the target model's — rejected drafts are resampled so the math is lossless, not approximate. And the trick only pays because decode is memory-bound: verifying γ tokens in one forward pass re-reads the same weights once instead of γ times — the roofline from the GPU-systems lab, cashed in.
§ 5 · The whole trick
Decoding is part of the model
temperature reshapes, truncation deletes
T bends the whole distribution; top-k/top-p amputate the tail and renormalize the survivors.
greedy is a different model than sampled
Same weights, different text, different benchmark score. Report T, k, p, and n — always.
speculation is free because decode is memory-bound
Verifying γ tokens re-reads the weights once. Lossless by construction — rejection sampling, not approximation.
Everything here runs after training is over — which means it's the cheapest place to change a model's behavior, and the easiest place to fool yourself when you measure it. When you post-train your own small model in this phase, fix your decoding params before you trust a single number.