Interactive explainer · Phase 3 — Training & systems
Optimization at scale
The Phase 1 lab taught you the optimizers. At scale the questions change: the gradient is no longer a fact but a poll of B examples, and someone pays for every extra vote — in wall-clock, in memory, or in tokens. This lab is about those three bills.
— the whole phase in one line: more samples, less noise, same truth.
§ 1 · The gradient is an estimate
A batch is a poll
Nobody computes the true gradient — that would mean the whole dataset every step. You poll B examples and act on the average. Below, the same ravine, the same learning rate, the same seeded noise — only B changes. Quadruple the batch and the noise halves (σ/√B); the trajectory goes from drunken walk to descent. Notice it never truly settles: near the bottom the true gradient vanishes but the noise doesn't, so the run orbits in a noise ball whose size B controls.
- noise per coordinate · σ/√B
- 1.50
- signal-to-noise at the start
- 5.29×
- settles into a noise ball of loss
- 0.121
▸ descending, but wobbling in the noise
fig. 1 — drag B and rerun: identical noise draws, rescaled. that's the entire difference between a batch of 4 and a batch of 512.
§ 2 · The critical batch size
Where parallelism stops being free
If noise falls as 1/√B, why not batch a million? Because past some point the extra votes tell you what you already know. McCandlish et al. put a shape on it: below a critical batch size, doubling B halves your steps almost for free; above it, you burn examples to save wall-clock. Every serious training run lives somewhere on these two curves.
- wall-clock cost · steps
- 9,000 (9.00× the floor)
- compute cost · examples
- 288,000 (1.13× the floor)
▸ below the knee — doubling B is a nearly free speedup
fig. 2 — the shape is the lesson, not the numbers: B_crit grows as training progresses and differs by task. measuring it for your run is the gradient-noise-scale paper's whole point.
§ 3 · Gradient accumulation
The same update, on a memory budget
Big batches need big activation memory — the one thing a single GPU doesn't have. The fix is almost embarrassingly simple: run 4 micro-batches of 8, sum their gradients, take one step. Below, both versions run on identical data with identical seeds: the hollow rings must land on the filled dots, every step, because the arithmetic is the same arithmetic. This is how million-token batches fit on one card — the price is wall-clock, never correctness.
- batch 32 at once
- 32 live examples
- 4 × micro-batch 8
- 8 live examples
- largest gap between the two trajectories, all 30 steps
- 2.2e-16 — floating-point dust
- loss floor (least-squares optimum)
- 0.059
fig. 3 — the largest gap between the trajectories is ~1e-16: floating-point summation order, nothing else. when a codebase's accumulation changes the loss, that's a bug, not a tradeoff.
§ 4 · The optimizer landscape, 2026
AdamW is the incumbent; Muon is the challenger
AdamW — the default
Adam with weight decay decoupled from the gradient, so regularization is not rescaled by √v̂. Boring, robust, what everything is tuned around.
Muon — the speedrun win
Orthogonalizes the momentum of 2-D weight matrices (Newton–Schulz), roughly ~1.35× data efficiency in modded-nanogpt. An empirical speedrun result — not settled theory.
The meta-lesson
Optimizer claims are measured in controlled speedruns with fixed data budgets, seeds, and baselines. Read the modded-nanogpt commit log like a textbook: every win is one commit, one number.
None of this replaces Phase 1: every optimizer here is still gradient times step size, and the ravine from that lab is still the failure mode they're all fighting. What changed at scale is the accounting around the step — noise, batch, memory — which is exactly what §1–§3 put on screen.
§ 5 · The whole trick
Three bills, one budget
noise ∝ 1/√B
A batch is a poll. Quadruple the sample, halve the noise — and never better than that.
below B_crit, parallelism is free
Doubling the batch halves the steps until the votes get redundant. Past the knee you pay in tokens.
accumulation is exact
Sum micro-batch gradients, step once: identical update, a fraction of the activation memory, more wall-clock.
If this page made sense, you can now read a training config — batch size, accumulation steps, LR schedule — and see the decisions instead of the numbers. The Phase 3 deliverable is to change exactly one of them on a real run and measure what happens: that's the modded-nanogpt reproduction.