Interactive explainer · Phase 3 — Training & systems
Scaling laws
Before you spend a single GPU-hour, a three-term formula tells you roughly what loss you'll get for any model size and token count — and therefore how a fixed budget should be split between the two. This page is that formula made draggable. It's also a calculator you'll actually reuse: every Phase 3 experiment starts with "what does this cost?"
— irreducible entropy, a params term, a data term. That's the whole law.
§ 1 · The law, live
Two dials, one predicted loss
Drag N and D and read off the loss a Chinchilla-style law predicts. The bar shows which term is holding you back: grey is the entropy of language itself (no model removes it), vermillion is "too few parameters", moss is "too few tokens". Whichever colored term is bigger is where your next FLOP should go.
predicted loss
2.530
tokens / param
20.0
near the compute-optimal ridge
params and tokens are pulling roughly equal weight — this budget is well spent, by the pretraining-loss yardstick.
fig. 1 — constants from the Epoch AI replication (Besiroglu et al. 2024) of Chinchilla's approach-3 fit, on MassiveText. Treat the shape as the lesson, not the digits.
§ 2 · One budget, spent well
The U-curve every lab argues about
Fix a compute budget C = 6ND and sweep the model size: too big and it's under-trained, too small and it's data-saturated. The bottom of the U is the compute-optimal split — and it lands near ~20 tokens per parameter at every budget. Park the gold marker where you'd have sized it and read the loss you'd give up.
fig. 2 — the optimum is found numerically from fig. 1's formula, not hardcoded. Slide C across seven orders of magnitude: the ratio barely moves.
§ 3 · What it costs
FLOPs → hours → dollars
A budget in FLOPs is an abstraction; a rental invoice isn't. Pick hardware, set an honest MFU (40% is good; 60% is a systems paper), and the §2 budget becomes wall-clock and money. The presets are the budgets you'll actually meet in Phase 2–3.
hardware
- budget (set in §2)
- 1.0e21 FLOPs
- 8×H100 node · 8 × 989 TF dense bf16
- ×40% MFU
- wall-clock
- 87.9 h
- ballpark cost
- $2,110
load a real budget
GPT-2 124M · 10B tokens
The classic first reproduction — nanoGPT territory.
Chinchilla-optimal 1B
A 1B model fed its ~20 tokens/param ration.
nanochat speedrun
The "$100 ChatGPT" tier — a few hours on one rented node.
fig. 3 — peak numbers are dense bf16, not the sparsity-doubled marketing sheet. Rental prices are editable defaults; they drift monthly.
§ 4 · Where the law bends
Chinchilla assumes you never serve the model
Compute-optimal minimizes training cost — but every parameter you keep costs 2N FLOPs on every token you ever serve. Tick "the model will be served" and watch the cheapest point slide toward smaller models trained far past 20 tokens/param. That's the Llama/SmolLM regime, and it's why the small-models field over-trains on purpose.
Repeated epochs bend it
The fit assumes fresh tokens. Past ~4 epochs on the same data, extra passes buy less — data-constrained scaling has its own paper (Muennighoff et al.).
Loss ≠ downstream ability
The law predicts pretraining loss. Benchmarks, reasoning, and chat quality correlate with it — loosely, and less at the frontier. Phase 4 is about measuring what you actually care about.
The constants are one dataset
E, A, B, α, β were fitted on MassiveText. Different data (or tokenizer) refits them. Carry the shape between projects, never the digits.
fig. 4 — the Sardana–Frankle framing: minimize train + lifetime-inference FLOPs at a fixed quality bar. The dashed curve is training cost alone.
§ 5 · The whole trick
Three numbers to carry into every experiment
compute splits between N and D
Six FLOPs per parameter per token. Every budget question starts here.
~20 tokens/param is the ridge
Chinchilla's answer at every budget — if training loss is all you optimize.
serving bends it small
Inference bills by the parameter. Models meant to be used are trained long past the ridge.
The law is a planning tool, not a physics constant — it moves with data quality, which is why the next lab is about data, and why "small but well-trained" is a real niche rather than a consolation prize. Before any Phase 3 run: price it here first.