Interactive explainer · Phase 2 — Transformers & LLMs
The training loop at language scale
It's the same loop you already ran — forward, loss, backward, step — with three new problems bolted on: a data firehose you can only sip from, a learning rate that has to change over time, and a number you have to learn to distrust. Below, the whole loop runs for real in your browser on the smallest language model that can learn anything at all.
— how surprised the model is, averaged over every cell of the batch.
§ 1 · Data → batches
The firehose, diced
A language model's dataset isn't rows with labels — it's one long stream, and the labels are free: every character's label is simply the next character. A batch is B random windows of T characters each. Drag the dials and watch what one training step actually sees.
the whole dataset · 284 chars · vocab 24
the bird ate the worm. the cat sat on the warm mat. the sun set and the moon rose over the quiet town. a small dog ran down the long road to the sea. the old man read his book by the fire and the rain fell all night. she sang a soft song to the wind and the stars came out one by one.
tokens per step = B × T = 4 × 8 = 32 — every cell is one exam: given everything to my left, guess the vermillion char below me. One step grades all 32 at once.
fig. 1 — real pretraining is this picture with T≈4096, B in the hundreds, and a stream of ~15 trillion tokens that the model sees roughly once.
§ 2 · The loop, live
Watch the number go down — for real
This trains a real bigram language model in your browser: one 24×24 matrix of logits — "given this char, how plausible is each next char" — cross-entropy loss, minibatch SGD. No framework, no fake curves. Press train, then crank η to the top of the dial and watch what too-hot steps do to a loss curve. The samples underneath are drawn from the live weights: gibberish organizes into letter-pair English as the loss falls.
step 0 · train — · val — · ppl — · untrained — W is all zeros, every guess uniform
the dashed val curve is graded on the held-out corpus tail the model never trains on. when train keeps falling and val doesn't, you're memorizing, not learning.
sampled from the current weights · seeded
‣ ssdhgbkvuy lceqkvldey.rimsdutfio klqfv
‣ bf dchh.f.skcoiglwecdaq.fogkdq.hvv caw
‣ cug lag yon vun.tcv.fbbcms.wlhtvwdd.g.
‣ hsrwnmiiqudfsqninyhurmcheuwtivngagvqcr
‣ gccccwgwiehmkaadtiamsse.eqmtuwlkybwrcr
fig. 2 — the model guesses the next char from the current char ONLY. that ignorance of context is exactly what the bigram-floor line measures, and what attention exists to fix.
§ 3 · The schedule
η is not a constant
Real runs don't pick one learning rate — they schedule it: warm up from ~zero (early gradients on random weights are violent, and Adam's statistics are garbage for the first few hundred steps), cruise at peak, then cosine-decay so the run can settle into a minimum instead of orbiting it. Shape the curve, then race it against a fixed η on identical batches.
§ 4 · Reading the number
Loss lies less as perplexity
Cross-entropy is exponential-scale: a drop from 3.2 to 2.5 and a drop from 1.5 to 0.8 are the same 0.7, but wildly different achievements. e^loss — perplexity — is the honest unit: "the model is effectively choosing between this many options." And always ask which loss: train loss can fall forever while the model just memorizes; the held-out val loss (the dashed curve in §2) is the only number that means anything — a discipline Phase 4 turns into a whole craft.
perplexity = e^loss
9.0
"choosing between ~9 chars"
- uniform over the alphabet · ln 24loss 3.18 → ppl 24.0
- a strong char model with real contextloss 1.00 → ppl 2.7
§ 5 · The whole loop
Three habits that scale
a batch is B×T exams
Every position in every window is a training example. Language data labels itself — the target is just the stream shifted by one.
η is the loudest dial
Too cold never arrives, too hot never settles. Schedule it: warm up while the weights are random, decay so the run can land.
read loss as perplexity
Exp-scale numbers deceive on a linear axis — and only the held-out loss counts. Train loss going down is not learning; val loss going down is.
nanoGPT's train.py is this exact page: the same get_batch dicing a stream, the same warmup+cosine schedule, the same cross-entropy read as val loss — with the 24×24 matrix swapped for a transformer. nanoGPT is deprecated and frozen now, which only makes that file better reading; its successor nanochat runs this same loop inside the full pipeline. You've now seen every part small; Phase 2's deliverable is to assemble them yourself at full size.