Interactive explainer · Phase 2 — Transformers & LLMs
The transformer block
A GPT is one unit repeated: attention, an MLP, two additions, two normalizations — stacked 12, 48, 96 times. Attention (the previous lab) is the clever part; this page is about the plumbing that lets you stack it deep — the residual stream that carries information up, and the normalization that keeps every layer on stable footing. The plumbing looks boring. It's why any of this trains.
— the whole block. two edits to a stream that flows through untouched.
§ 1 · The backbone
Edit the stream, don't rewrite it
Pretend each layer just multiplies the signal by a gain g. A bare 48-layer stack applies g forty-eight times — exponential, so anything but g = 1.00 vanishes or explodes (a straight line on this log plot). A residual layer instead adds a small edit to an untouched copy: the same imperfect g now barely dents the stream. Drag g away from 1 and watch the gap.
after 24 layers, a unit signal becomes 0.08 bare vs 0.79 with residuals — the bare stack starves the top layers of signal (and, mirrored, the bottom layers of gradient).
fig. 1 — a scalar caricature (real layers are matrices and the "edit" is damped here by 0.1, like careful init); the honest part is the shape: multiplication compounds exponentially, addition doesn't. the same story holds for gradients flowing down.
§ 2 · Constant footing
Normalize before every branch
The stream accumulates edits for 48 layers, so its scale drifts. Attention and MLP weights are tuned for inputs of a particular size — feed them something 6× larger and softmax saturates, gradients die. The fix: before each branch, reset the vector to a standard scale. Drag the drift sliders; the output barely moves.
the stream after some blocks — drifted
mean 1.24 · std 2.02 · rms 2.37
what the next branch actually receives
mean -0.00 · std 1.00 · rms 1.00
recipe
LayerNorm: subtract the mean, divide by the std — drag either slider anywhere, the output is always mean 0, std 1. The next block starts from the same footing every time.
fig. 2 — one token's 6-dim slice of the stream, really normalized live. (production models add learned per-dim scale γ back after the reset — omitted here; it re-introduces scale on purpose, not by drift.)
§ 3 · Division of labor
Attention mixes, the MLP thinks
The block has exactly two working parts, and they touch the tokens in opposite ways. Hover a row in either grid — same sentence as the attention lab:
attention · mixes across positions
Row = output token, column = where it reads from. "it" pulls from "bird" four positions away — information crosses tokens here and only here.
mlp · thinks per position
Same rows, same columns — and nothing off the diagonal. Each token is processed alone, by the same weights, all 9 in parallel. A per-token function applied 9 times.
hovering row: "it" — attention decides what flows between tokens; the MLP decides what to make of it. Remove attention and no token ever learns about another; remove the MLP and the network is (nearly) just weighted averaging.
§ 4 · Assembled
Walk the block, piece by piece
Put the three ideas together and the block builds itself: normalize, mix, add — normalize, think, add. Step through and read why each piece is there:
boxes = the two branches · circles = the residual stream absorbing their edits
the block, unassembled — press start and walk through it piece by piece, left to right. every GPT layer you will ever meet is this exact sequence.
§ 5 · The whole trick
Boring plumbing, deep networks
residual = edit, don't rewrite
Addition instead of replacement — signal and gradient ride an unbroken highway through 48 layers.
norm = constant footing
However far the stream drifts, every branch receives unit-scale input. RMSNorm skips μ and nobody misses it.
attention mixes, MLP thinks
Information crosses positions only in attention; the MLP holds most of the parameters and does the per-token work.
Stack N of these blocks between an embedding (words → vectors) and an unembedding (vectors → next-word logits) and you have the entire GPT architecture — there is no other secret ingredient. Which means you can now read nanoGPT's ~300 lines and recognize every one of them (frozen at its 2025 deprecation — readable code doesn't rot; nanochat is the living successor). That's the next rep: build the block in PyTorch, from this page, without looking.