Interactive explainer · Phase 2 — Transformers & LLMs
Self-attention
Every layer of a transformer, each word gets to rewrite itself as a blend of the words before it. Attention is the routing rule that decides the blend: each word broadcasts a question, every earlier word holds up an answer, and the match between them — one dot product per pair — sets who contributes how much. That's the whole mechanism. Below, it runs live on one sentence, small enough to read every number.
— one matrix of questions, one of answers, one of payloads.
§ 1 · The problem
A word alone knows nothing
The word "it" is an empty pointer — its meaning lives in some other word. Attention is how the model fills it in. Hover or tap any word to see where it looks; the underline weight is its real attention distribution, computed live from the Q·K arithmetic you'll meet in §3. Faded words sit in the future — a decoder can't read them.
"it" spends 36% of its update on "bird", 18% on "worm" — the star of the sentence. An empty pointer whose query screams "animate thing?" — and "bird" answers loudest. This is pronoun resolution as arithmetic.
fig. 1 — the queries and keys behind this are hand-crafted so every number stays readable; the scores, mask, and softmax are the genuine arithmetic.
§ 2 · Three hats
Ask, advertise, offer
Each word's embedding is projected three ways: a query ("what am I looking for?"), a key ("what do I advertise?"), and a value ("what will I hand over if picked?"). In a real model these live in hundreds of unlabeled dimensions the model invents for itself; here we hand-built four legible ones so you can read the vectors like a form.
pick a word
query — what it asks for
key — what it advertises
"it" — the star of the sentence. An empty pointer whose query screams "animate thing?" — and "bird" answers loudest. This is pronoun resolution as arithmetic. A high score needs the query and a key to be loud on the same dimensions — that's all a dot product measures.
§ 3 · The score sheet
Every query meets every key
Dot every query with every key and you get the score matrix — rows ask, columns answer. Walk the pipeline: raw scores → scale → mask → softmax. Hover any cell to see the arithmetic, dimension by dimension.
Raw agreement: every query dotted with every key. Big number = "you have what I'm looking for." Note the hot cell at row it × col bird.
hover any cell — each one is a single query·key dot product.
fig. 2 — notice what's missing: nothing in QKᵀ knows where a word sits. Shuffle the sentence and every score survives. That blindness is why attention needs positional information (RoPE) — the next explainer in this phase.
§ 4 · The mix
Output is a weighted blend
The softmax row is a spending budget: each word spends 100% of its update across the values of the words it attended to, then adds the result to its own vector (the residual stream — the detail that lets deep transformers train at all). This is the payoff: after this layer, "it" literally contains mostly bird.
what flows into…
- "bird"36%
- "worm"18%
- "it"16%
After this layer, the vector at position 7 is no longer generic "it" — it's 36% bird. The next layer up reads that enriched vector and never knows a pronoun was there.
§ 5 · Many heads
Different heads learn different jobs
One attention pattern can't do everything, so the layer runs several heads in parallel — same mechanism, different learned projections — and concatenates their outputs. Real trained models grow weirdly specialized heads; these three minis are caricatures of patterns actually found in GPT-2.
head 1 · the resolver
The head from §1–§4: pronouns and adjectives hunt down the animate subject. Content-based lookup.
head 2 · the previous-token head
Attends one step back, always. Sounds trivial — real GPT-2 has these, and they feed the induction heads that do in-context learning.
head 3 · the verb tracker
Keeps every word in touch with the verbs ("ate", "was") — a caricature of a syntactic head. Head outputs are concatenated, then mixed by W_O.
§ 6 · The whole trick
Three moves, one matrix multiply
score = agreement
A dot product only measures whether the query and key are loud on the same dimensions.
softmax = a budget
Each row becomes 100% of attention to spend. The mask just removes the future from the shop.
output = the blend
Weighted sum of value vectors, added back to the residual stream. That is the entire layer.
Everything on this page is a handful of small matrices — no magic survived. If it made sense, you're ready for the real Phase 2 deliverable: close this tab and implement multi-head causal self-attention from a blank file, then train it inside a GPT. Rebuild it until it's boring — that's the milestone.