Interactive explainer · Phase 2 — Transformers & LLMs
Positional information
The attention lab ended on a confession: nothing in QKᵀ knows where a word sits. Attention is a bag of words — shuffle the sentence and every score survives. Two fixes exist: bolt a position vector onto each embedding (learned absolute), or rotate position into the geometry of every query and key (RoPE). Both run live below.
— rotate both, and only the offset survives. That one identity is RoPE.
§ 1 · The blindness
Shuffle the sentence, keep every score
These are the real scores from the attention lab's sentence. Shuffle the words and watch the matrix: every cell keeps its exact value — it just moves house with its tokens. The outlined cell tracks q(it)·k(bird) through each shuffle.
thebirdatethewormbecauseitwashungry
The outlined cell is q(it) · k(bird) = 3.21 — every shuffle, same value, new address. To QKᵀ, "the bird ate the worm" and "the worm ate the bird" are the same sentence. Word order — who ate whom — has to come from somewhere else.
fig. 1 — a dot product sees vectors, not addresses. word order must be smuggled into the vectors themselves.
§ 2 · Fix 1 — learned positions
A trainable vector per slot
The GPT-2 way: keep a table of one trainable vector per position and add it to the token embedding before the first layer. Position and content share the same channels — the network learns to read both out of the sum. It works, with one built-in cliff: the table only has rows for positions it saw in training. Slide past the edge.
Slot 3 has a trained vector — during training, "bird" appeared here often enough for the table to learn what "position 3" feels like. Content and position share the same channels; downstream layers learn to disentangle the sum.
fig. 2 — position vectors hand-crafted on four legible channels; real models learn hundreds of unlabeled ones.
§ 3 · Fix 2 — RoPE
Rotate the pair, keep the angle
Rotary embeddings skip the table. Inside every attention call, rotate each query by its position × θ and each key by its position × θ. A dot product only measures the angle between two vectors — and rotating both by position makes that angle depend on the offset alone. Slide i and j, then lock the offset and slide the pair.
- rotation per step θ
- 0.35 rad
- offset i − j
- 4
- q·k after both rotations
- 0.39
q·k as a function of offset alone — every (i, j) pair with the same i − j lands on the same point of this curve.
fast · θ
local order — wraps quickly
slower · θ/4
mid-range structure
slowest · θ/16
long-range position sense
real RoPE rotates many 2-d pairs at once, each with its own θ — clock hands: a second hand for local order, an hour hand for long range. drag i above and watch the speeds.
fig. 3 — genuine 2-d rotations, computed live. real heads rotate d/2 such pairs at once, each with its own frequency.
§ 4 · Why relative wins
Absolute memorizes; relative generalizes
learned absolute
- · lives in the embedding — added once, before layer 1
- · knows "slot 7", not "three words back"
- · past trained length: untrained noise, no graceful decay
RoPE — rotary
- · lives inside every attention call, applied to q and k
- · q·k depends only on i − j — relative by construction
- · extrapolates further, and stretches further still (context scaling)
attention is permutation-blind
No subscript arithmetic anywhere — shuffle the words and every score follows its tokens.
absolute positions memorize slots
A lookup table over training positions. Ask for a row it never learned and you get init noise.
RoPE makes q·k a function of distance
Rotate both sides and absolute position cancels. Distance is all that reaches the softmax.
This closes the attention lab's open loop — the transformer now knows what to look at and where things are. The milestone question "why does self-attention need positional information, and how does RoPE provide it?" should now feel almost unfairly easy: because a dot product has no address book, and RoPE hides the address in the angle.