Interactive explainer · Phase 2 — Transformers & LLMs
Tokenization
A language model never sees letters. It sees integers — and the tokenizer is the frozen little program that decides which integers. Every strength and every strange weakness downstream (arithmetic, code, other languages) is shaped here first, which is why the roadmap calls it an underrated source of bugs and quality. Below: the actual algorithm, running live, small enough to read.
"the bird" → [31, 15, 24, 40, 9]— that list is all the model ever gets.
§ 1 · The tradeoff
Characters, words, or something in between
The same sentence, tokenized three ways. Characters make the vocabulary tiny but the sequence enormous; words make the sequence short but the vocabulary unbounded. Subword tokenization is the negotiated middle — and BPE, below, is how the split points get chosen: by frequency, not by linguistics.
characters
40 tokens · vocab ~50 symbols
tiny vocabulary, but every sequence is enormous — the model wastes its context window spelling.
words
7 tokens · vocab ~600,000 words
short sequences, but the vocabulary explodes — and "tokenizer" might still be missing. One typo = unknown token.
subword (BPE)
18 tokens · vocab 89 here · 50–200k in real models
the compromise: frequent strings get one token, rare ones fall apart into reusable pieces. Nothing is ever unknown.
§ 2 · The algorithm
Train a BPE tokenizer, one merge at a time
Byte-pair encoding is embarrassingly simple: spell every word out, count adjacent pairs, glue the most frequent pair into a new token, repeat. This is the real algorithm running on a 50-word corpus — step through it and watch "t h e" become "the", then watch "the" swallow its own end-of-word marker ⌟.
no merges learned — every word is spelled out character by character, 253 tokens of pure alphabet. press the button and watch the corpus coarsen.
fig. 1 — the vermillion chunks are the token the last merge created. GPT-2 ran this loop 50,000 times over bytes; you're running it {0…39} times over characters.
§ 3 · The payoff
Tokenize anything with what it learned
Your text, encoded with the merges from §2. Drag the slider and watch the token count fall as the vocabulary grows — that's the compression the model's context window lives on. Common words collapse to one token; anything the corpus never saw stays expensive.
52 tokens for 43 characters · vocab 50 · zero merges — train some below or in §2
§ 4 · How it hurts
Four ways a tokenizer sabotages a model
Each card loads a pathological input into the tokenizer above. The failures are toy-sized (this tokenizer learned from 50 words), but every one of them has a famous real-world counterpart — tokenization is where "the model is weirdly bad at X" stories usually start.
arithmetic
Digits never merged (our corpus had none), so "12345" is five tokens. GPT-2 was worse: frequency merged "123" into one token but split "1234" into two — inconsistent chunking is one reason LLMs struggle with arithmetic. Modern models special-case digits.
indented code
Every space burns one token here. GPT-2 had exactly this problem — one token per space made indented Python cripplingly expensive — so OpenAI added dedicated multi-space tokens for Codex. The tokenizer decides what the model is cheap at.
outside the alphabet
é and ï aren't in our alphabet at all — dashed chips, no id, the model would never see them. Real byte-level BPE has no unknowns (every byte is a base token), but the cost moves: rare scripts fragment into many byte-tokens. Same sentence, 3× the tokens, worse quality.
a made-up word
The corpus never taught bird-adjacent subwords, so "unbirdlike" shatters into characters and meaning must be reassembled from shards. Real vocabularies contain junk shards too — the infamous SolidGoldMagikarp token was so undertrained it made GPT-3 glitch on sight.
§ 5 · The whole trick
Compression decides the alphabet
frequent pairs become atoms
BPE has no idea what a word is — it only counts. Whatever repeats in the training corpus gets one id; linguistics is an accident of frequency.
the vocab freezes at training time
The merges are learned once, then bolted onto the model forever. Whatever the corpus lacked — digits, code, another language — pays per-character for the model's whole life.
weird text costs tokens
More tokens for the same meaning = less effective context, higher cost, and shards the model barely trained on. "How was this tokenized?" is a real debugging question.
You just watched the entire algorithm — there is no more magic in it. The real Phase 2 rep is building exactly this over bytes with regex pre-splitting: Karpathy's minbpe is the reference, and it's a weekend. Then go poke at how GPT-4 actually splits your text.