Interactive explainer · Phase 2 — Transformers & LLMs
The full pipeline
A chat model is not trained in one go. Raw web text goes in one end and something that answers questions comes out the other — through stages that all reuse one training loop with different data. Below is the whole shape at nanochat speedrun scale (≈$100, ~4 hours, 8×H100): small numbers, real proportions. Seeing it once, end to end, is the point — Phases 3–4 then deepen each part.
web text → base → (+ domain) → chat → a number you can trust
§ 1 · The map
Four stages, one loop
Each card is a stage; the chips show what the artifact is after it runs — watch the model card change identity along the flow. Click a stage to unpack what goes in, what the objective is, what comes out, and what breaks if you skip it.
- data in
- FineWeb-EDU-style web text — ≈11B tokens at speedrun scale
- objective
- next-token prediction on raw web text; every token is a training signal
- artifact out
- a base model — it completes text; it does not answer you
- cost · speedrun tier
- ≈3–4 h on 8×H100 · ≈$80 — the overwhelming bulk of the budget
- without it
- skip it and there is no language at all — SFT on a random init produces gibberish that apes chat formatting
fig. 1 — costs are nanochat speedrun ballpark (≈$100 total). the proportions are the lesson: knowledge is expensive, behavior is cheap, measurement is almost free.
§ 2 · The behavior
Same prompt, every stage
The clearest way to feel what each stage does is to ask the same question after each one. A base model continues your text; only after SFT does it answer — and only eval tells you whether any of it actually works.
ask the model after…
> What is self-attention?
base model · continues the text
…What is self-attention? What is multi-head attention? What is a residual stream? These are the questions every candidate faces in machine-learning interviews. In this 12-week course you will learn: Week 1 — Introduction to deep learning. Week 2 — Linear algebra refresher. Week 3 — ▌
pure continuation: the base model has seen a thousand pages that look like this and keeps the pattern going. It never answers, because answering was never the objective.
fig. 2 — outputs are illustrative, hand-written for this page to show each stage's characteristic behavior; they are not sampled from a real model.
§ 3 · The loop
The loop is the same. The data isn't.
Strip away the names and every training stage is the identical next-token loop — what changes is what the batches contain, how many tokens flow through, and how hard the optimizer pushes. Data is the steering wheel.
| stage | tokens seen | epochs | learning rate | what a "step" is |
|---|---|---|---|---|
| pretrain | ≈11B | ~1 | warmup → cosine decay, the big LR | one batch of shuffled web text |
| mid-train | ≈0.5–1B | ~1 | short schedule, lower peak | same loop, curated batches |
| SFT | ≈10–100M | 1–2 | small and brief | same loop, loss masked to assistant tokens |
| eval | 0 trained on | — | none — no gradient | forward pass, write the number down |
This split is exactly how the roadmap divides the next two phases: Phase 3 lives inside pretraining's constraints (data quality, scaling, GPU systems), and Phase 4 lives after it (post-training and evaluation). The pipeline isn't just how models are built — it's the curriculum's table of contents.
§ 4 · The takeaway
The shape to remember
one loop, different data
Pretrain, mid-train and SFT run the same next-token loop. The stages differ in what the batches contain — data is the steering wheel.
SFT changes behavior, not knowledge
The facts live in pretraining. SFT teaches the move "a question is followed by an answer" — small, cheap, and transformative.
eval is a stage, not an afterthought
It has its own data and its own discipline, and it is the only stage whose output you can defend. Phase 4 lives here.
The Phase 2 milestone is running this shape once, yourself — tokenizer to chat UI — so that none of it is folklore. nanochat is the speedrun: a weak model, a complete pipeline, and every stage of this page made concrete.