Phase 2 · ≈ 4–8 weeks

Transformers & LLMs

Build and train a GPT-style transformer from a blank file, understand every component (tokenizer → embeddings → attention → MLP → logits), and run the full small-LLM pipeline end to end at least once.

What to learn · in priority order

1
Self-attention interactive lab →
Queries/keys/values, multi-head attention, causal masking. Implement it from scratch; this is the centerpiece.
2
The transformer block interactive lab →
Attention + MLP, residual connections, LayerNorm/RMSNorm, why each is there.
3
Tokenization interactive lab →
BPE, why subword, how the tokenizer shapes everything downstream — an underrated source of bugs and quality.
4
Positional information interactive lab →
Learned vs RoPE; why attention needs it at all.
5
The training loop at language scale interactive lab →
Data loading, batching, LR schedule, evaluation by loss/perplexity.
6
The full pipeline interactive lab →
Pretrain → (optional) mid-train → SFT → eval, so you've seen the whole shape once before Phases 3–4 deepen each part.

Primary path · use these, not ten others

Karpathy — "Let's build GPT" + nanoGPT ↗

Build a GPT from a blank file, then study nanoGPT as the clean reference — deprecated since Nov 2025 in favor of nanochat (below), but still the shortest complete GPT you can hold in your head. Train on tiny-shakespeare, then a slightly bigger corpus.

Raschka — Build a Large Language Model (From Scratch) ↗

The most thorough code-first walk through every component, including loading real pretrained weights (GPT-2 → Llama).

Karpathy — nanochat ↗

nanoGPT's official successor. Reproduce the entire modern stack once: tokenizer → pretrain → mid-train → SFT → eval → chat UI. The model is weak; running the whole pipeline is the point.

Jay Alammar — The Illustrated Transformer ↗

The clearest visual explainer of attention. Read alongside the code.

Papers: Attention Is All You Need · GPT-1 · BERT ↗

Read them, don't just cite them. The Transformer paper is the one to actually read this phase.

The deliverable

Train a small GPT from a blank file and write up one thing you investigated — e.g. how final loss changes with depth at fixed params. A question → an experiment → a finding. Your first taste of research, not just implementation.

Milestones · check only when you can do it from a blank file

0/6 shipped

artifact: A from-scratch GPT repo + a short investigation writeup. → Level 1 reached.

rule: every phase must end with a public artifact (repo + writeup). no artifact → phase not done, regardless of what you've watched.

Traps

✗Treating attention as a formula to memorize

Implement it, visualize the attention weights, perturb it. Understand it as a mechanism, not an equation.

✗Skipping the tokenizer

It's boring and it's where subtle quality bugs live. Build a small BPE once.

✗Stopping at "it trains"

The deliverable is a question answered, not a running loop. That shift — building → investigating — is what makes this Phase 2 and not Phase 0.