Roadmap.

Phase 3 · ≈ 8–12 weeks→ Level 2 (Extender) begins

Training & systems

Understand how real models are trained efficiently — scaling laws, data, optimization at scale, GPU systems — and the efficiency toolkit (quantization, distillation, sparsity) that defines small-models work. Then reproduce a published result and run one clean ablation.

What to learn · in priority order

  1. 1

    Scaling lawsinteractive lab →

    Chinchilla (compute-optimal ~20 tokens/param), what they predict and where they break. The conceptual basis for "small but well-trained."

  2. 2

    Efficiency / GPU systemsinteractive lab →

    Mixed precision (bf16), MFU, memory vs compute, FlashAttention, a reading-level grasp of kernels (Triton) and parallelism. Know where the time and memory go.

  3. 3

    Datainteractive lab →

    The biggest lever in practice. Curation, filtering, deduplication; why data quality often beats architecture (TinyStories → Phi → SmolLM).

  4. 4

    Optimization at scaleinteractive lab →

    LR schedules, warmup, the modern optimizer landscape (AdamW, Muon), gradient accumulation, batch-size effects.

  5. 5

    The small-models efficiency toolkitinteractive lab →

    Quantization (GPTQ/AWQ/GGUF), knowledge distillation, pruning/sparsity, efficient architectures. Read enough to run experiments in each.

Primary path · use these, not ten others

Stanford CS336 — Language Modeling from Scratch ↗

The flagship for this phase. Tokenizer, Triton FlashAttention2, distributed training, Common Crawl → pretraining data, SFT + RL. Do the assignments — that's where the systems knowledge sticks.

modded-nanogpt ↗

The speedrun repo. Read the commit history like a textbook — a masterclass in concrete, citable efficiency wins (Muon, QK-norm, value embeddings).

Papers: Chinchilla · FlashAttention · GPTQ/AWQ · Hinton distillation

Read, don't just cite. See the resources page for the full list.

The deliverable

Reproduce a result on one GPU, then run ONE clean ablation: a single-GPU modded-nanogpt run isolating one trick (Muon vs AdamW), or a quantization quality ablation at matched bits-per-weight, or a distillation logit-sparsity study. Ship repo + reproducible eval harness + honest same-size baselines + a writeup with a plot. This artifact is your L2 credential.

Milestones · check only when you can do it from a blank file

0/7 shipped

artifact: Reproduction + ablation repo with eval harness, baselines, plot, writeup. Your L2 credential.

rule: every phase must end with a public artifact (repo + writeup). no artifact → phase not done, regardless of what you've watched.

Traps

✗Lectures without assignments

CS336's value is the implementation work. Watching lectures and skipping psets gives the illusion of systems knowledge.

✗Chasing the speedrun record

The 8×H100 leaderboard is a saturated multi-year competition. Reproduce on one GPU and isolate one trick.

✗Confounded ablations

The #1 way to produce a worthless result: vary two things at once, mismatch the baseline, or run one seed. Read the pitfalls list before designing.

✗Skipping data

Data quality is the highest-leverage variable in the field and the least glamorous. Don't under-weight it.