Master list · curated, not exhaustive
Resources
A short list you'll actually use beats a long list you won't. For any given phase, pick one primary resource and go deep; the others are references for when the primary leaves a gap. Collecting resources is procrastination; finishing one is progress.
Hands-on courses & code — the spine, do these
Karpathy — Neural Networks: Zero to Hero ↗1–2The canonical from-scratch course: micrograd, makemore, build-GPT. Do the exercises.
3Blue1Brown — Neural Networks series ↗1The best visual intuition for nets, gradients, and backprop. Watch when a concept won't click.
nanoGPT ↗2Clean minimal GPT to study + train. Deprecated & frozen since Nov 2025 — still the shortest complete read; nanochat is the successor.
nanochat ↗2nanoGPT's official successor: the full modern stack end-to-end (tokenizer→pretrain→SFT→eval→UI).
Raschka — Build an LLM (From Scratch) ↗2Thorough code-first book + repo (incl. GPT-2→Llama, Qwen-from-scratch).
Stanford CS336 — Language Modeling from Scratch ↗3–4The flagship systems course: tokenizer, FlashAttention2-in-Triton, distributed training, data, SFT+RL. Do the assignments.
ARENA (Callum McDougall) ↗5Alignment/research-engineering curriculum: DL fundamentals → transformers + mech interp → RL.
Books & long-form
Mathematics for Machine Learning — Deisenroth et al. ↗1Math reference (look up, don't read linearly). Free PDF.
Deep Learning — Goodfellow, Bengio, Courville ↗1–3The classic theory reference; dip in, don't read cover-to-cover.
Andrew Ng — Machine Learning Specialization1Gentle, rigorous ML fundamentals if you want more hand-holding. (Coursera)
Canonical papers — read, don't just cite
GPT-1 — Improving Language Understanding by Generative Pre-Training ↗2The generative-pretraining lineage (OpenAI report, not on arXiv).
Chinchilla — Training Compute-Optimal LLMs3Scaling laws; "small but well-trained."
FlashAttention3Memory-efficient attention; the systems mindset.
GPTQ / AWQ3Post-training quantization.
Distilling the Knowledge in a Neural Network (Hinton) + Sequence-Level KD3Distillation foundations.
TinyStories → Phi-1.5 → Phi-3 → SmolLM3The data-quality thread; small-model recipes.
InstructGPT / RLHF + DPO + a GRPO/RLVR paper4The post-training lineage.
Anthropic — Transformer Circuits / induction heads5 (interp)Mech-interp foundations.
People & newsletters — signal, low noise
Raschka + Han Lab + the Hugging Face science blog cover ~80% of what matters week-to-week.
Nathan Lambert — Interconnects4Post-training / RLHF pragmatics.
MIT Han Lab3/5The academic home of efficiency: AWQ/SmoothQuant/TinyChat.
Communities, compute, venues
ComputeallFree Colab/Kaggle to start; Vast/RunPod community tiers for cheap rentals; granted compute via TPU Research Cloud / EleutherAI.
Venues for independents5ICLR Blog Posts track; NeurIPS ENLSP workshop; ML Reproducibility Challenge. Not the main-track lottery.
How to choose · so you don't drown
- — Phase 1–2: Karpathy Zero-to-Hero is your primary. Raschka's book is the deeper companion. fast.ai/d2l only to patch gaps.
- — Phase 3: CS336 is your primary — do the assignments. modded-nanogpt + the canonical papers around it.
- — Phase 4: The RLHF Book + TRL.
- — Phase 5: the on-ramp + (if interpretability) ARENA.
- — RL branch: HF Deep RL course → Sutton & Barto + David Silver → Spinning Up + CleanRL + Gymnasium → RLHF Book / GRPO / RLVR.
- — One primary per phase. Everything else is a reference, not a queue.