A sibling branch · same trunk

The RL branch

ML has three classical paradigms — supervised, self-supervised (LLM pretraining), and reinforcement learning. RL is one branch, not the root; the root is the trunk you already know (backprop, optimization, neural nets). The twist that matters: the RL and LLM branches merge at the frontier — modern LLM post-training (RLHF, GRPO/RLVR) is RL applied to language models.

Classic deep RL

Games, robotics, control. Harder and less forgiving solo: sample-inefficient, compute-hungry, brittle, with a real reproducibility problem. Pick it for love of the problem, not tractability.

RL-for-LLMs / RLVR ← steer here

Reasoning models. The hottest area, most jobs, and the most tractable for a solo person on one GPU — it reuses your entire LLM-track skillset.

Even RL veterans call Sutton & Barto a slog. Do not read it cover-to-cover before touching code. Ship a working agent early, hit a wall, then pull the theory in. RL punishes the "learn everything first" instinct harder than any other branch.

The phases

Same template, different branch

RL-0Orientation

≈ days

Goal: Train one working RL agent end-to-end before you understand it, and pick your lane.

— Do Hugging Face Deep RL Course, Unit 1 — train an agent on a Gymnasium env with Stable-Baselines3 in an afternoon. Watch a reward curve go up.
— Decide your lane (classic deep RL vs RL-for-LLMs) — write it in your LOG.md. For tractability solo, default to the RLVR lane.

artifact: A trained agent + a reward curve in your repo.

✗ trap: Starting with Sutton & Barto chapter 1. Don't. Train something first.

RL-1RL foundations

≈ 4–6 wk

Goal: Understand and implement the core RL machinery from a blank file. The "micrograd of RL."

— Learn: MDPs, return + discounting, value functions (V, Q), the Bellman equations, dynamic programming, exploration vs exploitation, and tabular methods — Q-learning, SARSA, TD learning.
— Primary path: Sutton & Barto Part I (tabular) — read alongside coding, not before. Supplement with David Silver's RL lectures.

artifact: Tabular Q-learning and SARSA from a blank file on a gridworld / FrozenLake / Taxi — no library doing the learning for you. Plot the value function; show the policy converging.

✗ trap: Jumping to deep RL before tabular intuition. The deep stuff is function approximation bolted onto these ideas — skip them and nothing later makes sense.

RL-2Deep RL

≈ 8–12 wk

Goal: The modern algorithms, and a clean reproduction + ablation — your L2 credential on the RL side.

— Learn: function approximation, DQN (replay buffers, target nets), policy gradients (REINFORCE), actor-critic (A2C), and PPO — the workhorse you must know cold. GAE, advantage normalization, the bias-variance levers.
— Primary path: OpenAI Spinning Up + CleanRL (single-file, research-grade implementations — the "nanoGPT of RL") + Gymnasium + Stable-Baselines3 for trusted baselines.

artifact: Reproduce PPO on classic control with CleanRL (CartPole → LunarLander), then ONE clean ablation (e.g. GAE on/off, clip range) with ≥3 seeds and reported variance.

✗ trap: Reproducibility hell — RL variance is brutal; one seed is a lie. Seed everything, report spread. And env steps dominate: budget wall-clock time, not just GPU-dollars.

RL-3The LLM intersection — RLHF / GRPO / RLVR

≈ 4–8 wk

Goal: The cheap-compute, high-energy lane — where this branch rejoins the LLM track. This IS Phase 4.

— Learn: reward modeling (+ reward hacking), PPO-for-LLMs (RLHF), DPO, GRPO (critic-free, group-normalized advantages), and RLVR (verifiable rewards: math/code where correctness is auto-checkable).
— Primary path: The RLHF Book + HF TRL + CS336's RL assignment + the open frameworks (Open-Reasoner-Zero, DAPO).

artifact: A small GRPO/RLVR run on a 0.5–1.5B model with a programmatic/verifiable reward + an honest eval with a same-size baseline.

✗ trap: GRPO is finicky (KL control, reward hacking, wall-clock ≫ GPU-hours) — SFT+DPO is the safer first step. And don't overclaim RLVR gains: some 2026 work argues it makes models faster at what they already can do, not fundamentally smarter.

RL-4Specialization & research

ongoing

Goal: Pick a lane and ship an original result. Same research method, same L3 bar.

— Lanes (depth over breadth): RL-for-reasoning/RLVR (most tractable solo) · small-scale classic deep RL · model-based / offline / exploration / multi-agent RL.
— Reproduction-as-contribution: given RL's reproducibility crisis, a careful, seeded reproduction of a published RL result is a genuinely valued contribution (ICLR Blog Posts, MLRC).

artifact: A public, reproducible original result others can use.

✗ trap: Choosing classic deep RL because it looks like a shortcut. Choose it because you can't stop thinking about the control problem.

Resources

The RL shelf

Hugging Face Deep RL CourseRL-0Hands-on on-ramp (SB3 + Gymnasium). Start here.

Sutton & Barto — Reinforcement Learning: An IntroductionRL-1The bible. Part I (tabular) first; read alongside code. Free PDF.

David Silver — RL CourseRL-1The canonical lecture series (DeepMind/UCL).

OpenAI Spinning UpRL-2Best practical deep-RL intro.

CleanRLRL-2Single-file deep-RL implementations (PPO/DQN/SAC). Reproduce these.

Gymnasium (Farama)RL-0–2Standard env interface, successor to OpenAI Gym.

The RLHF Book + HF TRLRL-3The RL/LLM intersection (RLHF/DPO/GRPO/RLVR).

RL Field Manual (rl.paraz.in)RL-3Interactive guide to LLM RL; #frontier covers RLVR.

DeepSeek-R1RL-3The RLVR/GRPO reasoning result that defined the frontier.

The honest bottom line: for tractable, impactful research as a solo person on one GPU, take RL-3 (RLVR/reasoning) — it's the frontier, it's cheap-ish compute, and it reuses everything in your LLM track. Either way: same trunk, same method, same ship-the-artifact discipline.