Phase 4 · ≈ 4–8 weeks

Post-training & eval

Understand how a raw pretrained model becomes useful — SFT, preference/RL methods (RLHF, DPO, GRPO), reward modeling, and evaluation as a rigorous discipline. Post-train a small model yourself and measure it honestly.

What to learn · in priority order

1
SFT / instruction tuning interactive lab →
Turning a base model into one that follows instructions; data formats; LoRA/QLoRA for doing it cheaply on one GPU.
2
Preference & RL methods interactive lab →
RLHF (the canonical recipe), DPO (simpler, no separate reward model), GRPO/RLVR (RL from verifiable rewards — the current frontier for reasoning).
3
Reward modeling interactive lab →
What a reward model is, reward hacking, why a verifier with skin in the game can't be a fair verifier.
4
Evaluation as a discipline interactive lab →
How benchmarks lie: contamination, prompt sensitivity, the metric–behavior gap, why pairwise beats absolute scoring. The most important sub-topic in the phase.
5
Inference-time methods interactive lab →
Sampling, speculative decoding, why decoding choices change measured quality.

Primary path · use these, not ten others

Nathan Lambert — The RLHF Book ↗

The authoritative, current guide to post-training: RLHF, DPO, the RLVR renaissance, reward modeling, evaluation. Your spine for this phase. Free at rlhfbook.com.

Hugging Face TRL ↗

The practical toolkit for SFT, DPO, and GRPO. Use it to actually post-train.

DeepSeek-R1 ↗

The RL-for-reasoning (RLVR/GRPO) result that defined the current frontier. Your "small reasoning model" paper for the milestone.

The deliverable

Post-train a small model (0.5–1.5B) and evaluate it honestly: SFT (LoRA) on a focused dataset, then DPO or a small GRPO loop with a verifiable reward. The research content is in the evaluation — clean harness, same-size baseline, how the reward could be gamed, what the metric does NOT capture.

Milestones · check only when you can do it from a blank file

0/6 shipped

artifact: A post-trained model + honest-eval repo + writeup. → Level 2 reached.

rule: every phase must end with a public artifact (repo + writeup). no artifact → phase not done, regardless of what you've watched.

Traps

✗Trusting your own numbers

The default failure mode. Assume your eval is lying until you've checked for contamination, prompt sensitivity, and a missing baseline.

✗Reaching for GRPO first

It's the exciting frontier and a debugging swamp. Earn it with SFT+DPO first.

✗Self-evaluation

Don't let the model (or the same prompt) grade its own work — self-preference bias is real. Separate the worker from the judge.

✗Metric tunnel-vision

A number going up is not a behavior improving. Always ask what the metric fails to capture.