Interactive explainer · Phase 4 — Post-training & eval
SFT & LoRA
A base model is a text-completer with no concept of being asked. Supervised fine-tuning is just more pretraining — on transcripts of the behavior you want. No new algorithm, no reward signal: the same next-token loss, pointed at conversations. The craft lives in two places: what you put in the dataset, and how you afford the gradient on one GPU.
— LoRA's whole bet: the update ΔW is low-rank, so train B and A instead.
§ 1 · The data IS the method
Train only the answer
Every SFT example is rendered into a chat template before the model sees it. Two decisions hide in that rendering: the format itself, and the loss mask — which tokens carry gradient. Flip the mask below; then try the other training pairs, because the second lesson is harsher: whatever is in the response, masked or not, becomes the model's personality.
training pair
Clean pair — this is the behavior you are buying, one gradient step at a time.
fig. 1 — chunks are word-level for legibility; a real tokenizer cuts finer (see the tokenizer lab).
§ 2 · LoRA
The update is low-rank
Fine-tuning changes W by some ΔW. LoRA's empirical bet: ΔW has low intrinsic rank — a handful of directions do almost all the work. Below, a 16×16 update built from six directions of decaying strength (plus noise): drag the rank and watch how little of the matrix you need to train to capture almost all of it.
rank 2 explains 88% of the update
params: 64 vs 256 full · 25% of a full fine-tune
ΔW · full update
rank-2 approx
fig. 2 — constructed from orthonormal directions, so the rank-r truncation is exactly optimal (Eckart–Young) and the energy curve is exact. Real updates aren't built this cleanly — the LoRA paper's evidence is that they behave as if they were. vermillion = positive, moss = negative.
§ 3 · What it costs
Why LoRA exists: the optimizer bill
Training memory is dominated by optimizer states — 16 bytes per trained param under AdamW mixed precision. Freeze the base and train only adapters, and that bill collapses; quantize the frozen base to 4-bit (QLoRA) and the weights nearly vanish too.
| method | trained params | 1B model | 3B model | what's in memory |
|---|---|---|---|---|
| full fine-tune | every param | 14.9 GB △ tight | 44.7 GB ✗ OOM | optimizer states for every weight |
| LoRA r=16 | 4.2M adapters | 1.93 GB ✓ fits | 5.74 GB ✓ fits | frozen bf16 base + tiny trained states |
| QLoRA r=16 | 4.2M adapters | 0.57 GB ✓ fits | 1.69 GB ✓ fits | 4-bit frozen base (NF4 + scales) |
Verdicts are against a 24 GB RTX 4090. Add activations on top (batch- and recomputation-dependent — see the memory anatomy): full fine-tuning a 1B model is borderline on paper and dies in practice once a real batch lands; 3B is hopeless either way. LoRA moves the wall by an order of magnitude, and QLoRA's 4-bit frozen base is the quantization lab applied to fine-tuning.
§ 4 · Honest limits
What SFT can and can't do
It changes behavior, cheaply
Format, persona, instruction-following, refusal style — thousands of good pairs move all of them. This is the highest-leverage cheap intervention in the pipeline.
It doesn't add knowledge reliably
Facts the base model never learned don't appear because you showed it 3k transcripts. Knowledge is pretraining's job; SFT mostly reshapes what's already there.
It overcooks easily
Too many epochs on a narrow set and the model collapses into one voice and forgets base abilities — practitioner lore you will reproduce within your first week of trying.
For the base-vs-SFT behavior shift on one prompt, see the pipeline lab's §2 — same model lineage, completely different creature after a few thousand transcripts.
§ 5 · The whole trick
Three rules, one bet
SFT is pretraining on the behavior you want
Same loss, same loop — the dataset is the entire specification of the assistant.
Mask the prompt, train the answer
Gradient on the response (and its end marker). Context is for reading, not imitating.
LoRA bets the update is low-rank
Train 2dr params instead of d². The bet usually pays; the optimizer bill collapses.
The Phase 4 deliverable starts exactly here: LoRA-SFT a 0.5–1.5B model on a focused dataset with TRL, then evaluate it honestly — which is the hard half, and the next lab.