Interactive explainer · Phase 1 — Foundations
Core ML
The previous lab asked can it learn? This one asks the question that makes you a scientist instead of a spectator: how do you know what it learned is real? The answer is a held-out exam, a loss you understand, and a healthy fear of your own homework scores. This is the grammar of every experiment you'll ever run.
§ 1 · The loss
The price of confidence
Cross-entropy asks one question per example: how much probability did you give the true answer? Full marks costs nothing; hedging costs a little; confident wrongness costs without bound. That asymmetry is the whole personality of the loss — and why a falling loss curve means "less wrong," not "more right."
loss = −log(0.30) = 1.20 — hedging near the coin-flip line (ln 2 ≈ 0.69) — where untrained nets start.
§ 2 · The lab
Homework vs exam
Same rings, harder rules: the net trains on the filled dots only — and 10% of their labels are deliberately wrong, because real labels always are. The hollow rings are the exam: held out, clean, never trained on. Watch both losses. The homework score always improves; whether the exam follows is the entire question of machine learning.
capacity
weight decay · L2
epoch 0 · homework — · exam — · untrained — no grades yet
fig. 2 — filled = train (with 10% wrong labels) · hollow = held-out val. the moss dot on the chart is where early stopping would have saved you.
§ 3 · Run these
Bias, variance, and the two rescues
memorize the noise
An oversized net, no regularization. Train loss marches to zero — it learns every flipped label by heart — while val loss bottoms out early and then climbs forever. The two curves diverging is the most important picture in this phase.
the goldilocks net
Right-sized capacity, light decay. The exam tracks the homework all the way down and stays there. This is what "it generalizes" looks like — and note it never reaches zero train loss. It shouldn't: 10% of the homework answers are wrong.
too simple to overfit
Two hidden neurons can't draw a circle, so both losses park high and flat. That's bias: the model family can't express the truth, and no amount of training fixes it. Underfitting is failure too — just a quieter one.
the rescue
The same oversized net that memorized everything — now with heavy weight decay pulling every weight toward zero. It can no longer afford to wrap noise points, so it spends its budget on the real shape. Regularization buys generalization with capacity.
§ 4 · The grammar
Three sentences you now own
cross-entropy prices confidence
Being wrong is cheap; being confidently wrong is catastrophic. The loss is the exam's grading rubric — know what it punishes.
the split is the experiment
Homework grades flatter; only the held-out exam measures learning. Tune on val, and keep a test set you touch once. Evals lie when sets leak.
overfitting has three fixes
More data, a smaller/regularized model, or stop at the moss dot. (The milestone asks you for three — these are them.)
The Phase 1 milestone says it directly: explain cross-entropy, the train/val split, and overfitting with three fixes — in your own words, in your log. You've now watched all three happen to a real network. Write the log entry while the curves are fresh; that entry is the artifact.