Interactive explainer · Phase 1 — Foundations
Gradient descent & optimizers
Backprop tells every dial which way is downhill. This page is about the other half of training: how big a step to take, and what to do when "just follow the slope" isn't enough. The learning rate is the single most consequential number you'll set — here you can feel why, then watch the two classic fixes race.
§ 1 · The step size
One dial, one bowl, one fateful number
The whole algorithm: , repeated. On this bowl the regimes are mathematically exact — each step multiplies the distance to the bottom by . So drag η and find all four: a crawl, a glide, an overshooting convergence, and an explosion. At the bowl is solved in one step — real losses are never this kind.
step 0 · w = 2.50 · verdict: gliding down
fig. 1 — change η mid-run: you can rescue a diverging ball, or wreck a gliding one. the lower strip is what you'd see in a real training log.
§ 2 · The optimizers
The ravine — SGD on a knife's edge
Real losses aren't round bowls; they're ravines — many times steeper in some directions than others. One η must serve both: the steep wall sets SGD's stability limit (here η < 0.05) while the shallow floor sets its speed, so it must run close to its own explosion to compete. Momentum smooths the zigzag like a heavy ball and tolerates twice the heat; Adam gives every dial its own step size and barely notices the conditioning. Race them at the default — then nudge η one notch past 0.05 and race again.
- SGD— loss 34.63
- momentum— loss 34.63
- Adam— loss 34.63
fig. 2 — drag η one notch past 0.05 and race again: SGD explodes in ~16 steps while the other two survive. drag it tiny (0.01) and SGD stalls on the floor while the heavy ball still arrives. momentum's loops are real — it overshoots the turns, and gets there anyway.
§ 3 · The whole toolbox
One number, two fixes
learning rate η
The step size. Too small wastes compute, too big destroys the run, and the safe range depends on the curvature — which you never know in advance. Hence: watch the loss curve.
momentum — a heavy ball
Average your recent gradients and step along that. Zigzags across the ravine cancel out; the steady pull along it compounds. One extra number (β ≈ 0.9), dramatic effect.
Adam — a learning rate per dial
Track each dial's typical gradient size and divide by it, so steep dials get small steps and shallow dials get big ones. The default optimizer for most of deep learning.
The Phase 1 milestone asks you to diagnose a too-high learning rate from the loss curve — you've now watched that exact curve spike live. Next time it happens in a real run, you'll recognize the shape. Then go implement the loop: it's four lines, and you've already built the gradient machine it feeds on.