Interactive explainer · Phase 1 — Foundations
Backpropagation
A neural net is billions of dials, and training needs to know which direction to turn each one. Backpropagation is the bookkeeping that figures out, for every dial, which way is downhill — nothing more. It's the chain rule from calculus, applied backward through a graph of simple operations. Below is the smallest graph worth studying: turn the dials, then walk the gradient home.
— one multiply, one add, one squish, one scale. a one-neuron network in miniature.
§ 1 · The lab
Forward to compute, backward to blame
The forward pass (ink numbers) just evaluates the expression left to right. The backward pass (vermillion numbers) then asks, node by node, right to left: if this value wiggled, how much would L move? Each node answers with one local rule times whatever the node downstream already computed — that product is the chain rule, and it's the entire algorithm.
gradients hidden — the backward pass starts at the output, where the answer is trivially 1. press start, then watch the blame flow right to left.
fig. 2 — dashed boxes are leaves (the dials an optimizer would turn). drag a slider mid-walk: every revealed gradient recomputes live.
§ 2 · What the number means
A gradient is a slope, and a promise
Freeze every dial except a and plot L against it. The vermillion tangent's slope is exactly the the backward pass produced — backprop never saw this curve, yet it knows the steepness at your point. The promise is local: nudge a by ε and the tangent predicts ΔL ≈ slope × ε. Grow ε and watch the prediction (hollow ring) drift off the curve (moss dot). That gap is why training takes many small steps instead of one big one.
- tangent predicts ΔL = 0.95 × ε
- 0.76
- the curve actually moves
- 0.52
- the local promise breaks by
- 0.24
gradient descent is just: take the slope, step a small ε against it, recompute, repeat. the learning rate IS this ε.
§ 3 · The whole algorithm
Three local rules, multiplied backward
× swaps the inputs
Wiggle one factor, L moves by the other.
+ passes it through
Addition distributes the gradient unchanged.
tanh scales by its slope
Flat curve, dead gradient — saturation in one line.
Every node answers one tiny local question and multiplies it by the gradient arriving from downstream — no node ever sees the whole graph. A billion-parameter transformer trains with exactly this loop, just with matrices in the boxes. If this page made sense, you're ready for the real Phase 1 deliverable: close this tab and rebuild it from a blank file — that's micrograd.