Interactive explainer · Phase 1 — Foundations
A neural net from scratch
Backprop computes the gradients, gradient descent takes the steps — now assemble the dials into layers and watch a function take shape. The network below is real and trains in your browser: two inputs, a stack of number-mixers, one verdict. Its entire job is to tint this square correctly.
— mix, shift, bend. repeat per layer. that's the whole machine.
§ 1 · The lab
Carve the boundary
Each dot is a training example; the tint is the network's current opinion about every point in the square. Training is just backprop + gradient descent, the two labs you've already run, looping over these dots. Pick a dataset and an architecture, press train, and watch the boundary get carved — then break it on purpose with the experiments below.
data
architecture
activation
init
epoch 0 · loss — · accuracy 51% · 33 parameters · untrained — the tint is initialization luck
fig. 1 — every knob change starts a fresh network (that's the honest way to compare). re-roll to see how much initialization luck matters.
§ 2 · Run these
Four experiments, four lessons
the line that couldn't
No hidden layer on the rings. The loss flatlines just under the coin-flip line forever — no learning rate, no patience, no luck will fix it. A linear model draws one straight line, and no straight line separates inside from outside.
bend it
Same data, eight hidden neurons. Each neuron contributes one soft fold; together they close a loop around the inner class in a few hundred epochs. This is everything an activation function buys you.
dead symmetry
Same architecture, every weight initialized to exactly zero. All eight neurons compute the same thing, receive the same gradient, and stay identical forever — the loss never leaves ln 2. Random init isn't a nicety; it's what makes neurons differentiable from each other.
the deep carve
Two interleaved spirals, two hidden layers. Watch the boundary stay confused for the first thousand epochs, then snap into the spiral as layer two starts composing the folds layer one found. Features of features — give it ~30 seconds.
§ 3 · Why it works
Mix, bend, stack
activations bend
Stack linear layers without a nonlinearity and they collapse into one linear layer — all that depth, still a straight line. The bend between layers is load-bearing.
depth composes
Layer one finds simple folds; layer two combines folds into shapes. Features of features — the spiral falls to composition, not to more neurons in one row.
init breaks the tie
Identical neurons get identical gradients and never differentiate — so start random. Scale by fan-in so signals neither explode nor vanish as they cross layers.
Everything on this page is ~120 lines of plain code — the forward loop, the backward loop, the update. No framework, no magic. That's the Phase 1 bet: build exactly this from a blank file (micrograd gives you the gradients, then this MLP on top), and deep learning stops being an API and starts being a mechanism you own.