Jargon decoder

The machine itself

Model

A program that learned from examples instead of being given rules. The ones here are giant guess-the-next-word machines.

Neural network

How the machine is built inside: layers of simple number-mixers stacked on each other, loosely inspired by brain cells.

Parameters / weights

The millions or billions of tiny dials inside the machine. "Training a model" means turning these dials.

Token / tokenizer

Models don't read letters. The tokenizer chops text into word-pieces called tokens — those are what the model actually sees.

Embedding

Turning each token into a long list of numbers, so you can do math on words.

Attention

The trick that lets the model look back at all the earlier words and decide which ones matter most for guessing the next one.

Transformer

The blueprint built around attention. Nearly every modern AI model uses it. It's the T in GPT.

LLM / GPT

Large Language Model: a big transformer trained on a huge slice of the internet to predict the next token.

GPU

The chip that does the math. Designed for video games; turned out to be perfect for neural networks.

Teaching it

Training

Show the machine examples; each time it guesses wrong, nudge every dial a tiny bit toward "less wrong." Repeat billions of times.

Loss

One number that says "how wrong is it right now." A falling loss curve is what learning looks like on a chart.

Gradient descent

The nudging strategy: always take a small step downhill on the loss.

Backpropagation

The bookkeeping that figures out, for each of the billions of dials, which direction is downhill. The single most important algorithm in this roadmap.

Learning rate

How big each nudge is. Too big and training goes haywire; too small and it takes forever.

Overfitting

Memorizing the homework instead of learning the subject. The model aces the practice questions and fails the real exam.

Pretraining

The first, giant phase of training: pure next-word guessing over an enormous pile of text.

Scaling laws / Chinchilla

The math of how much text a model of a given size should be fed — roughly 20 tokens for every parameter.

MFU

What fraction of your GPU's theoretical horsepower you're actually using. Low MFU = paying for a sports car, driving it in first gear.

Shaping it after school (post-training)

Fine-tuning / SFT

A short second school after pretraining: show the model worked examples of being a helpful assistant, so it stops just rambling text.

RLHF

Teaching with thumbs-up / thumbs-down instead of right answers: humans rate responses, the model learns to get more thumbs-up.

Reward model / reward hacking

A judge program that scores the model's answers. Reward hacking is the model finding sneaky ways to score points without actually being good — like a student gaming a rubric.

DPO / GRPO / RLVR

Newer, simpler recipes for the same thumbs-up idea. RLVR is the special case where the reward can be checked mechanically (did the math answer come out right?), which is how reasoning models are trained.

LoRA

A fine-tuning shortcut: instead of adjusting all the dials, bolt on a small set of new ones and adjust only those. Cheap enough for one GPU.

Quantization / distillation / pruning

Three ways to shrink a model: store the dials with fewer decimal places · have a small student model copy a big teacher · cut the parts that turn out not to matter.

Doing science on it

Eval / benchmark

The model's exam. Evals "lie" when the model saw the answers during training, or when the exam doesn't measure what you actually care about.

Baseline

The thing you compare against. "My model scores 80" means nothing; "80 versus 72 for the same-size standard model" means something.

Ablation

Remove or change exactly one ingredient, rerun, and see what difference it made. The basic unit of ML science.

Seed

The starting roll of the dice for a training run. Run with at least 3 different seeds, or your "result" might just be luck.

Confound

A second difference that sneaks into your comparison and ruins it — you changed the optimizer and accidentally the model size, so which one helped?

Reproduce

Redo someone's published experiment from scratch and get the same numbers. Where most of the real learning lives.

Mechanistic interpretability

Opening up the machine to figure out what the dials are actually doing inside — circuit-tracing for neural networks.

Artifact

The proof you actually did the work: a public repo, a trained model, a writeup that a stranger could check and build on.