Interactive explainer · Phase 3 — Training & systems
The small-models efficiency toolkit
Three ways to shrink a model: store it coarser (quantization), teach a smaller one (distillation), delete parts of it (pruning). None of them is magic — each is a measurable trade you can reason about, and all three are one-GPU research territory. Below, each runs for real on numbers small enough to read.
— absmax quantization: one scale, everything snaps to a grid.
§ 1 · Quantization
Precision is a budget, outliers set the price
A weight is just a number stored at some precision. Drop from 16 bits to 4 and the model is 4× smaller — if the values survive the coarser grid. Turn the dials: then inject the outlier and watch per-tensor scaling fall apart. That single failure mode explains most of modern quantization research.
scale granularity
weight distribution + storable values
reconstruction error vs bits
original weights
stored at 4 bits
- rmse
- 0.0577
- bits/weight (incl. scales)
- 4.06
- 1B model
- 0.51 GB
The whole trick: scale = absmax ÷ levels. Every weight snaps to the nearest of 7 steps per sign. More bits, finer steps, lower error — smoothly.
GPTQ
Rounds weights one column at a time, compensating each rounding error with the weights not yet quantized. Smarter rounding, same grid.
AWQ
Protects the ~1% of channels the activations say matter most, scaling them out of harm before quantizing. Outlier-aware by design.
GGUF
Not an algorithm — the packaging: k-quant block formats plus metadata, so a quantized model is one portable file (llama.cpp).
fig. 1 — 256 seeded Gaussian weights; the quantizer, the errors, and the outlier are all computed live. Real methods differ only in how cleverly they round onto this same grid.
§ 2 · Distillation
Soft targets carry more bits than labels
A student trained on labels learns the answer. A student trained on the teacher's distribution learns the answer, the runners-up, and the absurdities — per token. Slide the temperature to see what the hard label throws away. This page shows what transfers; actually training a student on it is a Phase 3 project (how few teacher logits suffice? — the logit-sparsity study).
context: "the bird ate the ___"
hard label — what SFT data carries
entropy: 0.00 bits — "worm, and that's all you learn."
teacher distribution at T = 1.0
entropy: 1.51 bits — the ranking and the relative wrongness come along.
This is the working range: "seed" is plausible, "sofa" is absurd — that structure is what the hard label throws away, and it's the signal Hinton called dark knowledge. The student learns the teacher's whole similarity geometry, not one answer.
§ 3 · Pruning
Free memory, paid-for speed
Most weights in a trained net are small, and small weights mostly don't matter — so zero them. The same seeded tensor, magnitude-pruned: watch the error stay flat, then cliff. Then switch to the 2:4 pattern to see the shape hardware actually rewards.
pattern
surviving weights (128/256) · rmse 0.1577
reconstruction error vs sparsity
Free-form pruning keeps the globally largest weights — gentle at first, a cliff past ~80%. The catch: scattered zeros save memory, not time. A dense matmul kernel multiplies the zeros anyway.
§ 4 · The ledger
What a 1B model costs at each setting
| format | bits/weight | memory | decode speedup† | what you risk |
|---|---|---|---|---|
| bf16 | 16 | 2.0 GB | ×1.0 | none — this is the training-precision baseline |
| int8 (g=32) | 8.5 | 1.06 GB | ≈ ×1.9 | near-lossless on most models |
| int4 (g=32) | 4.5 | 0.56 GB | ≈ ×3.6 | quality dips; outliers must be handled (GPTQ/AWQ) |
| int4 + 2:4 sparse | ≈ 2.8 | ≈ 0.35 GB | ≈ ×5.7* | compounding damage — needs fine-tuning to recover |
† rule of thumb: single-stream decode is memory-bandwidth-bound, so speedup ≈ compression ratio. *the sparse row additionally assumes 2:4-aware kernels; scales and indices add the overhead shown.
This table is the Phase 3 deliverable in miniature: the quantization-quality ablation (same bits-per-weight across GPTQ / AWQ / GGUF, perplexity and a downstream suite) and the distillation logit-sparsity study are both this page turned into a real experiment — with seeds, baselines, and a writeup.
§ 5 · The whole toolkit
Three trades, all measurable
precision is a budget
Outliers set the price: one weight can stretch the grid for all of them. Group the scales and the damage is contained.
soft targets teach geometry
The teacher's near-misses and absurdities carry bits the hard label doesn't have. That's the dark knowledge.
sparsity needs hardware
Scattered zeros save memory, not time. Structure the zeros the way the silicon wants and the speedup becomes real.
Every number on this page came from ~260 floats you can inspect. The real versions differ in scale, not in kind — which is exactly why this niche fits one GPU: measure the trade, publish the curve, repeat.