Interactive explainer · Phase 3 — Training & systems
Data
Architecture gets the papers; data gets the results. Nobody can read a trillion tokens, so the real work of pretraining is building filters that read them for you — and the field keeps rediscovering that a small model on deliberately curated data beats a bigger one on sludge. Below is a fourteen-document "web crawl" small enough to check every decision by eye, run through the real algorithms.
raw crawl → quality filters → deduplication → training mix — the pipeline nobody screenshots, and the biggest lever you own.
§ 1 · The raw crawl
What the internet actually looks like
Five documents of honest prose are in here — alongside navigation boilerplate, SEO keyword stuffing, code noise, a useless fragment, and the same bird story five times (two byte-identical copies, two lightly reworded). Real crawls run about this dirty. Skim them; the next two sections will find the junk without reading.
doc-0134 tokens
the bird ate the worm because it was hungry. after the meal it sat in the sun and cleaned its feathers. when the rain came it hid under the leaves and waited for morning.
doc-0222 tokens
Home | About | Contact | Sign in | Privacy Policy | Terms | Cookies | Careers | © 2026 Example Corp
doc-0319 tokens
small birds learn songs from their parents. a young bird practices for weeks until the song comes out right.
doc-0434 tokens
the bird ate the worm because it was hungry. after the meal it sat in the sun and cleaned its feathers. when the rain came it hid under the leaves and waited for morning.
doc-0525 tokens
best bird food best bird food cheap bird food buy bird food online bird food deals top rated bird food near me bird food sale
doc-0619 tokens
the river rises in spring when the snow melts. farmers watch the water and plan their planting around it.
doc-0734 tokens
the bird ate the worm because it was hungry. after the meal it sat in the shade and cleaned its feathers. when the rain came it hid under the leaves and waited for morning.
doc-0824 tokens
if (x_1 == y_2) { return z_3; } else { q += 0xFF; } // ??? $$ %% @@ ~~ [[ ]] << >>
doc-0920 tokens
bread needs flour, water, salt and time. the dough rests overnight and bakes in a hot oven in the morning.
doc-1015 tokens
Menu Home Products Pricing Blog Careers Support Sign Up Follow Facebook Twitter Instagram Newsletter Subscribe
doc-1134 tokens
the bird ate the worm because it was hungry. after the meal it sat in the sun and cleaned its feathers. when the rain came it hid under the leaves and waited for morning.
doc-125 tokens
click here to read more
doc-1321 tokens
so it goes: sun up, sun down. we eat, we rest, we try again. a small life, but a good one.
doc-1434 tokens
the bird ate the worm because it was hungry. after the meal it sat in the sun and cleaned its wings. when the rain came it hid under the leaves and waited for morning.
14 documents · 340 tokens, uncurated
§ 2 · Quality filters
Heuristics read the data so you can't have to
Five statistics, five thresholds — each one computed live on every document. The defaults catch exactly the junk. Now loosen one and watch it sneak back in, or tighten one and watch honest prose die with it. False positives aren't a bug in your pipeline; they're the price of one, and you choose the exchange rate.
doc-0134 tokens
the bird ate the worm because it was hungry. after the meal it sat in the sun and cleaned its feathers. when the rain came it hid under the leaves and waited for morning.
doc-0222 tokens
Home | About | Contact | Sign in | Privacy Policy | Terms | Cookies | Careers | © 2026 Example Corp
stopword fraction 0.08 < 0.12
doc-0319 tokens
small birds learn songs from their parents. a young bird practices for weeks until the song comes out right.
doc-0434 tokens
the bird ate the worm because it was hungry. after the meal it sat in the sun and cleaned its feathers. when the rain came it hid under the leaves and waited for morning.
doc-0525 tokens
best bird food best bird food cheap bird food buy bird food online bird food deals top rated bird food near me bird food sale
stopword fraction 0.00 < 0.12top-word repeat 0.28 > 0.25
doc-0619 tokens
the river rises in spring when the snow melts. farmers watch the water and plan their planting around it.
doc-0734 tokens
the bird ate the worm because it was hungry. after the meal it sat in the shade and cleaned its feathers. when the rain came it hid under the leaves and waited for morning.
doc-0824 tokens
if (x_1 == y_2) { return z_3; } else { q += 0xFF; } // ??? $$ %% @@ ~~ [[ ]] << >>
symbol ratio 1.89 > 0.30stopword fraction 0.00 < 0.12
doc-0920 tokens
bread needs flour, water, salt and time. the dough rests overnight and bakes in a hot oven in the morning.
doc-1015 tokens
Menu Home Products Pricing Blog Careers Support Sign Up Follow Facebook Twitter Instagram Newsletter Subscribe
stopword fraction 0.07 < 0.12
doc-1134 tokens
the bird ate the worm because it was hungry. after the meal it sat in the sun and cleaned its feathers. when the rain came it hid under the leaves and waited for morning.
doc-125 tokens
click here to read more
word count 5 < 8
doc-1321 tokens
so it goes: sun up, sun down. we eat, we rest, we try again. a small life, but a good one.
doc-1434 tokens
the bird ate the worm because it was hungry. after the meal it sat in the sun and cleaned its wings. when the rain came it hid under the leaves and waited for morning.
9/14 documents pass · 249 of 340 tokens survive. tighten mean word length past 2.95 and watch doc-13 — honest prose — die with the junk. every threshold is a tradeoff you own.
fig. 1 — these five heuristics are toys only in scale: FineWeb's published pipeline thresholds the same kinds of statistics over 15 trillion tokens.
§ 3 · Deduplication
The same page, over and over
Filters can't catch a document that's junk only because it already exists. Exact copies fall to a hash check; the reworded ones need similarity — here, Jaccard overlap on word 3-grams, computed on the quality survivors from §2.
exact copies caught: 2 · near-duplicates caught: 2
doc-04 is a byte-identical copy of doc-01 — dropped by hash, no similarity math needed.
doc-11 is a byte-identical copy of doc-01 — dropped by hash, no similarity math needed.
doc-01 · kept
the bird ate the worm because it was hungry after the meal it sat in the sun and cleaned its feathers when the rain came it hid under the leaves and waited for morning
doc-07 · dropped — 83% similar
the bird ate the worm because it was hungry after the meal it sat in the shade and cleaned its feathers when the rain came it hid under the leaves and waited for morning
doc-01 · kept
the bird ate the worm because it was hungry after the meal it sat in the sun and cleaned its feathers when the rain came it hid under the leaves and waited for morning
doc-14 · dropped — 83% similar
the bird ate the worm because it was hungry after the meal it sat in the sun and cleaned its wings when the rain came it hid under the leaves and waited for morning
Duplicates are poison three ways: the model memorizes repeated text instead of learning from it, benchmark answers hiding in duplicated pages contaminate your evals, and every copy skews the distribution toward one document's words — §4 measures that skew directly. Drag the threshold down and watch it get greedy; real pipelines tune this on samples, at MinHash scale.
§ 4 · What survives
The funnel, and the proof it mattered
The point of all this deleting is a better distribution, not a bigger pile. Compare the top tokens: the raw crawl is dominated by boilerplate and one over-copied story; the curated mix looks like language. Then the measurement — a held-out sentence scored under both unigram distributions.
raw crawl
14 docs · 340 tokens
quality-filtered
9 docs · 249 tokens
deduplicated
5 docs · 113 tokens
raw — top 15 tokens
curated — top 15 tokens
held-out test · unigram, add-one smoothed
"the young bird practices its song in the spring when the water rises."
cross-entropy under raw crawl 4.33 nats · under curated corpus 3.96 nats — the same sentence is 31% less surprising after curation, with less than half the tokens.
fig. 2 — this lineage is the niche: TinyStories (curated synthetic stories), Phi ("textbooks are all you need"), SmolLM — small models that punch up because their data was chosen, not scraped. Data curation is a Phase 5 specialization option for exactly this reason.
§ 5 · The whole trick
Three rules of the mix
filters read for you
Cheap statistics — word length, symbol ratio, stopwords, repetition — separate prose from sludge at any scale. You own the thresholds and their false positives.
dedup before you trust anything
Duplicates memorize, contaminate evals, and skew the distribution. Hash the exact copies, MinHash the near ones, always.
curation is the cheapest scaling law
A third of the tokens, a better model. Improving the mix costs CPU; improving the architecture costs a research career.
Everything above ran on fourteen documents, but nothing about it was fake — the same statistics, the same Jaccard, the same funnel run over Common Crawl is literally a CS336 assignment (raw crawl → filter → dedup → pretraining data). When a training run disappoints, suspect the data before the architecture — it's the higher-leverage, less glamorous place to look.