Why data dominates — and why it's secret

Open-weight reports tell you everything about architecture and even training procedure — but the Llama 3 paper says essentially nothing about data. Two reasons: competitive secrecy (data is the secret sauce) and copyright liability (don't announce what you trained on).

The Llama 3 data section: full architecture transparency, but data is "a variety of sources." Source: CS336 Lecture 13.

Data scales with human effort (a long-tail problem), which is why data teams are huge. It enters at three stages, trending from large amounts of low-quality data → small amounts of high-quality data:

"Base model" = after pre+mid-training; "instruct/chat model" = after post-training — though the lines blur, and the biggest models increasingly release only the instruct model (no base checkpoint).

The web is live servers — you need a crawler, and you can't get it all

The web is a set of live servers; you can't train on those directly (unless you're an RL agent), so a crawler discovers and downloads pages from a seed set. But you can't download everything:

Dynamic content

Modern sites are apps; the URL isn't the content — you click buttons / submit forms (Discord, dashboards). The "deep web" isn't reachable by following hyperlinks.

Authentication / walled gardens

Huge content needs login/payment (Facebook, X, LinkedIn, NYTimes). Only the owner can train on it.

Technical restrictions

robots.txt (voluntary, lists disallowed bots), Cloudflare/CAPTCHAs, IP/country blocks, rate limits.

Legal restrictions

Terms of service may prohibit bot downloading; you may lack a license to copy pages for training.

"Consent in Crisis": robots.txt and ToS restrictions on common datasets' URLs shot up around mid-2023 — the legally-crawlable web is far smaller than the web. Source: CS336 Lecture 13.

Crawlers can also misbehave (Anthropic was accused of hitting a site ~1M times/24h) — overloading servers costs the host money before copyright even enters. And shadow libraries (LibGen, Anna's Archive, Sci-Hub) host millions of copyrighted books/papers for free — legally, piracy and copyright infringement.

Copyright: everything online is copyrighted

Intellectual-property law exists to incentivize creation. Under the US Copyright Act (1976): protection applies to "original works of authorship fixed in a tangible medium." Key facts: it covers expression, not ideas (you can't copyright quicksort); collections aren't copyrightable (a phone book) unless creatively arranged; no registration needed (your website is copyrighted the moment it's fixed; $65 to register before suing); it lasts 75 years, then enters the public domain. Net: basically everything on the internet is copyrighted. Two ways to use it:

Copyright is about semantics and economics, not verbatim n-gram overlap — characters/plots (Harry Potter) are protectable; parody is likely fair use. For LMs: copying data is already a potential violation; training feels transformative (extracting ideas, not re-hosting); but LMs can clearly affect the market. And ToS adds a layer (YouTube forbids downloading even CC-licensed videos). Recent rulings:

So far: training has been deemed fair use in specific instances (not in general); pirating books is clearly illegal; the area is active and evolving.

The key specialized sources

Common Crawl

Non-profit (since 2007); a monthly crawl adds ~2–5B pages (300B+ total). Two formats: WARC (raw HTTP/HTML) and WET (lossily converted to text). The HTML→text tool matters for downstream accuracy — trafilatura/resiliparse beat WET.

Wikipedia

67M articles; no original thought (everything cited), notability-gated. Download the periodic dumps (don't crawl). Caveat: data-poisoning — inject malicious edits right before a dump so they're captured before being reverted (even high-quality sources can be attacked).

GitHub

Code (helps reasoning, not just coding). Train only on permissively-licensed public repos (MIT/Apache). Two data types: repos (via git) and metadata (issues/PRs via the GitHub Archive event stream). Software Heritage aggregates repos across GitHub/GitLab/Bitbucket.

arXiv

~3M papers since 1991; each has metadata (CC0) + PDF + optional LaTeX source. Authors choose all-rights-reserved or Creative Commons. Bulk download from S3.

The chronological tour

Read across these and you see the same moves recur — crawl, then filter by rules or by a model/classifier, then deduplicate — with sizes ballooning and the rules-vs-classifiers divide running throughout.

The rules → classifiers shift, and the filtering funnel

Early datasets filtered with hand-written rules (C4, Gopher, RefinedWeb explicitly avoided ML filtering to dodge bias). DCLM marked the turn to model-based quality filtering as the norm — and it works strangely well: train a cheap fastText classifier on "good" (GPT-4 instruction data + ELI5) vs "bad" (raw RefinedWeb), then run it over a massive pool.

The funnel: DCLM's 240T-token raw pool → English + rules + dedup + model-based filtering → ~3.8T tokens (~1.4%). Source: CS336 Lecture 13.

And legally-clean training is possible but hard: Common Pile (8TB permissive-only) trains a decent model but can't yet match Qwen-class models — and even "permissive" is subtle (license laundering, collection licenses not extending to individual works, synthetic data from models trained on unlicensed data).

Golden rules & takeaways

• Data doesn't fall from the sky — it flows live service → raw data (crawl/dump) → processed data (transform → filter → dedup), and each step shapes the model.
• "Trained on the internet" is wrong — dynamic content, walled gardens, robots.txt/ToS, and rate limits make the accessible web far smaller, and shrinking over time.
• Everything online is copyrighted; you use it via a license (Creative Commons / paid deals) or fair use (transformative, factual, snippet, market-neutral). Training has been ruled fair use case-by-case; pirating is not.
• Know the sources — Common Crawl (WARC vs WET, conversion matters), Wikipedia/GitHub/arXiv (download dumps, don't crawl; mind licenses).
• Filtering is the highest-leverage step — the field shifted from hand rules to model-based quality classifiers (DCLM), with a 240T→~3T funnel.
• Data is the key differentiator across otherwise-similar transformers — and the pipeline is still mostly heuristic "vibes," so there's lots of room to improve.

Next: more on data — post-training data and a deeper look at filtering/deduplication/mixing.

References

Primary source is the lecture (executable lecture_13.py). Key works:

1. Longpre et al. (2024) — Consent in Crisis
2. Wenzek et al. (2019) — CCNet · Raffel et al. (2019) — T5 / C4
3. Gao et al. (2021) — The Pile · Rae et al. (2021) — Gopher / MassiveText
4. Touvron et al. (2023) — LLaMA · Penedo et al. (2023) — RefinedWeb
5. Soldaini et al. (2024) — Dolma · Li et al. (2024) — DataComp-LM (DCLM)
6. Kocetkov et al. (2022) — The Stack · Lozhkov et al. (2024) — The Stack v2
7. Kandpal et al. (2025) — The Common Pile
8. Carlini et al. (2023) — Poisoning Web-Scale Datasets