📄 Building a 70 billion word corpus of English from ClueWeb

Summary

Large corpora are needed because rare words, collocations and patterns only appear with enough text. Running a multi-terabyte crawl is hard in an academic setting, so the authors built an English web corpus from the English half of ClueWeb09, a fixed crawl of about 500 million pages. Their pipeline had five steps:

• boilerplate stripping with jusText;
• a character-trigram language filter;
• removal of exact duplicates and near-duplicates with their tool onion;
• POS tagging with TreeTagger;
• indexing in the Manatee/Bonito corpus manager.

The hardest parts were de-duplication and indexing at the scale of tens of billions of words. For onion, they cut memory use by discarding n-grams that occur only once in the whole corpus. For Manatee, they parallelised encoding by first building the lexicon sequentially. The result has roughly 70 billion alphanumeric tokens and more than 5× faster indexing. Word sketches were computed in parallel in about 8 days. The work shows that web corpora of 10–100 billion words can be built reproducibly on a single large server.

Key points

• Final corpus: about 68.8 billion word forms, 70.3 billion alphanumeric tokens and about 82 billion tokens in total, from close to 139 million documents.
• On a 7 % sample of ClueWeb, the steps removed text as follows:
  • jusText kept only 46.1 % of the words.
  • The language filter (trigram.py, threshold 0.4) kept 84.4 % of the remainder.
  • Exact-duplicate removal kept 93.7 %.
  • Near-duplicate removal kept 81.0 %, leaving 7.28 billion words out of 24.6 billion.
• onion compares each document's word 10-grams against the 10-grams already seen. A document is dropped if too large a share of its text is duplicated, which catches near-duplicates at any level of similarity.
• Hardware was one server with 64 Xeon cores, 440 GB RAM and a 70 TB RAID-6 array.
  • Boilerplate removal and language filtering took 108 hours.
  • The single-threaded de-duplication took 65 hours and 148 GB RAM.
  • POS tagging took 44 hours.
• Parallel encoding in Manatee was more than 5× faster than sequential encoding. Parallel word-sketch computation was 8× faster, finishing in about 8 days.

Interesting ideas

• I1. Before de-duplicating, run one pass to find which n-gram hashes occur at least twice, then never store the singletons (over 90 % of all n-grams); this shrinks the "seen" set enormously without losing any detection.
• I2. Find duplicates by splitting hashes into buckets by hash range, writing each bucket to its own file and sorting each file in memory; this is a simple external-memory method that needs no special infrastructure.
• I3. Because a vocabulary grows fast early on and then levels off (Zipf), build the string-to-id lexicon sequentially on the first tens of millions of tokens, then let parallel workers that only rarely need to add new entries continue.
• I4. Record how much data each pipeline stage removes, so each filter's effect is visible instead of being hidden inside the final number.
• I5. Fields that are unique per item, such as a document URL, need only a simple string-to-id map rather than full inverted indexes, which saves a lot of indexing work.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): Use the same trick to cheaply find overlap between silver training text, LLM-teacher outputs and held-out test sets, so leaked evaluation items are caught before training.
• I2 (learn): This lets us remove repeated sentences from large silver treebanks on a single machine, so frequent boilerplate sentences do not dominate parser training.
• I3 (tokens): Build word-form and lemma id tables this way in crate en, so corpus-scale tagging and lemmatisation can run in parallel while ids stay consistent.
• I4 (learn): Log how many sentences each silver-data filter (witness agreement, repair rules) drops, so an overly aggressive filter or a silent failure shows up immediately.
• I5 (world): Store unique identifiers such as source IDs or Wikidata QIDs in the knowledge graph as plain lookup entries, and keep the heavier indexes for attributes that are actually queried.