📄 Building Large Corpora from the Web Using a New Efficient Tool Chain

Summary

Schäfer and Bildhauer address two problems in building very large web corpora. Seed URLs and crawls can be dominated by a small number of hosts, and raw HTML needs heavy cleaning before linguists can use it. Their approach takes seeds from search engines (Yahoo, later Bing), queried with random triples of mid-frequency content words, and feeds them to long Heritrix 1.4 crawls restricted to national TLDs. The harvested pages then pass through their own ObjectPascal tool chain, which handles markup cleanup, neural-network boilerplate removal, connected-text detection and shingling-based near-duplicate removal. The output is the COW corpora, the largest of which (DECOW2012) holds over 9 billion tokens. The authors measure host bias with cumulative plots and show that a large seed set plus a long crawl spreads documents across far more hosts than deWaC does. They also estimate the genre composition of their corpora by manually coding samples. The work matters because it shows that heavy reliance on search engines can skew a corpus, and it offers a reusable, efficient pipeline for comparable corpora in many languages.

Key points

• Corpus sizes:
  • SECOW2011: 1.91M documents and about 1.75 billion tokens.
  • ESCOW2012: 1.30M documents and about 1.59 billion tokens.
  • DECOW2012: 7.63M documents and about 9.1 billion tokens.
• Seed collection: queries were 3-tuples of content word forms ranked 1,001–6,000 by frequency, with at most 10 results per query. The DECOW2012 seed set contained over 900,000 unique URLs, and the German crawl lasted 28 days, compared with 7 days for the Swedish one.
• Host bias, DECOW2012: in the seeds, 50% of URLs came from 258 hosts. In the final corpus, the same half of documents was spread over 1,276 hosts. The average was 20 documents per host, against 158 for a shingled version of deWaC.
• Host bias, SECOW2011: the Yahoo seeds were highly concentrated, with 29 hosts supplying half of them. A breadth-first crawl then made the problem worse: 75.5% of the documents come from blogg.se, which skews the corpus towards blog language.
• Boilerplate and genre evaluation:
  • The boilerplate classifier is a single neural network trained on 1,000 human-coded German paragraphs using the textrain tool. Its performance held up just as well on Swedish, Spanish, French and English as on German.
  • Genre was coded on samples of n = 200 documents. Inter-rater Cohen's κ ranged from .61 to .94 across categories, and results are reported with 90% confidence intervals.

Interesting ideas

• I1. Host concentration as a cheap bias check: plot what share of a URL sample comes from the top-n hosts. This gives a quick, language-independent warning that a crawl is being swallowed by a few sites. It could be applied to any Ukrainian web data collection.
• I2. Why a simple per-host cap falls short: limiting the raw number of pages per host is not enough, because what should be capped is the number of good documents per host. Document quality only becomes visible after cleaning, so crawl control and post-processing should be designed together.
• I3. A deliberately minimal annotation guideline: paragraphs were labelled text versus boilerplate using one rule, roughly "coherent full sentences or their headings count as text, everything else is boilerplate." Keeping the rule this crude makes human labelling fast and consistent. A lightweight classifier trained on such labels may transfer across languages if it avoids language-specific features.
• I4. Hierarchical fallback labels for mixed pages: when a document fits several genre categories, the annotator picks the topmost value in the category's list. For frequent mixtures such as blog posts with comments, the authors added a dedicated value ("blogmix"). Both choices keep manual genre coding tractable.
• I5. Confidence intervals on small manual samples: the genre percentages are reported with 90% confidence intervals. This makes it explicit how uncertain composition estimates from n = 200 documents are, and signals when a larger sample is needed.