📄 Introducing, evaluating ukWaC, a very large web-derived corpus of English

Summary

The authors wanted a freely available English corpus that is much larger than traditional reference collections but still mixes genres and topics the way a balanced corpus does. They describe ukWaC, built in 2007 within the WaCky project by crawling the .uk web domain. Seed URLs came from Google queries made of random word pairs. The words were drawn from three vocabulary sources so that both formal and personal-interest pages would turn up. The crawled pages then passed through a cleaning pipeline:

• size and mime-type filters;
• removal of exact duplicates;
• a heuristic boilerplate stripper based on tag density;
• a filter based on the share of function words;
• removal of pornographic spam;
• detection of near-duplicates with shingling.

The result has more than 2 billion tokens, POS tags and lemmas. The authors evaluated what the corpus contains by comparing its vocabulary with the BNC. The resource matters because, at the time, it was among the largest English corpora of its kind and the only freely available, linguistically annotated English corpus taken from the web.

Key points

• Seeds came from 1,000 word pairs of mid-frequency BNC content words, 500 pairs from the spoken BNC and 500 pairs from a learners' vocabulary list. Each query returned at most 10 URLs, and only one URL was kept per domain.
• The crawl used Heritrix, was restricted to .uk and ran for 10 days. Only text/html documents between 5 and 200KB were kept, and every document with an exact duplicate was dropped, including all of its copies.
• The boilerplate stripper keeps the text span that maximises the number of tokens minus the number of tags. On the CLEANEVAL text-only task it scored 85.41 on average, while the best competitor scored 84.07.
• The connected-text filter used 151 function words. A page needed at least 10 types and 30 tokens, and at least a quarter of its words had to be function words. Near-duplicates were found by comparing 25 randomly chosen 5-grams per document, after function words had been removed.
• The corpus is distributed in a format ready for the IMS Open Corpus Workbench (CWB) and is also available through the Sketch Engine. The authors later parsed it with a dependency parser.

Interesting ideas

• I1. The genre balance of a crawl can be steered by where the seed words come from: words from newspapers and reference corpora bring public-sphere texts, while basic everyday vocabulary brings blogs and forums.
• I2. Boilerplate can be found without supervision by picking the stretch of a page where text is dense and markup is sparse. This cheap heuristic beat the supervised competitors in its evaluation.
• I3. A minimum share of function words is a single test that rejects lists and catalogues, keeps real connected prose and also works as a rough language identifier.
• I4. When data is plentiful, prefer precision over recall. Drop every copy of an exact duplicate, and detect near-duplicates only after cleaning, so that shared boilerplate does not cause false matches.
• I5. Judge a new corpus by comparing its vocabulary with an established reference corpus, so that differences in topic and register become visible.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): when collecting raw English for silver parsing, draw query or selection words from several vocabulary sources, so the silver data covers both formal prose and casual narrative.
• I2 (tokens): add a tag-density cleaner in front of the tokenizer, so navigation menus and disclaimers never become sentences that the tagger and parser have to handle.
• I3 (learn): run this test before sending text to the witness parsers, so that agreement among them is not inflated by easy fragments that are not real sentences.
• I4 (learn): deduplicate silver and teacher-labelled sentences with shingles on cleaned text, which also lowers the risk of evaluation items leaking into training data.
• I5 (learn): compare the vocabulary of the silver data with the vocabulary of the UD treebanks, to find domains where the parsers are likely to be weak and need negative controls.