📄 Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages

Summary

The paper reports on how the Leipzig Corpora Collection (LCC) is scaling its corpus-based monolingual dictionaries to many more languages, with special attention to "low density" languages that have little text online. Each LCC dictionary lists word-form frequencies (no lemmatization), example sentences, significant co-occurrences and a semantic map of the strongest co-occurrences. At the time of writing, 136 languages were available online.

The authors gather text from four kinds of sources:

• online newspapers listed by ABYZ News Links,
• generic Web crawling with the distributed FindLinks client and an HTTrack-based crawler,
• search-engine bootstrapping seeded with frequent words from UDHR and Watchtower texts,
• Wikipedia dumps.

A largely language-independent pipeline then processes the text: HTML-stripping, language identification with LangSepa, sentence segmentation, cleaning, deduplication, sentence scrambling for copyright reasons, database construction and statistics.

Because hundreds of corpora cannot be checked by hand, quality is monitored automatically with statistical features. The resulting resources are free to query and download, which supports typological and comparative research across many languages.

Key points

• About 32,000 ABYZ News Links sources in over 120 languages were crawled. This covered roughly 100 languages and produced more than 250 GB of stripped text, from which dictionaries for 70 languages were built.
• The FindLinks volunteer client had downloaded around 1 TB of raw text, with about 10 users fetching over 30 million pages per day. None of this data had yet been turned into corpora.
• Wikipedia dumps for more than 200 languages yielded 34 GB of text and about 70 dictionaries. Small Wikipedias were excluded because their stub articles create near-duplicate sentences.
• LangSepa identifies languages from stop-words and character uni- and trigram distributions. Using profiles built from LCC, UDHR and Watchtower texts, it covers about 450 languages.
• For comparison, Crúbadán covers 1023 languages, but for nearly 60% of them it holds 5 or fewer documents. Web As Corpus queries Bing directly for 34 languages without any cleaning or statistics.

Interesting ideas

• I1. Parallel texts as seeds for rare languages: Widely translated texts like the UDHR can supply seed vocabularies and language-ID profiles where nothing else exists. Combining 3–5 frequent words into search queries then retrieves fresh Web pages, a cheap way to start a corpus for an under-resourced language.
• I2. Statistical quality checks instead of manual review: Corpora can be checked by measuring sentence-length distributions, character and n-gram profiles, rare characters and conformity to Zipf's law. In the paper's example, an odd two-peaked sentence-length curve in the Sundanese Wikipedia exposed boilerplate, so such checks scale to hundreds of corpora.
• I3. Sentence-level scrambling for licensing: The pipeline breaks up document structure and keeps only individual sentences, relying on German law, which does not protect single sentences. This lets the project redistribute Web-derived data, and it could be a model for sharing corpora when source licenses are restrictive.
• I4. Biasing crawls toward smaller countries: The crawler deliberately limits URLs from large TLDs like .com or .de, which surfaces sites from smaller countries and possibly rare languages. Using language-speaker data such as Ethnologue to steer crawling toward languages that still lack coverage could help, for example, find more Ukrainian or minority-language content.
• I5. Standardized corpus sizes: Publishing corpora in fixed sizes makes statistics comparable across languages and sources. This is a useful design choice for any multilingual benchmark or frequency resource.