📄 Estimation of English and non-English Language Use on the WWW

Summary

The paper asks how much text in each language is reachable on the Web. Earlier studies counted pages. This one estimates raw word volume per language. The method relies on a simple observation: frequent function words occur at fairly stable rates, so the observed count of such a word, divided by its expected relative frequency, predicts the size of the whole text. For 32 Latin-alphabet languages, the authors built a set of 20 high-frequency predictor words per language from 1 MB of training text. Words shared between languages were dropped so each list stays specific to its language. The individual predictions are combined with a trimmed mean. On held-out 1 MB files the error stayed near 10%, even in mixed-language text with added noise. Applied to Altavista hit counts, the method gives per-language word totals for February 2000 and growth figures since 1996. English dominates, but other languages are growing faster. The result offers a cheap tool for measuring language volume where only aggregate word frequencies are available.

Key points

• Building the predictors:
  • For each language, the authors kept the top 100 alphabetic tokens from 1 MB of text.
  • They removed any token that appeared in another language's list (for example, que).
  • The 20 most frequent remaining tokens, with their relative frequencies, became the predictors.
• Estimation procedure:
  • Tokenize by splitting on non-alphabetic characters, as Altavista appeared to do.
  • Divide each predictor's observed count by its relative frequency.
  • Discard the 2 highest and 2 lowest estimates, then average the remaining 16.
• Local evaluation on 11 languages:
  • Average error was ±6.2% on monolingual files.
  • On an 11 MB mixed file, average error was ±5.4%.
  • With added Russian, Thai and Maltese noise, average error was ±5.6%.
  • The worst case was English in the mixed file, at about +10–13%.
• Altavista estimates, February 2000:
  • English: about 47.3 billion words.
  • German: about 3.07 billion words.
  • French: about 2.2 billion words.
  • The English-to-German ratio of roughly 15:1 is close to the 14:1 page-based ratio from the Excite study.
• Growth from 1996 to 2000:
  • English grew about 800%, German about 1500% and Spanish about 1800%.
  • The authors warn that Altavista's index reaches roughly 16% of the indexable Web, and that this coverage may not be representative.

Interesting ideas

• I1. Removing high-frequency words that appear in several languages' lists leaves short, highly discriminative fingerprints for each language, which makes cheap language detection work even inside mixed text.
• I2. Many weak single-word estimators become reliable when outliers are trimmed and the rest are averaged, so no single unusual word can dominate the result.
• I3. The method was tested first on clean monolingual files, then on concatenated mixtures, then on mixtures with unrelated-language noise, which shows how robust it is step by step.
• I4. The authors deliberately matched their crude tokenization to the one used by the system whose counts they were querying, because predictors only work if tokens are counted the same way.
• I5. Expected rates of frequent words can serve as a quick sanity check on the size and composition of an opaque text collection.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (tokens): A small list of English-only function words could flag foreign-language spans in input, so the tagger and lemmatizer do not mis-analyse them as English.
• I2 (learn): The same trimmed aggregation could combine confidence or agreement scores from several witness parsers when building silver data, discarding the extreme witnesses before deciding.
• I3 (learn): Evaluation suites could follow the same pattern, adding off-target or noisy material as a negative control and checking that scores stay stable.
• I4 (tokens): When comparing token statistics with an external corpus or treebank, Mova's tokenizer should be aligned to that resource's conventions so frequency comparisons are meaningful.
• I5 (learn): Before training on scraped or teacher-generated text, comparing function-word rates against reference values could estimate how much genuine English it contains and flag contaminated batches.