Elamur Research⌕
☰
home / papers / 2014 / N-gram Counts and Language Models from the Common Crawl

N-gram Counts and Language Models from the Common Crawl

Summary

The paper tackles a gap in public web-scale n-gram resources. The released Google n-gram counts discard rare n-grams and rare words, which rules out proper Kneser-Ney estimation. They also contain undeduplicated boilerplate, and the Microsoft web service covers only English and could not handle the query load. The authors built a multilingual resource from three Common Crawl snapshots (2012, early 2013, winter 2013) using only the text-only files. Their pipeline has four main steps:

  • CLD2 language identification at the level of monolingual spans
  • line-level deduplication via hashing
  • normalization, Moses tokenization and truecasing
  • disk-based streaming estimation of unpruned interpolated modified Kneser-Ney 5-gram models with KenLM

The largest model, for English, comes from about 975 billion deduplicated tokens and holds over 500 billion unique n-grams. On English newstest 2014 it clearly beats an interpolation of all WMT constrained corpora in perplexity. Adding it to strong Moses systems raised BLEU by 0.5–1.4 points. The resource matters because it supplies unpruned counts, deduplicated text and models for many languages beyond English.

Key points

  • Input data: 3.8B + 2B + 2.3B crawled pages, about 35.23 TiB of text in total. CLD2, which recognizes 175 languages, processed the whole corpus in roughly a week. Of those languages, 73 had 1 GiB or more of text, and 42 had 10 GiB or more.
  • Deduplication: Each line got a 64-bit MurmurHash, and the data was sharded by a second hash. This removed about 80% of English lines and roughly two thirds of German and Spanish lines. A 10 GiB check found no false duplicates from hash collisions.
  • English model size: The model has about 2.6B unigrams and about 264B 5-grams, occupying 5.6 TB. For scale, a model on 535B tokens took 8.2 days on one machine with 140 GiB RAM.
  • Perplexity:
    • English newstest 2014: 58.44 (skipping unknown words) versus 92.69 for interpolated WMT data.
    • French: the cleaner interpolated WMT data still did better (60.78 vs 65.50).
  • Machine translation:
    • BLEU improved for English→Czech, Hindi, Russian and German, and for Hindi→English.
    • In English–Spanish tests, gains grew as more Common Crawl lines were used, from 50M up to 1.3B, and even the smallest sample helped.

Interesting ideas

  • I1. Split documents into monolingual spans. Instead of giving each page one language, segment it into same-language spans. Mixed-language pages are common on the web, so this recovers usable text for smaller languages that would otherwise be mislabeled or lost.
  • I2. Deduplicate at scale with hashes and shards. Keep only hashes of lines in memory, and route identical lines to the same shard with a second hash so each shard can be deduplicated independently. This is a cheap, memory-bounded way to strip boilerplate from very large crawls.
  • I3. Keep singletons, prune later. Kneser-Ney needs full counts even if the final model will be pruned. Discarding rare items early can make a resource unusable for some estimation methods.
  • I4. Fix the vocabulary size when comparing perplexities. Give all models the same vocabulary size, so words unseen in a model's training data behave like copies of the unknown token. This stops a model from looking good merely by having a small vocabulary.
  • I5. Make releases reproducible and measure data scaling carefully.
    • Release both the untokenized text and frozen preprocessing scripts plus the truecasing model, so others can either reuse the exact pipeline or apply their own tokenizer.
    • For scaling experiments, use nested samples, where each larger sample contains the smaller ones, so differences reflect data quantity rather than sample composition.
    • Choose the newest test sets to reduce the chance of test-set leakage from web data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Machine translation

Cited by

76 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 60 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{buck2014n,
  title = {{N-gram Counts and Language Models from the Common Crawl}},
  author = {Buck, Christian and Heafield, Kenneth and Ooyen, Bas van},
  booktitle = {Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14)},
  year = {2014},
  doi = {10.63317/3uyduafd26my},
  url = {https://doi.org/10.63317/3uyduafd26my}
}

References

25 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shane Bergsma, Emily Pitler, and Dekang Lin. 2010. Creating robust supervised classifiers via web-scale n-gram data. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 865– 874. Association for Computational Linguistics, July. find
  2. Alexandra Birch, Nadir Durrani, and Philipp Koehn. 2013. Edinburgh SLT and MT System Description for the IWSLT 2013 Evaluation. In Proceedings of the 10th International Workshop on Spoken Language Translation, pages 40–48, Heidelberg, Germany, December. find
  3. Ondřej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Transl… find our card
  4. Thorsten Brants and Alex Franz. 2006. The Google web 1T 5-gram corpus version 1.1. LDC2006T13. find
  5. Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. 2007. Large language models in machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Language Learning, pages 858–867, June. find our card
  6. Ciprian Chelba and Johan Schalkwyk, 2013. Empirical Exploration of Language Modeling for the google.com Query Stream as Applied to Mobile Voice Search, pages 197–229. Springer, New York. find
  7. Stanley Chen and Joshua Goodman. 1998. An empirical study of smoothing techniques for language modeling. Technical Report TR-10-98, Harvard University, August. find our card
  8. Nadir Durrani, Alexander Fraser, Helmut Schmid, Hieu Hoang, and Philipp Koehn. 2013. Can Markov Models Over Minimal Translation Units Help Phrase-Based SMT? In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria, August. Association for Computatio… find
  9. Nadir Durrani, Hassan Sajjad, Hieu Hoang, and Philipp Koehn. 2014. Integrating an Unsupervised Transliteration Model into Statistical Machine Translation. In Proceedings of the 15th Conference of the European Chapter of the ACL (EACL 2014), Gothenburg, Sweden, April. Association for Computational Li… find
  10. Marcello Federico, Nicola Bertoldi, and Mauro Cettolo. 2008. IRSTLM: an open source toolkit for handling large scale language models. In Proceedings of Interspeech, Brisbane, Australia. find
  11. David Guthrie and Mark Hepple. 2010. Storing the web in memory: Space efficient language models with constant time retrieval. In Proceedings of EMNLP 2010, Los Angeles, CA. find
  12. Eva Hasler, Barry Haddow, and Philipp Koehn. 2012. Sparse Lexicalised features and Topic Adaptation for SMT. In Proceedings of the seventh International Workshop on Spoken Language Translation (IWSLT), pages 268–275. find
  13. Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. Scalable modified Kneser-Ney language model estimation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria, August. find our card
  14. Liang Huang and David Chiang. 2007. Forest Rescoring: Faster Decoding with Integrated Language Models. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 144–151, Prague, Czech Republic, June. Association for Computational Linguistics. find our card
  15. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing, pages 181–184. find
  16. Philipp Koehn and Hieu Hoang. 2007. Factored Translation Models. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 868–876, Prague, Czech Republic, June. Association for Computational Lingu… find our card
  17. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondřej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Ann… find
  18. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proceedings of MT Summit. find
  19. Shankar Kumar and William J. Byrne. 2004. Minimum Bayes-Risk Decoding for Statistical Machine Translation. In HLT-NAACL, pages 169–176. find our card
  20. Dekang Lin, Kenneth Church, Heng Ji, Satoshi Sekine, David Yarowsky, Shane Bergsma andKailash Patil, Emily Pitler, Rachel Lathbury, Vikram Rao, Kapil Dal-wani, and Sushant Narsale. 2010. Final report of the 2009 JHU CLSP workshop, June. find
  21. Dekang Lin. 2013. Personal communication, October. find
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evalution of machine translation. In Proceedings 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, PA, July. find
  23. Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English gigaword fifth edition, June. LDC2011T07. find
  24. Jason Smith, Hervé Saint-Amand, Magdalena Plamada, Philipp Koehn, Chris Callison-Burch, and Adam Lopez. 2013. Dirt cheap web-scale parallel text from the common crawl. In Proceedings of ACL. Association for Computational Linguistics, August. find
  25. Kuansan Wang, Christopher Thrasher, Evelyne Viegas, Xiaolong Li, and Bo-june Paul Hsu. 2010. An overview of Microsoft web n-gram corpus and applications. In Proceedings of the NAACL HLT 2010 Demonstration Session, pages 45–48. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/buck-2014-n-gram-counts-language-models/