Elamur Research⌕
☰
home / papers / 2020 / Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus

Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus

Summary

The paper argues that language identification (LangID), often considered a finished problem, breaks down when used to build web corpora for many low-resource languages. The Google Research authors trained n-gram CLD3-style models covering up to 1,629 language varieties. These models scored well on held-out test sets. However, when native speakers checked the resulting web-crawl data, only a small fraction of sentences for many languages were actually in the target language. The authors trace the failures to several causes:

  • a mismatch between clean training text and noisy web content;
  • extreme class imbalance on the web, which makes balanced-test precision misleading;
  • confusion with closely related or more widespread languages;
  • the limited capacity of n-gram models.

They propose two remedies. The first is wordlist-based filters whose precision can be tuned. The second is semi-supervised Transformer LangID models (SS-LID). Together, these turn mostly unusable crawls into reasonably clean corpora. The work matters because monolingual web text is a key input for extending MT, ASR and other cross-lingual NLP to the thousands of languages that current tools ignore.

Key points

  • Benchmark vs. real-world quality: The 1,629-language model reached 90.4% macro-average F1 and 97.9% median F1 on held-out data. Human-judged in-language rates on crawled data were nonetheless around 5% for many low-resource languages.
  • Effect of filtering: Filtering raised median dataset precision from 5.5% to 71.2%. Wordlist methods (TF-IDF-IIF lists, percent-threshold filtering) were better for some languages, such as Oromo and Cherokee. SS-LID was better for others, such as Bhojpuri and Swiss German.
  • Hardest cases: Languages with a dominant related language ("High-Prevalence Cousin") and/or a related language missing from the model ("Out-of-Model Cousin") were the most difficult. Guinea-Bissau Creole could not be cleaned by any method tested.
  • Released resource: Curated wordlists for about 500 languages.
  • Final corpus: A deep crawl of more than 100B pages with a 600-language model produced 100K+ fairly clean sentences in each of 500+ languages. For shared low-resource languages, this is roughly 20x more data than CCNet and 180x more than OSCAR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Computing & GPUs

Cited by

69 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 62 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{caswell2020language,
  title = {{Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus}},
  author = {Caswell, Isaac and Breiner, Theresa and Esch, Daan van and Bapna, Ankur},
  booktitle = {COLING 2020},
  year = {2020},
  doi = {10.18653/v1/2020.coling-main.579},
  url = {https://doi.org/10.18653/v1/2020.coling-main.579}
}

References

61 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Željko Agić and Ivan Vulić. 2019. JW300: A wide-coverage parallel corpus for low-resource languages. find our card
  2. Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A. Smith. 2016. Massively Multilingual Word Embeddings. ArXiv, abs/1602.01925. find our card
  3. Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer, Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 4218–4… find our card
  4. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. CoRR, abs… find our card
  5. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2017. Unsupervised neural machine translation. arXiv preprint arXiv:1710.11041. find our card
  6. Mikel Artetxe, Sebastian Ruder, Dani Yogatama, Gorka Labaka, and Eneko Agirre. 2020. A Call for More Rigor in Unsupervised Cross-lingual Learning. In ACL. find
  7. Anton Bakalov, Alex Salcianu, Andy Golding, Chris Alberti, Daniel Andor, David Weiss, Emily Pitler, Greg Coppola, Jason Riesa, Kuzman Ganchev, Michael Ringgaard, Nan Hua, Ryan McDonald, Slav Petrov, Stefan Istrate, and Terry Koo. 2016. Compact Language Detector v3 (CLD3), October. find
  8. Adrien Barbaresi, Felix Bildhauer, Roland Schäfer, and Egon Stemle, editors. 2020. Proceedings of the 12th Web as Corpus Workshop, Marseille, France, May. European Language Resources Association. find
  9. P. Biggs. 2017. The state of broadband: Broadband catalyzing sustainable development. Technical report, International Telecommunications Union and Broadband Commission for Sustainable Development and UNESCO. find
  10. Harry Bleyan, Sandy Ritchie, Jonas Fromseier Mortensen, and Daan van Esch. 2019. Developing Pronunciation Models in New Languages Faster by Exploiting Common Grapheme-to-Phoneme Correspondences Across Languages. In Proceedings of Interspeech 2019. find
  11. Su Lin Blodgett, Johnny Tian-Zheng Wei, and Brendan O’Connor. 2017. A Dataset and Classifier for Recognizing Social Media English. ACL. find
  12. Sascha Brawer. 2017. Corpus Crawler. https://github.com/google/corpuscrawler, September. ↗
  13. Ralf D Brown. 2012. Finding and identifying text in 900+ languages. Digital Investigation, 9:S34–S43. find
  14. Ralf D Brown. 2013. Selecting and weighting n-grams to identify 1100 languages. In International Conference on Text, Speech and Dialogue, pages 475–483. Springer. find
  15. Ralf Brown. 2014. Non-linear mapping for improved identification of 1300+ languages. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 627–632, Doha, Qatar, October. Association for Computational Linguistics. find
  16. Christian Buck, Kenneth Heafield, and Bas Van Ooyen. 2014. N-gram Counts and Language Models from the Common Crawl. In LREC, volume 2, page 4. Citeseer. find our card
  17. Gina Bustamante, Arturo Oncevay, and Roberto Zariquiey. 2020. No Data to Crawl? Monolingual Corpus Creation from PDF Files of Truly low-Resource Languages in Peru. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 2914–2923, Marseille, France, May. European Language Reso… find
  18. Cornelia Channing. 2020. A Biologist Explains Why She and 1.8 Million Others Are Role-Playing as Ants on Facebook. https://slate.com/culture/2020/05/ facebook-ants-roleplay-coronavirus-biologist-interview.html. Accessed: 2020-06-01. ↗
  19. Mason Chua, Daan van Esch, Noah Coccaro, Eunjoon Cho, Sujeet Bhandari, and Libin Jia. 2018. Text Normalization Infrastructure that Scales to Hundreds of Language Varieties. In Proc. of the 11th edition of the Language Resources and Evaluation Conference, LREC 2018, 7-12 May 2018, Miyazaki, Japan. find
  20. Aliya Deri and Kevin Knight. 2016. Grapheme-to-phoneme models for (almost) any language. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 399–408, Berlin, Germany, August. Association for Computational Linguistics. find
  21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  22. Jonathan Dunn. 2020. Mapping languages: the Corpus of Global Language Use. Language Resources and Evaluation, April. find
  23. David M. Eberhard, Gary F. Simons, and Charles D. Fennig, editors. 2019. Ethnologue: Languages of the World. SIL International, Dallas, Texas, 22 edition. find
  24. Dirk Goldhahn, Thomas Eckart, and Uwe Quasthoff. 2012. Building large monolingual dictionaries at the leipzig corpora collection: From 100 to 200 languages. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12), pages 759–765, Istanbul, Turkey, May. Eur… find our card
  25. Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and Harnessing Adversarial Examples. ICLR. find our card
  26. E. Grave. 2017. Language identification, October. Accessed: October 22, 2020. find
  27. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. arXiv preprint arXiv:2003.11080. find our card
  28. Aaron Jaech, George Mulcaire, Shobhit Hathi, Mari Ostendorf, and Noah A. Smith. 2016. Hierarchical Character-Word Models for Language Identification. find
  29. Miloš Jakubı́ček, Vojtěch Kovář, Pavel Rychlý, and Vit Suchomel. 2020. Current challenges in web corpus building. In Proceedings of the 12th Web as Corpus Workshop, pages 1–4, Marseille, France, May. European Language Resources Association. find
  30. Tommi Jauhiainen, Marco Lui, Marcos Zampieri, Timothy Baldwin, and Krister Lindén. 2018. Automatic Language Identification in Texts: A Survey. CoRR, abs/1804.08186. find our card
  31. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing Crosslingual Distributed Representations of Words. In COLING. find
  32. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. arXiv preprint arXiv:1808.06226. find our card
  33. Guillaume Lample and Alexis Conneau. 2019. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291. find
  34. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2017. Unsupervised machine translation using monolingual corpora only. arXiv preprint arXiv:1711.00043. find our card
  35. Jackson L. Lee, Lucas F.E. Ashby, M. Elizabeth Garza, Yeonju Lee-Sikka, Sean Miller, Alan Wong, Arya D. McCarthy, and Kyle Gorman. 2020. Massively multilingual pronunciation modeling with WikiPron. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 4223–4228, Marseille, F… find
  36. Xinjian Li, Siddharth Dalmia, Juncheng Li, Matthew Lee, Patrick Littell, Jiali Yao, Antonios Anastasopoulos, David R. Mortensen, Graham Neubig, Alan W Black, and Florian Metze. 2020. Universal Phone Recognition with a Multilingual Allophone System. find
  37. Kelly Marchisio, Kevin Duh, and Philipp Koehn. 2020. When Does Unsupervised Machine Translation Work? find
  38. Thomas Mayer and Michael Cysouw. 2014. Creating a Massively Parallel Bible Corpus. In Ninth International Conference on Language Resources and Evaluation, LREC 2014, 26-31 May 2014, Reykjavik, Iceland, pages 3158–3163. Research Unit Quantitative Language Comparison, Philipps University of Marburg. find our card
  39. David R. Mortensen, Siddharth Dalmia, and Patrick Littell. 2018. Epitran: Precision G2P for many languages. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May. European Language Resources Association (ELRA). find our card
  40. Pedro Javier Ortiz Suárez, Benoı̂t Sagot, and Laurent Romary. 2019. Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7), Cardiff, United Kingdom, July. find
  41. T. Ouyang, D. Rybach, F. Beaufays, and M. Riley. 2017. Mobile Keyboard Input Decoding with Finite-State Transducers. CoRR abs/1704.03987. — [arXiv:1704.03987](https://arxiv.org/abs/1704.03987) find
  42. Isabel Papadimitriou and Dan Jurafsky. 2020. Pretraining on Non-linguistic Structure as a Tool for Analyzing Learning Bias in Language Models. arXiv preprint arXiv:2004.14601. — [arXiv:2004.14601](https://arxiv.org/abs/2004.14601) find
  43. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-based Framework for Multi-task Cross-lingual Transfer. find our card
  44. Manasa Prasad, Theresa Breiner, and Daan van Esch. 2018. Mining training data for language modeling across the world’s languages. In Proc. The 6th Intl. Workshop on Spoken Language Technologies for Under-Resourced Languages, pages 61–65. find
  45. Manasa Prasad, Daan van Esch, Sandy Ritchie, and Jonas Fromseier Mortensen. 2019. Building Large-Vocabulary ASR Systems for Languages Without Any Audio Training Data. In Proceedings of Interspeech 2019. find
  46. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683. find
  47. Sandy Ritchie, Richard Sproat, Kyle Gorman, Daan van Esch, Christian Schallhart, Nikos Bampounis, Benoit Brard, Jonas Fromseier Mortensen, Millie Holt, and Eoin Mahon. 2019. Unified Verbalization for Speech Recognition & Synthesis Across Languages. In Proceedings of Interspeech 2019. find
  48. Sandy Ritchie, Eoin Mahon, Kim Anne Heiligenstein, Nikos Bampounis, Daan van Esch, Christian Schallhart, Jonas Fromseier Mortensen, and Benoit Brard. 2020. Data-Driven Parametric Text Normalization: Rapidly Scaling Finite-State Transduction Verbalizers to New Languages. In Proceedings of the 1st Joi… find
  49. Brian Roark, Lawrence Wolf-Sonkin, Christo Kirov, Sabrina J. Mielke, Cibu Johny, Isin Demirsahin, and Keith Hall. 2020. Processing South Asian languages written in the Latin script: the dakshina dataset. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 2413–2423, Marsei… find
  50. K. P. Scannell. 2007. The Crúbadán Project: Corpus building for under-resourced languages. In 3rd Web as Corpus Workshop, 2007, Louvain-la-Neuve, Belgium. find
  51. Aditya Siddhant, Ankur Bapna, Yuan Cao, Orhan Firat, Mia Chen, Sneha Kudugunta, Naveen Arivazhagan, and Yonghui Wu. 2020. Leveraging monolingual data with self-supervision for multilingual neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Ling… find
  52. UmrinderPal Singh and Vishal Goyal. 2013. Font identifier and Unicode converter for Hindi. Nepalese Linguistics. find
  53. Claudia Soria. 2018. Digital Language Survival Kit. http://www.dldp.eu/en/content/digital-language-survival-kit. ↗
  54. Daan van Esch, Elnaz Sarbar, Tamar Lucassen, Jeremy O’Brien, Theresa Breiner, Manasa Prasad, Evan Elizabeth Crew, Chieu Nguyen, and Francoise Beaufays. 2019. Writing Across the World’s Languages: Deep Internationalization for Gboard, the Google Keyboard. Technical report. find
  55. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  56. Duy Tin Vo and Richard Khoury. 2019. Language Identification on Massive Datasets of Short Message using an Attention Mechanism CNN. find
  57. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. find our card
  58. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, and Edouard Grave. 2019. CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data, 11. find our card
  59. Shijie Wu and Mark Dredze. 2020. Are all languages created equal in multilingual BERT? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 120–130, Online, July. Association for Computational Linguistics. find our card
  60. Yinfei Yang and Fanxiaoyu Feng. 2020. Language-Agnostic BERT Sentence Embedding, August. Accessed: September 25, 2020. find our card
  61. George Zipf. 1935. The Psychology of Language. Houghton-Mifflin. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/caswell-2020-language-id-wild-unexpected/