Elamur Research
☰
home / papers / 2020 / The State and Fate of Linguistic Diversity and Inclusion in the NLP World

The State and Fate of Linguistic Diversity and Inclusion in the NLP World

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2004.09095

Summary

Although the world has more than 7,000 languages, only a small share benefit from modern NLP, and many systems described as "language agnostic" have only been tested on a few related, well-resourced languages. Joshi and colleagues at Microsoft Research India measure this gap. For each language they count two kinds of data. Labeled resources are datasets listed in the LDC catalog and the ELRA Map. Unlabeled resources are measured by Wikipedia pages. These two counts sort languages into a six-class taxonomy, from class 0, the "Left-Behinds", to class 5, the "Winners". The authors then use the classes to ask three questions. How do resources relate to speaker numbers? Which typological features do resource-rich languages fail to cover? How have ACL, NAACL, EACL, EMNLP, LREC and workshop papers spread their attention across languages over the decades? To answer the last one they use entropy measures and embeddings of conferences, languages and authors. The same hierarchy shows up in every analysis. The paper matters because it gives a shared vocabulary for resource inequality. Ukrainian falls in class 3, the "Rising Stars": it has plenty of unlabeled text but few labeled datasets.

Key points

  • Class 0 holds 2,191 languages (88.17% of the total, about 1B speakers). Class 5 has only 7 languages, including English, Spanish, German, Japanese and French, which together cover about 2.5B speakers.
  • Class 3 contains Ukrainian, Indonesian, Cebuano, Afrikaans and Hebrew. These languages have a strong web presence, and the authors expect them to gain most from multilingual pre-training such as mBERT, provided that more labeled data is collected.
  • The opening puzzle is revealed as Dutch versus Somali. Dutch has about 29M speakers, 69 LDC/ELRA items and 2M Wikipedia articles. Somali has about 18M speakers, 2 items and 5.5K articles.
  • LREC and workshops cover the widest range of language classes, and newer conferences are more inclusive than older ones. Some typological features, such as 144E, are common in low-resource languages but rare in high-resource ones.
  • An author–language MRR analysis finds that dedicated research communities exist for Japanese, Mandarin, Turkish and Hindi (MRR(10) > 0.75). Burmese (0.02), Igbo (0.13) and Javanese (0.23) get little attention despite having millions of speakers. The authors recommend adding diversity and inclusion questions to submission and reviewer forms.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

758 papers of our library cite it (of the 140673 whose bibliographies we read); 32 of them have a page here, 726 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2004.09095 · arXiv 2004.09095

@misc{joshi2020state,
  title = {{The State and Fate of Linguistic Diversity and Inclusion in the NLP World}},
  author = {Joshi, Pratik and Santy, Sebastin and Budhiraja, Amar and Bali, Kalika and Choudhury, Monojit},
  year = {2020},
  doi = {10.48550/arxiv.2004.09095},
  eprint = {2004.09095},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2004.09095}
}

References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roee Aharoni, Melvin Johnson, and Orhan Firat. 2019. Massively multilingual neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3874–3884… find our card
  2. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610. find our card
  3. Emily M. Bender. 2011. On achieving and evaluating language-independence in NLP. In Vol 6: Interaction of Linguistics and Computational Linguistics. find
  4. Steven Bird, Robert Dale, Bonnie J Dorr, Bryan Gibson, Mark Thomas Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir R Radev, and Yee Fan Tan. 2008. The ACL anthology reference corpus: A reference dataset for bibliographic research in computational linguistics. LREC’08. find
  5. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 7059– 7069. Curran Associates, Inc. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  7. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  8. David Kamholz, Jonathan Pool, and Susan Colowick. 2014. PanLex: Building a resource for panlingual lexical translation. In Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14), pages 3145– 3150, Reykjavik, Iceland. European Language Resources Association (… find
  9. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39, Vancouver. Association for Computational Linguistics. find
  10. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of machine learning research, 9(Nov):2579–2605. find
  11. Tomas Mikolov, Kai Chen, G. S. Corrado, and J. Dean. 2013. Efficient estimation of word representations in vector space. In ICLR. find
  12. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996– 5001, Florence, Italy. Association for Computational Linguistics. find our card
  13. Edoardo Maria Ponti, Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019. Modeling language variation and universals: A survey on typological linguistics for natural language processing. Computational Linguistics, 45(3):559–601. find
  14. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics. find our card
  15. Gary Simons and Steven Bird. 2003. The open language archives community: An infrastructure for distributed archiving of language resources. Computing Research Repository - CORR, 18:117–128. find