Elamur Research
☰
home / papers / 2020 / XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization

XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2003.11080

Summary

Multi-task benchmarks like GLUE and SuperGLUE pushed English NLP forward. Multilingual encoders, by contrast, were usually tested on only a few tasks, mostly translation and classification, and on closely related languages. XTREME addresses this by combining nine existing datasets into one benchmark. It covers 40 typologically diverse languages from 12 families and four task types: sentence classification, structured prediction, extractive QA and sentence retrieval. The evaluation is zero-shot cross-lingual transfer: models are fine-tuned only on English labelled data and then tested in every target language. The tasks were chosen to be hard and to test different levels of meaning. They are also cheap to train (one GPU, under a day) and freely licensed. Machine-translated pseudo test sets extend the NLI and QA checks to all 40 languages. Baselines include mBERT, XLM, XLM-R and translation-based pipelines. Models close to human level in English lose a lot of accuracy in other languages, especially on syntactic and retrieval tasks and on Sino-Tibetan, Japonic, Koreanic and Niger-Congo languages. With a public leaderboard and released code, XTREME gives multilingual models a shared reference point.

Key points

  • The nine tasks are XNLI and PAWS-X (accuracy), POS tagging on Universal Dependencies v2.5 treebanks with 17 universal tags, NER (F1), XQuAD, MLQA and TyDiQA-GoldP (span F1/EM), and BUCC and Tatoeba (retrieval).
  • Training data is English only. For example, MultiNLI (about 393k pairs) is used for XNLI, and SQuAD v1.1 for XQuAD and MLQA.
  • mBERT's score in a language follows the size of that language's Wikipedia. The correlation is ρ ≈ 0.8 for most tasks but only ≈ 0.35 for structured prediction.
  • Errors point to weak syntactic transfer. On POS tag trigrams not seen in training, mBERT scores 63.0 in English but 12.1 on average in other languages. On XNLI, only 71.8% of the examples it gets right in English are also right in other languages.
  • The benchmark is at sites.research.google/xtreme and the code at github.com/google-research/xtreme.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

273 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 254 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2003.11080 · arXiv 2003.11080

@misc{hu2020xtreme,
  title = {{XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization}},
  author = {Hu, Junjie and Ruder, Sebastian and Siddhant, Aditya and Neubig, Graham and Firat, Orhan and Johnson, Melvin},
  year = {2020},
  doi = {10.48550/arxiv.2003.11080},
  eprint = {2003.11080},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.11080}
}

References

64 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agić, Ž. and Schluter, N. Baselines and test data for cross-lingual inference. In Proceedings of LREC 2018, 2018. find
  2. Aharoni, R., Johnson, M., and Firat, O. Massively Multilingual Neural Machine Translation. In Proceedings of NAACL 2019, 2019. find our card
  3. Arivazhagan, N., Bapna, A., Firat, O., Lepikhin, D., Johnson, M., Krikun, M., Chen, M. X., Cao, Y., Foster, G., Cherry, C., Macherey, W., Chen, Z., and Wu, Y. Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges. arXiv preprint arXiv:1907.05019, 2019. find our card
  4. Artetxe, M. and Schwenk, H. Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond. Transactions of the ACL 2019, 2019. find our card
  5. Artetxe, M., Labaka, G., and Agirre, E. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of ACL 2017, pp. 451–462, 2017. find our card
  6. Artetxe, M., Labaka, G., and Agirre, E. A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings. In Proceedings of ACL 2018, pp. 789–798, 2018. find our card
  7. Artetxe, M., Ruder, S., and Yogatama, D. On the Cross-lingual Transferability of Monolingual Representations. In Proceedings of ACL 2020, 2020. find our card
  8. Barnes, J., Klinger, R., and Schulte im Walde, S. Bilingual sentiment embeddings: Joint projection of sentiment across languages. In Proceedings of ACL 2018, pp. 2483–2493, Melbourne, Australia, 2018. Association for Computational Linguistics. find
  9. Clark, J. H., Choi, E., Collins, M., Garrette, D., Kwiatkowski, T., Nikolaev, V., and Palomaki, J. TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages. In Transactions of the Association of Computational Linguistics, 2020. find our card
  10. Conneau, A., Lample, G., Ranzato, M., Denoyer, L., and Jégou, H. Word translation without parallel data. In Proceedings of ICLR 2018, 2018a. find
  11. Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S., Schwenk, H., and Stoyanov, V. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP 2018, pp. 2475–2485, 2018b. find
  12. Conneau, A., Khandelwal, K., Goyal, N., Chaudhary, V., Wenzek, G., Guzmán, F., Grave, E., Ott, M., Zettlemoyer, L., and Stoyanov, V. Unsupervised Cross-lingual Representation Learning at Scale. In Proceedings of ACL 2020, 2020. find
  13. Czarnowska, P., Ruder, S., Grave, E., Cotterell, R., and Copestake, A. Don’t forget the long tail! a comprehensive analysis of morphological generalization in bilingual lexicon induction. In Proceedings of EMNLP 2019, pp. 973–982, 2019. find
  14. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL 2019, 2019. find
  15. Eisenschlos, J., Ruder, S., Czapla, P., Kadras, M., Gugger, S., and Howard, J. MultiFiT: Efficient Multi-lingual Language Model Fine-tuning. In Proceedings of EMNLP 2019, 2019. find
  16. Eriguchi, A., Johnson, M., Firat, O., Kazawa, H., and Macherey, W. Zero-shot cross-lingual classification using multilingual neural machine translation. arXiv preprint arXiv:1809.04686, 2018. — [arXiv:1809.04686](https://arxiv.org/abs/1809.04686) find
  17. Faruqui, M. and Dyer, C. Improving vector space word representations using multilingual correlation. In Proceedings of EACL 2014, pp. 462–471, 2014. find
  18. Glavaš, G., Litschko, R., Ruder, S., and Vulić, I. How to (Properly) Evaluate Cross-Lingual Word Embeddings: On Strong Baselines, Comparative Analyses, and Some Misconceptions. In Proceedings of ACL 2019, 2019. find
  19. Gouws, S., Bengio, Y., and Corrado, G. BilBOWA: Fast bilingual distributed representations without word alignments. In Proceedings of ICML 2015, pp. 748–756, 2015. find
  20. Gururangan, S., Swayamdipta, S., Levy, O., Schwartz, R., Bowman, S. R., and Smith, N. A. Annotation Artifacts in Natural Language Inference Data. In Proceedings of NAACL-HLT 2018, 2018. find
  21. Guzmán, F., Chen, P.-J., Ott, M., Pino, J., Lample, G., Koehn, P., Chaudhary, V., and Ranzato, M. The FLoRes Evaluation Datasets for Low-Resource Machine Translation: Nepali-English and Sinhala-English. In Proceedings of EMNLP 2019, pp. 6100–6113, 2019. find our card
  22. Howard, J. and Ruder, S. Universal language model fine-tuning for text classification. In Proceedings of ACL 2018, pp. 328–339, 2018. find
  23. Hsu, T.-y., Liu, C.-l., and Lee, H.-y. Zero-shot Reading Comprehension by Cross-lingual Transfer Learning with Multi-lingual Language Representation Model. In Proceedings of EMNLP 2019, pp. 5935–5942, 2019. find
  24. Kementchedjhieva, Y., Hartmann, M., and Søgaard, A. Lost in evaluation: Misleading benchmarks for bilingual dictionary induction. In Proceedings of EMNLP 2019, pp. 3327–3332, 2019. find
  25. Klementiev, A., Titov, I., and Bhattarai, B. Inducing Crosslingual Distributed Representations of Words. In Proceedings of COLING 2012, 2012. find
  26. Koppel, M. and Ordan, N. Translationese and its dialects. In Proceedings of ACL 2011, pages=1318–1326, year=2011, organization=Association for Computational Linguistics. find
  27. Lample, G. and Conneau, A. Cross-lingual Language Model Pretraining. In Proceedings of NeurIPS 2019, 2019. find
  28. Lee, K., Yoon, K., Park, S., and Hwang, S. W. Semi-supervised training data generation for multilingual question answering. In Proceedings of LREC 2018, pp. 2758– 2762, 2018. find
  29. Lewis, P., Ouz, B., Rinott, R., Riedel, S., and Schwenk, H. MLQA: Evaluating Cross-lingual Extractive Question Answering. arXiv preprint arXiv:1910.07475, 2019. find our card
  30. Lin, Y.-H., Chen, C.-Y., Lee, J., Li, Z., Zhang, Y., Xia, M., Rijhwani, S., He, J., Zhang, Z., Ma, X., Anastasopoulos, A., Littell, P., and Neubig, G. Choosing Transfer Languages for Cross-Lingual Learning. In Proceedings of ACL 2019, 2019. find our card
  31. Luong, T., Pham, H., and Manning, C. D. Bilingual word representations with monolingual quality in mind. In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pp. 151–159, 2015. find our card
  32. Manning, C. D. Part-of-speech tagging from 97% to 100%: is it time for some linguistics? In International conference on intelligent text processing and computational linguistics, pp. 171–189. Springer, 2011. find our card
  33. McCann, B., Bradbury, J., Xiong, C., and Socher, R. Learned in translation: Contextualized word vectors. In Proceedings of NIPS 2017, pp. 6294–6305, 2017. find our card
  34. McDonald, R., Petrov, S., and Hall, K. Multi-source transfer of delexicalized dependency parsers. In Proceedings of EMNLP 2011, pp. 62–72, 2011. find our card
  35. Mikolov, T., Le, Q. V., and Sutskever, I. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168, 2013. find
  36. Mohammad, S. M., Salameh, M., and Kiritchenko, S. How translation alters sentiment. Journal of Artificial Intelligence Research, 55:95–130, 2016. find
  37. Nangia, N. and Bowman, S. R. Human vs. Muppet: A Conservative Estimate of Human Performance on the GLUE Benchmark. In Proceedings of ACL 2019, pp. 4566–4575, 2019. find
  38. Nivre, J., Abrams, M., Agić, Ž., Ahrenberg, L., Antonsen, L., Aranzabe, M. J., Arutie, G., Asahara, M., Ateyah, L., Attia, M., et al. Universal dependencies 2.2. 2018. find
  39. Pan, X., Zhang, B., May, J., Nothman, J., Knight, K., and Ji, H. Cross-lingual name tagging and linking for 282 languages. In Proceedings of ACL 2017, pp. 1946–1958, 2017. find
  40. Peters, M., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. In Proceedings of NAACL 2018, pp. 2227–2237, 2018. find
  41. Pires, T., Schlinger, E., and Garrette, D. How multilingual is Multilingual BERT? In Proceedings of ACL 2019, 2019. find our card
  42. Popović, M. chrF: character n-gram f-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 392–395, Lisbon, Portugal, 2015. find our card
  43. Rahimi, A., Li, Y., and Cohn, T. Massively Multilingual Transfer for NER. In Proceedings of ACL 2019, 2019. find our card
  44. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of EMNLP 2016, 2016. find our card
  45. Ruder, S., Vulić, I., and Søgaard, A. A Survey of Cross-lingual Word Embedding Models. Journal of Artificial Intelligence Research, 65:569–631, 2019. find
  46. Schuster, T., Ram, O., Barzilay, R., and Globerson, A. Cross-Lingual Alignment of Contextual Word Embeddings, with Applications to Zero-shot Dependency Parsing. In Proceedings of NAACL 2019, 2019. find
  47. Schwenk, H. and Li, X. A Corpus for Multilingual Document Classification in Eight Languages. In Proceedings of LREC 2018, 2018. find
  48. Siddhant, A., Johnson, M., Tsai, H., Arivazhagan, N., Riesa, J., Bapna, A., Firat, O., and Raman, K. Evaluating the Cross-Lingual Effectiveness of Massively Multilingual Neural Machine Translation. In Proceedings of AAAI 2020, 2020. find
  49. Smith, L., Giorgi, S., Solanki, R., Eichstaedt, J., Schwartz, H. A., Abdul-Mageed, M., Buffone, A., and Ungar, L. Does well-being translate on twitter? In Proceedings of EMNLP 2016, pp. 2042–2047, 2016. find
  50. Snyder, B., Naseem, T., and Barzilay, R. Unsupervised multilingual grammar induction. In Proceedings of ACL 2009, pp. 73–81, 2009. find
  51. Täckström, O., Das, D., Petrov, S., McDonald, R., and Nivre, J. Token and Type Constraints for Cross-Lingual Part-of-Speech Tagging. In Transactions of the Association for Computational Linguistics, 2013. find
  52. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I. Attention Is All You Need. In Proceedings of NIPS 2017, 2017. find
  53. Vulić, I., Glavaš, G., Reichart, R., and Korhonen, A. Do We Really Need Fully Unsupervised Cross-Lingual Embeddings? In Proceedings of EMNLP 2019, 2019. find
  54. Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. Superglue: A stickier benchmark for general-purpose language understanding systems. In Proceedings of NeurIPS 2019, 2019a. find our card
  55. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of ICLR 2019, 2019b. find our card
  56. Williams, A., Nangia, N., and Bowman, S. R. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of NAACL-HLT 2018, 2018. find
  57. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771, 2019. find our card
  58. Wu, S. and Dredze, M. Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT. In Proceedings of EMNLP 2019, 2019. find our card
  59. Yang, Y., Zhang, Y., Tar, C., and Baldridge, J. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of EMNLP 2019, pp. 3685–3690, 2019. find our card
  60. Zhang, M., Liu, Y., Luan, H., and Sun, M. Earth mover’s distance minimization for unsupervised bilingual lexicon induction. In Proceedings of EMNLP 2017, pp. 1934– 1945, 2017. find
  61. Zhang, Y., Baldridge, J., and He, L. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of NAACL 2019, pp. 1298–1308, 2019. find our card
  62. Zweigenbaum, P., Sharoff, S., and Rapp, R. Overview of the second bucc shared task: Spotting parallel sentences in comparable corpora. In Proceedings of the 10th Workshop on Building and Using Comparable Corpora, pp. 60–67, 2017. find
  63. Zweigenbaum, P., Sharoff, S., and Rapp, R. Overview of the third bucc shared task: Spotting parallel sentences in comparable corpora. In Proceedings of 11th Workshop on Building and Using Comparable Corpora, pp. 39–42, 2018. find
  64. A. Languages We show a detailed overview of languages in the cross-lingual benchmark including interesting typological differences in Table 5. Wikipedia information is taken from Wikipedia 14 and linguistic information from WALS Online 15 . XTREME includes members of the Afro-Asiatic, Austro-Asiatic… find