Elamur Research
☰
home / papers / 2020 / XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation

XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.484.pdf

Summary

Cross-lingual pre-trained models aim to transfer task skills from a data-rich language such as English to languages with little or no labeled data. Existing benchmarks fall short: GLUE is English-only and covers only understanding, and the concurrent XTREME also omits generation. XGLUE addresses this with 11 tasks across 19 languages, in three groups: single-input understanding, pair-input understanding and generation. Training labels are provided only in English, so models must transfer zero-shot to the other languages' test sets. Five tasks reuse existing datasets. The six new ones come from real Bing and Microsoft News scenarios (search, ads, news). The benchmark also ships two pre-training corpora. As a baseline, the authors extend Unicoder. For understanding, it is pre-trained with MLM and TLM. For generation, it becomes an encoder-decoder trained with xDAE or xFNP. They compare it with base-size M-BERT, XLM and XLM-R. Fine-tuning on several languages together clearly beats English-only fine-tuning, while multi-task fine-tuning leaves results essentially unchanged. XGLUE thus gives a practical, shared yardstick for both multilingual understanding and generation.

Key points

  • Tasks and data sources:
    • Reused datasets: NER (CoNLL-2002/2003), POS (UD Treebanks v2.5, 18 languages), MLQA, XNLI and PAWS-X.
    • New datasets: NC, QADSM, WPR, QAM, QG and NTG, built from Bing and MSN.
    • Metrics: F1, ACC, nDCG and BLEU-4.
  • Pre-training corpora:
    • Small Corpus: 101G of Wikipedia text in 100 languages, plus 146G of English-centric bilingual pairs covering 27 languages.
    • Large Corpus: adds about 2,500G of filtered Common Crawl text in 89 languages.
  • XNLI fine-tuning: Multi-language fine-tuning raises the XNLI average compared with English-pivot fine-tuning.
    • Unicoder LC: from 75.3 to 79.4.
    • XLM-R base: from 74.2 to 79.1.
  • NTG fine-tuning: On NTG (BLEU-4), multi-language fine-tuning lifts the average sharply.
    • Unicoder xDAE SC: from 9.6 to 22.8.
    • Unicoder xFNP SC: from 10.7 to 16.3.
  • Abstractive summarization: In zero-shot evaluation on fr and zh, Unicoder xDAE SC scores 40.1 average ROUGE-L, compared with 37.6 for XNLG. The authors credit this to their fourth text-noising strategy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

135 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 129 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.484

@inproceedings{liang2020xglue,
  title = {{XGLUE: A New Benchmark Dataset for Cross-lingual Pre-training, Understanding and Generation}},
  author = {Liang, Yaobo and Duan, Nan and Gong, Yeyun and Wu, Ning and Guo, Fenfei and Qi, Weizhen and Gong, Ming and Shou, Linjun and Jiang, Daxin and Cao, Guihong and Fan, Xiaodong and Zhang, Ruofei and Agrawal, Rahul and Cui, Edward and Wei, Sining and Bharti, Taroon and Qiao, Ying and Chen, Jiun-Hung and Wu, Winnie and Liu, Shuguang and Yang, Fan and Campos, Daniel and Majumder, Rangan and Zhou, Ming},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.484},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.484}
}

References

24 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019a. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461. find our card
  2. Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Uniter: Learning universal image-text representations. arXiv. find
  3. Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2019b. Mlqa: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475. find our card
  4. Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, Xian-Ling Mao, and Heyan Huang. 2019. Cross-lingual natural language generation via pre-training. In AAAI. find
  5. Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Zhou Zhou. 2020. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. AAAI. find
  6. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv. find
  7. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  8. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In NeurIPS. find
  9. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. NeurIPS. find our card
  10. Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. arXiv preprint arXiv:1809.05053. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL. find
  12. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. NeurIPS. find our card
  13. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. arXiv. find
  14. Erik Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142–147. find
  15. Tjong Kim Sang. 2002. Ef: Introduction to the conll-2002 shared task. In Proceedings of the 6th Conference on Natural Language Learning. find
  16. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, and Armand Joulin. 2019. Ccmatrix: Mining billions of high-quality parallel sentences on the web. arXiv. find our card
  17. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. ICLR. find our card
  18. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzman, Armand Joulin, and Edouard Grave. 2019. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359. find our card
  19. Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, and Ming Zhou. 2020. Prophetnet: Predicting future ngram for sequence-to-sequence pre-training. arXiv preprint arXiv:2001.04063. — [arXiv:2001.04063](https://arxiv.org/abs/2001.04063) find
  20. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019a. Paws-x: A cross-lingual adversarial dataset for paraphrase identification. arXiv preprint arXiv:1908.11828. find our card
  21. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019b. Xlnet: Generalized autoregressive pretraining for language understanding. NeurIPS. find our card
  22. Daniel Zeman, Joakim Nivre, Mitchell Abrams, Noëmi Aepli, Željko Agić, Lars Ahrenberg, Gabrielė Alek-sandravičiūtė, Lene Antonsen, Katya Aplonova, Maria Jesus Aranzabe, Gashaw Arutie, Masayuki Asahara, Luma Ateyah, Mohammed Attia, Aitziber Atutxa, Liesbeth Augustinus, Elena Badmaeva, Miguel B… find
  23. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. Paws: Paraphrase adversaries from word scrambling. arXiv preprint arXiv:1904.01130. find our card
  24. Luowei Zhou, Hamid Palangi, Lei Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. 2020. Unified vision-language pre-training for image captioning and vqa. AAAI. find