Elamur Research
☰
home / papers / 2020 / XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning

XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.emnlp-main.185.pdf

Summary

Research on commonsense reasoning has been almost entirely English-only. This limits NLP tools for other languages and builds an Anglocentric view of "typical" situations into models. XCOPA addresses this gap. It is a benchmark for causal commonsense reasoning, created by translating and re-annotating the validation and test portions of English COPA into 11 languages. Each language had one carefully chosen translator, who could adapt content to local culture.

The languages were picked through variety sampling rather than by data availability. To support this, the authors define simple indices of typological, genealogical and areal diversity. On these indices XCOPA scores higher than other multilingual NLU benchmarks.

The authors test XLM-R, MBERT and multilingual USE. Zero-shot transfer from English fine-tuning beats chance but trails the approach of translating test data into English. Lightweight post-hoc adaptation of XLM-R gives clear gains for two languages unseen during pretraining. This adaptation uses only small monolingual corpora and/or bilingual dictionaries. Masked "adversarial" variants of the data show that the task cannot be solved through spurious cues.

Key points

  • Languages: 11 languages from 11 families across 5 macro-areas: ET, HT, ID, IT, QU (Eastern Apurímac Quechua), SW, TA, TH, TR, VI, ZH. HT and QU were out-of-sample for the pretrained models.
  • Size: each language has 100 validation and 500 test instances, aligned with English COPA. Each instance has a premise, a cause/result question and two alternatives, framed as binary choice.
  • Annotation agreement: Fleiss' κ was 0.921 (validation) and 0.911 (test). About 2.1% and 2.4% of labels, respectively, differ from the majority label.
  • Diversity scores: the typology index uses 103 URIEL/WALS features. XCOPA scores 0.41 on typology, 1 on family and 1.67 on geography, the highest among compared datasets, with typology on par with TyDiQA.
  • Adaptation of unseen languages: XLM-R variants were tested with extra target-language MLM training and/or dictionary-based methods (XLM-R+T, +S+T, +T-REP, +D). Gains were largest when fine-tuning on the larger SIQA data. English COPA offers only 400 training instances. Data: github.com/cambridgeltl/xcopa.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

312 papers of our library cite it (of the 140673 whose bibliographies we read); 22 of them have a page here, 290 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.185 · arXiv 2005.00333

@inproceedings{ponti2020xcopa,
  title = {{XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning}},
  author = {Ponti, Edoardo Maria and Glavaš, Goran and Majewska, Olga and Liu, Qianchu and Vulić, Ivan and Korhonen, Anna},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.185},
  eprint = {2005.00333},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.185}
}

References

72 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Željko Agić and Ivan Vulić. 2019. JW300: A wide-coverage parallel corpus for low-resource languages. In Proceedings of ACL, pages 3204–3210. find our card
  2. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDiQA: A benchmark for information-seeking question answering in typologically diverse languages. Transactions of the ACL. find
  3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL, pages 8440–8451. find
  4. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of EMNLP, pages 2475–2485. find
  5. Pascal Amsili and Olga Seminck. 2017. A Google-proof collection of French Winograd schemas. In Proceedings of the 2nd Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2017), pages 24–29. find
  6. Ernest Davis and Gary Marcus. 2015. Commonsense reasoning and commonsense knowledge in Artificial Intelligence. Communications of the ACM, 58(9):92–103. find
  7. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020a. Translation artifacts in cross-lingual transfer learning. arXiv preprint arXiv:2004.04721. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186. find
  9. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020b. On the cross-lingual transferability of monolingual representations. In Proceedings of ACL, pages 4623–4637. find our card
  10. Jeremy Barnes, Roman Klinger, and Sabine Schulte im Walde. 2018. Bilingual sentiment embeddings: Joint projection of sentiment across languages. In Proceedings of ACL, pages 2483–2493. find
  11. Emily M. Bender. 2011. On achieving and evaluating language-independence in NLP. Linguistic Issues in Language Technology, 6(3):1–26. find
  12. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. In Proceedings of ICLR. find our card
  13. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: Reasoning about physical commonsense in natural language. In Proceedings of AAAI. find our card
  14. Jürgen Bohnemeyer, Nick Enfield, James Essegbey, and Sotaro Kita. 2011. The macro-event property: The segmentation of causal chains. The Mental Lexicon. find
  15. Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multilingual alignment of contextual word representations. In Proceedings of ICLR. find
  16. Matthew S. Dryer. 1989. Large linguistic areas and language sampling. Studies in Language, 13(2):257– 292. find
  17. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  18. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  19. Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. arXiv preprint arXiv:2004.06063. find our card
  20. Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. 2018. On the relation between linguistic typology and (limitations of) multilingual language modeling. In Proceedings of EMNLP, pages 316–327. find
  21. Goran Glavaš and Ivan Vulić. 2020. Is supervised syntactic parsing beneficial for language understanding? an empirical investigation. arXiv preprint arXiv:2008.06788. — [arXiv:2008.06788](https://arxiv.org/abs/2008.06788) find
  22. Andrew Gordon and Reid Swanson. 2009. Identifying personal stories in millions of weblog entries. In Third International Conference on Weblogs and Social Media, Data Challenge Workshop, San Jose, CA, volume 46. find
  23. Felix Hill, Roi Reichart, and Anna Korhonen. 2015. SimLex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics, 41(4):665–695. find our card
  24. Patrick Lewis, Barlas Oğuz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating cross-lingual extractive question answering. In Proceedings of ACL, pages 7315–7330. find our card
  25. Paul J. Hopper. 1979. Aspect and foregrounding in discourse. In Discourse and Syntax, pages 211–241. Brill. find
  26. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. In Proceedings of ICML. find our card
  27. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Bruce Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Rangan Majumder, and Ming Zhou. 2020. X… find our card
  28. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of EMNLP-IJCNLP, pages 2391–2401. find our card
  29. Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of EACL, pages 8–14. find our card
  30. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of ACL, pages 6282–6293. find our card
  31. Gabriela Melo, Vinicius Imaizumi, and Fábio Cozman. 2020. Esquemas de Winograd em português. In Proceedings of the 16th Encontro Nacional de In-teligência Artificial e Computacional (ENIAC 2020), pages 787–798. find
  32. David Kamholz, Jonathan Pool, and Susan Colowick. 2014. PanLex: Building a resource for panlingual lexical translation. In Proceedings of LREC, pages 3145–3150. find
  33. Matti Miestamo. 2004. Clausal negation: A typological study. Ph.D. thesis, University of Helsinki. find
  34. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. find
  35. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR. find
  36. Moshe Koppel and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the ACL-HLT, pages 1318–1326. find
  37. András Kornai. 2013. Digital language death. PloS One, 8(10):e77056. find
  38. Leora Morgenstern and Charles L. Ortiz. 2015. The Winograd Schema Challenge: Evaluating progress in commonsense reasoning. In Proceedings of AAAI, pages 4024–4026. find
  39. Timothy Niven and Hung-Yu Kao. 2019. Probing neural network comprehension of natural language arguments. In Proceedings of ACL, pages 4658–4664. find
  40. Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. 2018. MCScript: A novel dataset for assessing machine comprehension using script knowledge. In Proceedings of LREC. find
  41. Charles Sanders Peirce. 1960. Collected papers of Charles Sanders Peirce, volume 2. Harvard University Press. find
  42. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, pages 159–174. find
  43. Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An adapter-based framework for multi-task cross-lingual transfer. In Proceedings of EMNLP. find our card
  44. Anne Lauscher, Vinit Ravishankar, Ivan Vulić, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot cross-lingual transfer with multilingual transformers. arXiv preprint arXiv:2005.00633. — [arXiv:2005.00633](https://arxiv.org/abs/2005.00633) find
  45. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. find our card
  46. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd schema challenge. In Proceedings of the 13th International Conference on the Principles of Knowledge Representation and Reasoning. find
  47. Edoardo Maria Ponti, Helen O’Horan, Yevgeni Berzak, Ivan Vulić, Roi Reichart, Thierry Poibeau, Ekaterina Shutova, and Anna Korhonen. 2019a. Modeling language variation and universals: A survey on typological linguistics for natural language processing. Computational Linguistics, 45(3):559–601. find
  48. Edoardo Maria Ponti, Ivan Vulić, Ryan Cotterell, Roi Reichart, and Anna Korhonen. 2019b. Towards zero-shot language modeling. In Proceedings of EMNLP-IJCNLP, pages 2893–2903. find
  49. Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Explain yourself! Leveraging language models for commonsense reasoning. In Proceedings of ACL, pages 4932–4942. find our card
  50. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP, pages 2383–2392. find our card
  51. Hannah Rashkin, Maarten Sap, Emily Allaway, Noah A. Smith, and Yejin Choi. 2018. Event2Mind: Commonsense inference on events, intents, and reactions. In Proceedings of ACL, pages 463–473. find
  52. Jan Rijkhoff, Dik Bakker, Kees Hengeveld, and Peter Kahrel. 1993. A method of language sampling. Studies in Language, 17(1):169–203. find
  53. Jörg Tiedemann. 2015. Cross-lingual dependency parsing with Universal Dependencies and predicted PoS labels. In Proceedings of the Third International Conference on Dependency Linguistics (Depling 2015), pages 340–349. find
  54. Vered Volansky, Noam Ordan, and Shuly Wintner. 2015. On the features of translationese. Digital Scholarship in the Humanities, 30(1):98–118. find
  55. Ivan Vulić, Simon Baker, Edoardo Maria Ponti, Ulla Petti, Ira Leviant, Kelly Wing, Olga Majewska, Eden Bar, Matt Malone, Thierry Poibeau, Roi Reichart, and Anna Korhonen. 2020. Multi-SimLex: A large-scale evaluation of multilingual and cross-lingual lexical semantic similarity. Computational Lingui… find
  56. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL-HLT, pages 1112–1122. find
  57. Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In Proceedings of the 2011 AAAI Spring Symposium Series. find
  58. Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-Hsuan Sung, et al. 2019a. Multilingual universal sentence encoder for semantic retrieval. arXiv preprint arXiv:1907.04307. find our card
  59. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. WinoGrande: An adversarial Winograd Schema Challenge at scale. In Proceedings of AAAI. find
  60. Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019b. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In Proceedings of EMNLP, pages 3687–3692. find our card
  61. Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In Proceedings of EMNLP-IJCNLP, pages 4463–4473. find
  62. Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019a. From recognition to cognition: Visual commonsense reasoning. In Proceedings of CVPR, pages 6720–6731. find
  63. Holger Schwenk and Xian Li. 2018. A corpus for multilingual document classification in eight languages. In Proceedings of LREC. find
  64. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of EMNLP, pages 93–104. find
  65. Tomohide Shibata, Shotaro Kohama, and Sadao Kurohashi. 2015. 日本語 Winograd Schema Challenge の構築と分析. 言語処理学会第 21 回年次大会 論文集, pages 493–496. find
  66. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019b. HellaSwag: Can a machine really finish your sentence? In Proceedings of ACL, pages 4791–4800. find
  67. Yoav Shoham. 1990. Nonmonotonic reasoning and causation. Cognitive Science, 14(2):213–252. find
  68. Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint arXiv:1810.12885. find our card
  69. Murray Singer, Michael Halldorson, Jeffrey C Lear, and Peter Andrusiak. 1992. Validation of causal bridging inferences in discourse understanding. Journal of Memory and Language, 31(4):507–524. find
  70. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of NAACL-HLT, pages 4149–4158. find
  71. Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scrambling. In Proceedings of NAACL-HLT, pages 1298– 1308. find our card
  72. Jenny Thomas. 1983. Cross-cultural pragmatic failure. Applied linguistics, 4(2):91–112. find