Elamur Research⌕
☰
home / papers / 2021 / Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution

Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution

Summary

Winograd schemas test whether a model can link an ambiguous pronoun to its antecedent using commonsense knowledge, but nearly all such data exists only in English. The authors build Wino-X, which aligns English schemas from the physical-domain part of WinoGrande with German, French and Russian versions. Translations come from the Google Translate API, and heuristic filters, supported by Stanza analysis, remove disfluent, cross-sentence or gender-undecidable items. The dataset has two formats. In MT-Wino-X, an English sentence containing "it" is paired with two translations whose pronouns agree with different antecedents, and NMT systems are scored by which translation receives lower perplexity. LM-Wino-X keeps WinoGrande's gap-filling format for multilingual language models (MLLMs). Transformer-BASE, transformer-BIG and mBART50 translators score essentially at chance. They show gender and positional biases and overlook the disambiguating trigger word; the authors quantify both problems and suggest mitigations. For MLLMs, fine-tuning on English schemas improves commonsense reasoning in other languages, sometimes as much as direct fine-tuning. Wino-X thus exposes a gap between translation quality and genuine commonsense coreference.

Key points

  • Source data: The physical subset of WinoGrande, with 19,260 samples (9,630 schemas). The social domain was excluded because its gaps cannot be replaced with "it".
  • MT-Wino-X size (schemas / samples):
    • EN-DE: 1,887 / 3,774
    • EN-FR: 1,499 / 2,988
    • EN-RU: 1,119 / 2,238
  • LM-Wino-X size (schemas / samples):
    • EN-DE: 2,917 / 5,834
    • EN-FR: 1,396 / 2,792
    • EN-RU: 743 / 1,486
  • Human check: Bilingual raters judged 100 samples per language pair.
    • Accuracy: 0.84 (DE), 0.88 (FR), 0.87 (RU).
    • Cohen's Kappa: 0.69, 0.75 and 0.77 respectively.
  • NMT results: Accuracies of fairseq BASE/BIG and mBART50 range from 0.496 to 0.511 across all pairs. Gender and positional biases, measured as |RBC|, are statistically significant, reaching up to 0.48 for gender bias (BIG, EN-RU). Data and code are available at github.com/demelin/Wino-X.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

22 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 19 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.670

@inproceedings{emelin2021wino,
  title = {{Wino-X: Multilingual Winograd Schemas for Commonsense Reasoning and Coreference Resolution}},
  author = {Emelin, Denis and Sennrich, Rico},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.670},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.670}
}

References

46 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pascal Amsili and Olga Seminck. 2017. A google-proof collection of french winograd schemas. In Proceedings of the 2nd Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2017), pages 24–29. find
  2. Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. 2018. Evaluating discourse phenomena in neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Paper… find our card
  3. Emily M Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
  4. Timothée Bernard and Ting Han. 2020. Mandarino-grad: A chinese collection of winograd schemas. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 21–26. find
  5. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2019. Abductive commonsense reasoning. arXiv preprint arXiv:1908.05739. find our card
  6. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  7. Jacob Cohen. 2013. Statistical power analysis for the behavioral sciences. Academic press. find
  8. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Édouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association fo… find
  9. Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2475–2485. find
  10. Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, et al. 2020. Evaluating models’ local decision boundaries via contrast sets. In Proceedings of the 2020 Conference on Empirical Methods in Natural Languag… find our card
  11. Yvette Graham, Barry Haddow, and Philipp Koehn. 2020. Translationese in machine translation evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 72–81. find
  12. Liane Guillou and Christian Hardmeier. 2016. Protest: A test suite for evaluating pronouns in machine translation. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 636–643. find
  13. Jie He, Tao Wang, Deyi Xiong, and Qun Liu. 2020. The box is in the pen: Evaluating commonsense reasoning in neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 3662–3672. find
  14. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalisation. In International Conference on Machine Learning, pages 4411–4421. PMLR. find
  15. Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Langua… find our card
  16. Nicos Isaak and Loizos Michael. 2019. Winoflexi: A crowdsourcing platform for the development of winograd schemas. In Australasian Joint Conference on Artificial Intelligence, pages 289–302. Springer. find
  17. Vid Kocijan, Thomas Lukasiewicz, Ernest Davis, Gary Marcus, and Leora Morgenstern. 2020. A review of winograd schema challenge datasets and approaches. arXiv preprint arXiv:2004.13831. find
  18. Edward E Cureton. 1956. Rank-biserial correlation. Psychometrika, 21(3):287–290. find
  19. Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. Citeseer. find
  20. Denis Emelin, Ivan Titov, and Rico Sennrich. 2020. Towards detecting and exploiting disambiguation biases in neural machine translation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7635–7653. find
  21. Jierui Li, Lemao Liu, Huayang Li, Guanlin Li, Guoping Huang, and Shuming Shi. 2020. Evaluating explanation methods for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 365– 375. find
  22. Xintong Li, Guanlin Li, Lemao Liu, Max Meng, and Shuming Shi. 2019. On the word alignment from neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1293–1303. find
  23. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732–8740. find
  24. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, et al. 2020. Xglue: A new benchmark datasetfor cross-lingual pre-training, understanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Languag… find
  25. Julian Salazar, Davis Liang, Toan Q Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712. find our card
  26. Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao, and Xiang Ren. 2021. Common sense beyond english: Evaluating and improving multilingual language models for common sense reasoning. In To appear. find
  27. Henry B Mann and Donald R Whitney. 1947. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pages 50–60. find
  28. Robert E McGrath and Gregory J Meyer. 2006. When effect sizes disagree: the case of r and d. Psychological methods, 11(4):386. find
  29. Gabriela Melo, Vinicius Imaizumi, and Fábio Cozman. 2019. Winograd schemas in portuguese. In Anais do XVI Encontro Nacional de Inteligência Artificial e Computacional, pages 787–798. SBC. find
  30. Mathias Müller, Annette Rios, Elena Voita, and Rico Sennrich. 2018. A large-scale test set for the evaluation of context-aware pronoun translation in neural machine translation. In WMT. find our card
  31. Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. 2019. Facebook fair’s wmt19 news translation task submission. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 314–319. find our card
  32. Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 1–9. find our card
  33. Matt Post. 2018. A call for clarity in reporting bleu scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191. find our card
  34. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D Manning. 2020. Stanza: A python natural language processing toolkit for many human languages. arXiv preprint arXiv:2003.07082. find our card
  35. Annette Rios, Laura Mascarell, and Rico Sennrich. 2017. Improving word sense disambiguation in neural machine translation with sense embeddings. In Proceedings of the Second Conference on Machine Translation, pages 11–19. find our card
  36. Danielle Saunders and Bill Byrne. 2020. Reducing gender bias in neural machine translation as a domain adaptation problem. arXiv preprint arXiv:2004.04498. find
  37. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725. find
  38. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31. find
  39. Gabriel Stanovsky, Noah A Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684. find our card
  40. Dario Stojanovski, Benno Krojer, Denis Peskov, and Alexander Fraser. 2020. Contracat: Contrastive coreference analytical templates for machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4732– 4749. find
  41. Yuqing Tang, Chau Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensible multilingual pretraining and finetuning. arXiv preprint arXiv:2008.00401. find
  42. Raphael Vallat. 2018. Pingouin: statistics in python. The Journal of Open Source Software, 3(31):1026. find
  43. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. arXiv preprint arXiv:1706.03762. find
  44. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. arXiv preprint arXiv:1905.00537. find our card
  45. Terry Winograd. 1972. Understanding natural language. Cognitive psychology, 3(1):1–191. find
  46. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/emelin-2021-wino-x-multilingual-winograd-schemas/