Elamur Research⌕
☰
home / papers / 2019 / A Surprisingly Robust Trick for the Winograd Schema Challenge

A Surprisingly Robust Trick for the Winograd Schema Challenge

Paper · 2019 · Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu et al.

Summary

The paper addresses the Winograd Schema Challenge, a test in which a system must decide which of two candidate nouns an ambiguous pronoun refers to. The task is designed to require commonsense reasoning, and very little training data exists for it. The authors mask the pronoun and let a pre-trained language model (BERT-large, BERT-base or GPT) score each candidate by its masked-token probability. Fine-tuning combines a log-likelihood term with a margin-based ranking penalty. Two supervision sources are used. The first is the Definite Pronoun Resolution dataset (WSCR), an easier relative of WSC273. The second is MaskedWiki, a new set of 2.4M automatically built WSC-like items created from English Wikipedia by masking a repeated noun. Earlier work found little benefit in transferring from WSCR, yet here it improved every model tested. BERT trained on MaskedWiki and then WSCR reached 72.5% on WSC273 and 74.7% on WNLI. This exceeds the previous best results by 8.8 and 9.6 points and beats the WNLI majority baseline, which no earlier system had surpassed.

Key points

  • Best model (BERT_WIKI_WSCR): 72.5% on WSC273 and 74.7% on the WNLI test set. The prior state of the art was about 63.7% on WSC273, and the WNLI majority baseline is 65.1 (human performance 95.9).
  • MaskedWiki was built from the enwiki-20181201 dump using the Stanford POS tagger. About 130M candidate items were randomly downsampled to 2.4M. A manual check of 200 items found 45% hard, 45.5% easy, 8.5% unsolvable and 1% noise.
  • WSCR was used with 1322 training and 564 test examples, the test set serving for validation. Six items that overlap with WSC273 were removed beforehand.
  • Training settings were α = 20, β = 0.2, a learning rate of 5·10⁻⁶ with Adam, and batch size 64. The models ran one epoch on MaskedWiki and up to 30 epochs on WSCR.
  • MaskedWiki alone did not help BERT, but it did once WSCR fine-tuning followed. Keeping the paired "special word" examples in WSCR mattered: accuracy on WSC273 was 0.703 with pairs versus 0.663 without them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom104 works · 157,183 citations🎓Imperial College LondonLondon · United Kingdom48 works · 12,181 citations🔬The Alan Turing InstituteLondon · United Kingdom29 works · 4,368 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov and Thomas Lukasiewicz. 2019. A Surprisingly Robust Trick for the Winograd Schema Challenge. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1478

@inproceedings{kocijan2019surprisingly,
  title = {{A Surprisingly Robust Trick for the Winograd Schema Challenge}},
  author = {Vid Kocijan, Ana-Maria Cretu, Oana-Maria Camburu, Yordan Yordanov and Thomas Lukasiewicz},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1478},
  url = {https://doi.org/10.18653/v1/p19-1478}
}

References

19 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ernest Davis, Leora Morgenstern, and Charles L. Ortiz. 2017. The first Winograd Schema Challenge at IJCAI-16. AI Magazine, 38(3):97–98. find
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. Computing Research Repository, arXiv:1810.04805.
  3. Ali Emami, Noelia De La Cruz, Adam Trischler, Kaheer Suleman, and Jackie Chi Kit Cheung. 2018. A knowledge hunting framework for common sense reasoning. Computing Research Repository, arXiv:1810.01375. find
  4. Hany Hassan, Anthony Aue, Chang Chen, Vishal Chowdhary, Jonathan Clark, Christian Federmann, Xuedong Huang, Marcin Junczys-Dowmunt, William Lewis, Mu Li, Shujie Liu, Tie-Yan Liu, Renqian Luo, Arul Menezes, Tao Qin, Frank Seide, Xu Tan, Fei Tian, Lijun Wu, Shuangzhi Wu, Yingce Xia, Dongdong Zhang, Zh… — Achieving Human Parity on Automatic Chinese to English News Translation
  5. Jeremy Howard and Sebastian Ruder. 2018. Fine-tuned language models for text classification. Computing Research Repository, arXiv:1801.06146. find
  6. Rie Johnson and Tong Zhang. 2017. Deep pyramid convolutional neural networks for text categorization. In Proceedings of ACL, pages 562–570. ACL. find
  7. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2011. The Winograd Schema Challenge. AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, 46. find
  8. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In Proceedings of KR. AAAI Press. find
  9. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. Computing Research Repository, arXiv:1901.11504.
  10. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60.
  11. Leora Morgenstern, Ernest Davis, and Charles L. Ortiz. 2016. Planning, executing and evaluating the Winograd Schema Challenge. AI Magazine. find
  12. Nikita Nangia and Samuel R. Bowman. 2018. A conservative human baseline estimate for GLUE: People still (mostly) beat machines. find
  13. Juri Opitz and Anette Frank. 2018. Addressing the Winograd Schema Challenge as a sequence ranking task. In Proceedings of the First International Workshop on Language Cognition and Computational Models, pages 41–52. ACL. find
  14. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  15. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: The Winograd Schema Challenge. In Proceedings of EMNLP.
  16. Paul Trichelair, Ali Emami, Jackie Chi Kit Cheung, Adam Trischler, Kaheer Suleman, and Fernando Diaz. 2018. On the evaluation of common-sense reasoning in natural language understanding. Computing Research Repository, arXiv:1811.01778. — [arXiv:1811.01778](https://arxiv.org/abs/1811.01778) find
  17. T. H. Trinh and Q. V. Le. 2018. A Simple Method for Commonsense Reasoning. Computing Research repository, arXiv:1806.02847. find
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Computing Research Repository, arXiv:1706.03762.
  19. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of ICLR.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kocijan-2019-surprisingly-robust-trick-winograd-schema/