Elamur Research
☰
home / papers / 2012 / Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge

Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/D12-1071.pdf

Summary

This paper looks at a hard kind of pronoun resolution. Each sentence has two clauses joined by a connective such as "because". The first clause names two people or things, and the second contains a pronoun that could match either one in gender, number and semantic class. Usual cues like Binding Constraints, string matching and syntactic salience don't help here. Each sentence also has a "twin" that differs only after the connective and points the pronoun to the other candidate. A restricted version of this task is the Winograd Schema Challenge, which Levesque proposed as an alternative to the Turing Test. The authors had 30 undergraduates write 941 twin-sentence pairs and split them 70/30 into training and test sets. Their system ranks the two candidates with an SVM ranker. Its features come from eight knowledge sources, including narrative chains, Google hit counts, FrameNet, polarity cues and lexical features. It beats leading resolvers such as Stanford, Reconcile and BART by about 18 accuracy points. The paper shows that world knowledge, not just syntax, can make progress on this deeper language-understanding problem.

Key points

  • Dataset: 941 twin-sentence pairs written and annotated by 30 students, each checked by another student. Every sentence has exactly two candidate antecedents. Topics range from news events to films to made-up scenes.
  • Model: a pairwise ranking model trained with Joachims' SVM light. The twin structure is not used in training or testing.
  • Feature groups: Narrative Chains (Chambers & Jurafsky), Google search-count queries, FrameNet, Heuristic Polarity, Learned Polarity (OpenFinder-based), Connective-Based Relation, Semantic Compatibility, and about 68K Lexical Features.
  • Results: the full system reaches about 73.1% adjusted accuracy. Without the Lexical Features it still reaches 67.4%, which also beats all baselines. In ablation, the Google and Lexical features help most, while FrameNet and Learned Polarity help least.
  • Error analysis: Google queries become sparse when they have to link facts across both clauses. Narrative chains only cover verb events, so they miss pairs like "angry" and "shout". The simple prior-polarity heuristics also cause errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution · Semantic role labeling

Cited by

106 papers of our library cite it (of the 140673 whose bibliographies we read); 21 of them have a page here, 85 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{rahman2012resolving,
  title = {{Resolving Complex Cases of Definite Pronouns: The Winograd Schema Challenge}},
  author = {Rahman, Altaf and Ng, Vincent},
  booktitle = {Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning},
  year = {2012},
  url = {https://aclanthology.org/D12-1071/}
}

References

43 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga. 1998. Coreference, Cross-Document Coreference, and Information Extraction Methodologies. Ph.D. thesis, Duke University. find
  2. Collin F. Baker, Charles J. Fillmore, and John B. Lowe. 1998. The Berkeley FrameNet project. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics, pages 86– 90. find our card
  3. Shane Bergsma and Dekang Lin. 2006. Bootstrapping path-based pronoun resolution. In Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, pages 33–40. find
  4. Volha Bryl, Claudio Guiliano, Luciano Serafini, and Kateryna Tymoshenko. 2010. Using background knowledge to support coreference resolution. In Proceedings of the 19th European Conference on Artificial Intelligence, pages 759–764. find
  5. Donna K. Byron. 2002. Resolving pronominal reference to abstract entities. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 80–87. find
  6. Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 787–797. find
  7. Eugene Charniak and Micha Elsner. 2009. EM works for pronoun anaphora resolution. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics, pages 148–156. find
  8. Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the 5th International Conference on Language Resources and Evaluation, pages 449–454. find our card
  9. Pascal Denis and Jason Baldridge. 2007. A ranking approach to pronoun resolution. In Proceedings of the Twentieth International Conference on Artificial Intelligence, pages 1588–1593. find
  10. Pascal Denis and Jason Baldridge. 2008. Specialized models and ranking for coreference resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, pages 660–669. find
  11. Aria Haghighi and Dan Klein. 2009. Simple coreference resolution with rich syntactic and semantic features. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1152–1161. find our card
  12. Graeme Hirst. 1981. Anaphora in Natural Language Understanding. Springer Verlag. find
  13. Ryu Iida, Kentaro Inui, Hiroya Takamura, and Yuji Matsumoto. 2003. Incorporating contextual cues in trainable models for coreference resolution. In Proceedings of the EACL Workshop on The Computational Treatment of Anaphora. find
  14. Joseph Irwin, Mamoru Komachi, and Yuji Matsumoto. 2011. Narrative schema as world knowledge for coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 86–92. find
  15. Thorsten Joachims. 2002. Optimizing search engines using clickthrough data. In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 133–142. find
  16. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 28–34. find our card
  17. Hector J. Levesque. 2011. The Winograd Schema Challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning. find
  18. Bing Liu. 2012. Sentiment Analysis and Opinion Mining. Morgan & Claypool Publishers. find
  19. Ruslan Mitkov, Richard Evans, and Constantin Orasan. 2002. A new, fully automatic version of Mitkov’s knowledge-poor pronoun resolution method. In Al. Gelbukh, editor, Computational Linguistics and Intelligent Text Processing, pages 169–187. Springer. find
  20. Natalia N. Modjeska, Katja Markert, and Malvina Nissim. 2003. Using the web in machine learning for other-anaphora resolution. In Proceedings of the 2003 Conference on Empirical Methods in Natural Language Processing, pages 176–183. find
  21. Christoph Müller. 2007. Resolving it, this, and that in unrestricted multi-party dialog. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 816–823. find
  22. Bo Pang and Lillian Lee. 2008. Opinion mining and sentiment analysis. Foundations and Trends in Information Retrieval 2(1–2):1–135. find
  23. Massimo Poesio and Mijail A. Kabadjov. 2004. A general-purpose, off-the-shelf anaphora resolution module: Implementation and preliminary evaluation. In Proceedings of the 4th International Conference on Language Resources and Evaluation, pages 663–666. find
  24. Massimo Poesio, Rahul Mehta, Axel Maroudas, and Janet Hitzeman. 2004. Learning to resolve bridging references. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, pages 143–150. find
  25. Massimo Poesio, David Day, Ron Artstein, Jason Duncan, Vladimir Eidelman, Claudio Giuliano, Rob Hall, Janet Hitzeman, Alan Jern, Mijail Kabadjov, Stanley Yong Wai Keong, Gideon Mann, Alessandro Moschitti, Simone Ponzetto, Jason Smith, Josef Steinberger, Michael Strube, Jian Su, Yannick Versley, Xiaofeng Yang, and Michael Wick. 2007. EL-ERFED: Final report of the research group on Exploiting Lexical and Encyclopedic Resources For Entity Disambiguation. Technical report, Summer Workshop on Language Engineering, Center for Language and Speech Processing, Johns Hopkins University, Baltimore, MD. find
  26. Simone Paolo Ponzetto and Michael Strube. 2006. Exploiting semantic role labeling, WordNet and Wikipedia for coreference resolution. In Proceedings of the Human Language Technology Conference and Conference of the North American Chapter of the Association for Computational Linguistics, pages 192– 199. find
  27. Simone Paolo Ponzetto and Michael Strube. 2007. Knowledge derived from Wikipedia for computing semantic relatedness. Journal of Artificial Intelligence Research, 30:181–212. find
  28. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 Shared Task: Modeling unrestricted coreference in OntoNotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–27. find our card
  29. Long Qiu, Min-Yen Kan, and Tat-Seng Chua. 2004. A public reference implementation of the RAP anaphora resolution algorithm. In Proceedings of the 4th International Conference on Language Resources and Evaluation, pages 291–294. find
  30. Karthik Raghunathan, Heeyoung Lee, Sudarshan Rangarajan, Nate Chambers, Mihai Surdeanu, Dan Jurafsky, and Christopher Manning. 2010. A multi-pass sieve for coreference resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 492–501. find our card
  31. Altaf Rahman and Vincent Ng. 2009. Supervised models for coreference resolution. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 968–977. find
  32. Altaf Rahman and Vincent Ng. 2011. Coreference resolution with world knowledge. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 814–824. find
  33. Roger C. Schank and Robert P. Abelson. 1977. Scripts, Plans, Goals, and Understanding. Lawrence Erlbaum. find
  34. Veselin Stoyanov, Claire Cardie, Nathan Gilbert, Ellen Riloff, David Buttler, and David Hysom. 2010. R EC - ONCILE : A coreference resolution research platform. In Proceedings of the ACL 2010 Conference Short Papers, pages 156–161. find
  35. Michael Strube and Christoph Müller. 2003. A machine learning approach to pronoun resolution in spoken dialogue. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 168–175. find
  36. Alan M. Turing. 1950. Computing machinery and intelligence. Mind, 59:433–460. find
  37. Olga Uryupina, Massimo Poesio, Claudio Giuliano, and Kateryna Tymoshenko. 2011. Disambiguation and filtering methods in using Web knowledge for coreference resolution. In Proceedings of the 24th International Florida Artificial Intelligence Research Society Conference, pages 317–322. find
  38. Yannick Versley, Simone Paolo Ponzetto, Massimo Poesio, Vladimir Eidelman, Alan Jern, Jason Smith, Xiaofeng Yang, and Alessandro Moschitti. 2008. BART: A modular toolkit for coreference resolution. In Proceedings of the ACL-08: HLT Demo Session, pages 9–12. find
  39. Theresa Wilson, Paul Hoffmann, Swapna Somasundaran, Jason Kessler, Janyce Wiebe, Yejin Choi, Claire Cardie, Ellen Riloff, and Siddharth Patwardhan. 2005a. Opinionfinder: A system for subjectivity analysis. In Proceedings of HLT/EMNLP 2005 Interactive Demonstrations, pages 34–35. find
  40. Theresa Wilson, Janyce M. Wiebe, and Paul Hoffmann. 2005b. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of the Joint Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 347–354. find
  41. Terry Winograd. 1972. Understanding Natural Language. Academic Press, Inc., New York. find
  42. Xiaofeng Yang, Guodong Zhou, Jian Su, and Chew Lim Tan. 2003. Coreference resolution using competitive learning approach. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 176–183. find
  43. Xiaofeng Yang, Jian Su, and Chew Lim Tan. 2005. Improving pronoun resolution using statistics-based semantic compatibility information. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 165–172. find