Elamur Research⌕
☰
home / papers / 2015 / Removing the Training Wheels: A Coreference Dataset that Entertains Humans and Challenges Computers

Removing the Training Wheels: A Coreference Dataset that Entertains Humans and Challenges Computers

Summary

Existing coreference benchmarks (MUC, ACE, CoNLL/OntoNotes) consist almost entirely of newswire. Journalistic style keeps references sparse and easy to resolve, so these benchmarks rarely test world knowledge or nested mentions. The authors propose quiz bowl questions as a harder domain. These trivia questions describe an answer entity through increasingly revealing indirect clues, which makes them dense with coreference.

They built a web annotation tool and recruited quiz bowl players preparing for a national tournament, motivating them with a leaderboard and prizes. They also added active learning to the pipeline so that the most informative documents were selected for annotation. The resulting corpus covers literature questions and is much denser in mentions and nested mentions than OntoNotes.

On this data, the widely used Berkeley coreference system performs worse than a simple pipeline: a CRF mention detector plus a pairwise logistic regression classifier using pretrained word embeddings. Error analysis shows that many cases depend on external world knowledge. The work argues that coreference research needs evaluation data beyond newswire.

Key points

  • The dataset contains 400 quiz bowl documents (averaging 5.2 sentences each), with 1,890 sentences, 50,347 tokens and 9,471 mentions. These mentions include 2,461 singletons, 7,010 anaphoric mentions and 2,194 nested mentions.
  • Mention density is about 3.7 per sentence, compared with 2.1 in the CoNLL 2011 / OntoNotes 4.0 data. Nested mentions occur at 1.16 per sentence, compared with 0.25 in OntoNotes newswire.
  • Annotation followed these rules:
    • Annotators tagged authors, characters, works and the answer entity.
    • A mention is the maximal span that a pronoun could replace.
    • Appositives are marked as separate mentions in the same cluster.
    • Some documents had author-provided gold labels so volunteers could check their work.
  • When both systems were trained on the quiz bowl data, the simple CRF + pairwise classifier beat the Berkeley system (Durrett and Klein, 2013). This suggests that mention-ranking models tuned for newswire do not transfer well to this domain.
  • The dataset, annotation guide and code are released at cs.umd.edu/~aguha/qbcoreference.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

17 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 9 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/n15-1117

@inproceedings{guha2015removing,
  title = {{Removing the Training Wheels: A Coreference Dataset that Entertains Humans and Challenges Computers}},
  author = {Guha, Anupam and Iyyer, Mohit and Bouman, Danny and Boyd-Graber, Jordan},
  booktitle = {Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2015},
  doi = {10.3115/v1/n15-1117},
  url = {https://doi.org/10.3115/v1/n15-1117}
}

References

42 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In International Language Resources and Evaluation. Citeseer. find our card
  2. Eric Bengtson and Dan Roth. 2008. Understanding the value of features for coreference resolution. In Proceedings of Emperical Methods in Natural Language Processing. Association for Computational Linguistics. find our card
  3. Anders Björkelund and Richárd Farkas. 2012. Data-driven multilingual coreference resolution using resolver stacking. In Conference on Computational Natural Language Learning. find our card
  4. Anders Björkelund and Jonas Kuhn. 2014. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Proceedings of the Association for Computational Linguistics. find our card
  5. A. Boyd, P. Stewart, and R. Alexander. 2008. Broadcast Journalism: Techniques of Radio and Television News. Taylor & Francis. find
  6. Jordan Boyd-Graber, Brianna Satinoff, He He, and Hal Daume III. 2012. Besting the quiz master: Crowdsourcing incremental classification games. In Proceedings of Emperical Methods in Natural Language Processing. find
  7. Marie-Catherine De Marneffe, Bill MacCartney, Christopher D Manning, et al. 2006. Generating typed dependency parses from phrase structure parses. In International Language Resources and Evaluation. find our card
  8. George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie Strassel, and Ralph M Weischedel. 2004. The automatic content extraction (ACE) program-tasks, data, and evaluation. In International Language Resources and Evaluation. find our card
  9. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of Emperical Methods in Natural Language Processing. find our card
  10. Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. Transactions of the Association for Computational Linguistics. find our card
  11. Eraldo Rezende Fernandes, Cı́cero Nogueira Dos Santos, and Ruy Luiz Milidiú. 2012. Latent structure perceptron with feature induction for unrestricted coreference resolution. In Proceedings of Emperical Methods in Natural Language Processing. find our card
  12. N. Goldstein and A. Press. 2004. The Associated Press Stylebook and Briefing on Media Law. Associated Press Stylebook and Briefing on Media Law. Basic Books. find
  13. Aria Haghighi and Dan Klein. 2010. Coreference resolution in a modular, entity-centered model. In Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  14. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. Ontonotes: the 90% solution. In Conference of the North American Chapter of the Association for Computational Linguistics. find
  15. Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daumé III. 2014. A neural network for factoid question answering over paragraphs. In Proceedings of Emperical Methods in Natural Language Processing. find
  16. Jonathan K Kummerfeld, Mohit Bansal, David Burkett, and Dan Klein. 2011. Mention detection: heuristics for the ontonotes annotations. In Conference on Computational Natural Language Learning. find
  17. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. find
  18. Florian Laws, Florian Heimerl, and Hinrich Schütze. 2012. Active learning for coreference resolution. In Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  19. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the conll-2011 shared task. In Conference on Computational Natural Language Learning. find our card
  20. Xiaoqiang Luo, Abe Ittycheriah, Hongyan Jing, Nanda Kambhatla, and Salim Roukos. 2004. A mention-synchronous coreference resolution algorithm based on the bell tree. In Proceedings of the Association for Computational Linguistics. find our card
  21. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Emperical Methods in Natural Language Processing. find our card
  22. Andrew Kachites McCallum. 2002. Mallet: A machine learning for language toolkit. http://mallet.cs.umass.edu. ↗
  23. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  24. Timothy A Miller, Dmitriy Dligach, and Guergana K Savova. 2012. Active learning for coreference resolution. In Proceedings of the 2012 Workshop on Biomedical Natural Language Processing. Proceedings of the Association for Computational Linguistics. find our card
  25. George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39–41. find our card
  26. MUC-6. 1995. Coreference task definition (v2.3, 8 sep 95). In Proceedings of the Sixth Message Understanding Conference (MUC-6). find
  27. MUC-7. 1997. Coreference task definition (v3.0, 13 jun 97). In Proceedings of the Seventh Message Understanding Conference (MUC-7). find
  28. Vincent Ng and Claire Cardie. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of the Association for Computational Linguistics. find our card
  29. Vincent Ng. 2010. Supervised noun phrase coreference research: The first fifteen years. In Proceedings of the Association for Computational Linguistics. find our card
  30. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of Emperical Methods in Natural Language Processing. find
  31. Thierry Poibeau and Leila Kosseim. 2001. Proper name extraction from non-journalistic texts. Language and computers, 37(1):144–157. find
  32. Simone Paolo Ponzetto and Michael Strube. 2006. Exploiting semantic role labeling, WordNet and Wikipedia for coreference resolution. In Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  33. Sameer S Pradhan, Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2007. Ontonotes: A unified relational semantic representation. International Journal of Semantic Computing, 1(04). find
  34. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 shared task: Modeling unrestricted coreference in Ontonotes. In Conference on Computational Natural Language Learning. find our card
  35. Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Conference on Computational Natural Language Learning. find our card
  36. Lev Ratinov and Dan Roth. 2012. Learning-based multi-sieve co-reference resolution with knowledge. In Proceedings of Emperical Methods in Natural Language Processing. find our card
  37. Burr Settles. 2010. Active learning literature survey. University of Wisconsin, Madison, 52:55–66. find our card
  38. Wee Meng Soon, Hwee Tou Ng, and Daniel Chung Yong Lim. 2001. A machine learning approach to coreference resolution of noun phrases. Computational linguistics, 27(4). find our card
  39. Veselin Stoyanov, Claire Cardie, Nathan Gilbert, Ellen Riloff, David Buttler, and David Hysom. 2010. Coreference resolution with reconcile. In Proceedings of the Association for Computational Linguistics. find our card
  40. Olga Uryupina. 2006. Coreference resolution with and without linguistic knowledge. In International Language Resources and Evaluation. find our card
  41. Yannick Versley, Simone Paolo Ponzetto, Massimo Poesio, Vladimir Eidelman, Alan Jern, Jason Smith, Xiaofeng Yang, and Alessandro Moschitti. 2008. Bart: A modular toolkit for coreference resolution. In Proceedings of the Association for Computational Linguistics. find our card
  42. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the conference on Message understanding, pages 45–52. find our card

↑ back to top

This page in Markdown →