Elamur Research
☰
home / papers / 2019 / A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation

A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N19-1176.pdf

Summary

Coreference datasets such as ONTONOTES and PRECO release a single adjudicated reading per mention, even though annotators often genuinely disagree about anaphora. This paper presents a corpus that preserves those competing readings. The data were gathered through Phrase Detectives, a game-with-a-purpose (GWAP) in which players either annotate a highlighted markable or vote on interpretations that others proposed. The resource covers roughly 108,000 markables, each with about 20 judgments. That makes it the largest crowdsourced coreference corpus and the largest NLP dataset built with a GWAP.

Its annotation scheme goes beyond mainstream corpora by marking singletons, expletives, predicative NPs and split-antecedent plurals. To turn the raw judgments into training material, the authors aggregate them with the probabilistic MPA model. MPA produces silver labels with probabilities and may assign several labels, or none, to a markable. A state-of-the-art resolver trained on these labels performs well, including on non-referring expressions. The corpus therefore serves standard coreference work and also research on ambiguity that does not assume a single gold standard.

Key points

  • Scale: over 2.2M judgments in total, averaging 20.6 per markable and reaching 90 for some markables. The ~108K markables amount to about 55% of the markable count in ONTONOTES.
  • Silver-label training: a resolver trained on the MPA silver labels scored 3 CONLL points above the baseline when singletons are excluded. Including singletons in training added about 1 point even for non-singleton clusters. For non-referring mentions it reached F1 54.6% overall, 72.3% on expletives and 48.7% on predicative NPs.
  • MPA ambiguity estimate: using a 0.5 probability threshold, MPA gives exactly one interpretation to 93.4% of markables in PD gold and 94% in PD silver.
  • Expert analysis: specialists examined two documents, Little Red Cap and Robber Bridegroom. About 9% of markables (80 of 865) were genuinely ambiguous, which is 12.6% of the disagreement cases. As a detector of genuine ambiguity, MPA reached only about 20% recall and just under 50% precision.
  • Distribution: the corpus is available from LDC and dali-ambiguity.org. It comes in XML with all judgments, and in CONLL and CRAC 18 formats with only the gold or silver labels.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

43 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 34 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n19-1176

@inproceedings{poesio2019crowdsourced,
  title = {{A Crowdsourced Corpus of Multiple Judgments and Disagreement on Anaphoric Interpretation}},
  author = {Poesio, Massimo and Chamberlain, Jon and Paun, Silviu and Yu, Juntao and Uma, Alexandra and Kruschwitz, Udo},
  booktitle = {Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)},
  year = {2019},
  doi = {10.18653/v1/n19-1176},
  url = {https://doi.org/10.18653/v1/n19-1176}
}

References

58 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Luis von Ahn. 2006. Games with a purpose. Computer, 39(6):92–94. find
  2. Lora Aroyo and Chris Welty. 2015. Truth is a lie: Crowd truth and the seven myths of human annotation. AI Magazine, 36(1):15–24. find our card
  3. Anders Björkelund and Jonas Kuhn. 2014. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Proceedings of the 52nd ACL, volume 1, pages 47– 57. find
  4. Johan Bos, Valerio Basile, Kilian Evang, Noortje J. Venhuizen, and Johannes Bjerva. 2017. The groningen meaning bank. In N. Ide and J. Pustejovsky, editors, The Handbook of Linguistic Annotation, chapter 18, pages 463–496. Springer. Jon Chamberlain, Massimo Poesio, and Udo Kruschwitz. phrase detectives corpus. find
  5. Jon Chamberlain, Massimo Poesio, and Udo Kruschwitz. 2008. Phrase Detectives: A web-based collaborative annotation game. In Proceedings of I-Semantics 2008. find
  6. Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A large-scale dataset in preschool vocabulary for coreference resolution. In Proceedings of EMNLP, pages 172–181, Brussels, Belgium. find our card
  7. Marie-Catherine De Marneffe, Marta Recasens, and Christopher Potts. 2015. Modeling the lifespan of discourse entities with application to coreference resolution. Journal of Artificial Intelligence Research, 52(1):445–475. find
  8. Kees van Deemter and Rodger Kibble. 2000. On coreferring: Coreference in MUC and related annotation schemes. Computational Linguistics, 26(4):629– 637. Squib. find our card
  9. George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie Strassel, and Ralph M Weischedel. 2004. The automatic content extraction (ace) program-tasks, data, and evaluation. In Proceedings of LREC, volume 2, page 1. find
  10. Eraldo R. Fernandes, Cı́cero N. dos Santos, and Ruy L. Milidiú. 2014. Latent trees for coreference resolution. Computational Linguistics, 40(4):801–835. find
  11. Karen Fort, Bruno Guillaume, and H. Chastant. 2014. Creating Zombilingo, a game with a purpose for dependency syntax annotation. In Proceedings of the 1st International Workshop on Gamification for Information Retrieval (GamifIR’14), pages 2–6. ACM. find
  12. Ralph Grishman and Beth Sundheim. 1995. Design of the muc-6 evaluation. In Proceedings of the 6th Conference on Message Understanding, pages 1–11. Association for Computational Linguistics. find
  13. Nancy Chinchor. 1998. Overview of MUC-7. In Proceedings of the Seventh Message Understanding Conference (MUC-7). find
  14. Anupam Guha, Mohit Iyyer, Danny Bouman, and Jordan Boyd-Graber. 2015. Removing the training wheels: A coreference dataset that entertains humans and challenges computers. In Proceedings of NAACL. find
  15. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of the ACL. find our card
  16. Liane Guillou and Christian Hardmeier. 2016. Protest: A test suite for evaluating pronouns in machine translation. In Proceedings of LREC, Paris, France. find
  17. Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of EMNLP. find our card
  18. Erhard W. Hinrichs, Sandra übler, and Karin Naumann. 2005. A unified representation for morphological, syntactic, semantic and referential annotations. In Proc. of the ACL Workshop on Frontiers in Corpus Annotation II: Pie in the Sky, Ann Arbor, Michigan. find
  19. Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the ACL. find our card
  20. Daniel Clery. 2011. Galaxy evolution. Galaxy Zoo volunteers share pain and glory of research. Science, 333(6039):173–5. find
  21. Seth Cooper, Firsas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popovic, and the Foldit Players. 2010. Predicting protein structures with a multiplayer online game. Nature, 466:756– 760. find
  22. Alexander P. Dawid and Allan M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28(1):20–28. find
  23. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of NAACL, pages 1120–1130. find our card
  24. Jeff Howe. 2008. Crowdsourcing: Why the power of the crowd is driving the future of business. Crown Publishing Group. find
  25. Alan Joubert and Mathieu Lafourcade. 2008. Jeuxdemots : Un prototype ludique pour l’émergence de relations entre termes. In Proceedings of JADT. find
  26. Kian Kenyon-Dean, Eisha Ahmed, Jeremy Georges-Filteau, Christopher Glasz, Barleen Kaur, Auguste Lalande, Shruti Bhanderi, Robert Belfer, Nirmal Kanagasabai, Roman Sarrazingendron, Rohit Verma, and Derek Ruths. 2018. Sentiment analysis: it’s complicated! In Proceedings of NAACL, pages 1886–1895. ACL. find
  27. Olga Krasavina and Christian Chiarcos. 2007. The Potsdam coreference scheme. In Proceedings of the 1st Linguistic Annotation Workshop, pages 156– 163. find
  28. Mathieu Lafourcade, Alain Joubert, and Nathalie Le Brun. 2015. Games with a Purpose (GWAPs). Wiley. find
  29. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In Proceedings of CONLL: Shared Task, pages 28–34, Stroudsburg, PA, USA. find our card
  30. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of EMNLP. find our card
  31. Kenton Lee, Luheng He, and Luke S. Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of ACL. find our card
  32. Chris Madge, Jon Chamberlain, Udo Kruschwitz, and Massimo Poesio. 2017. Experiment-driven development of a gwap for marking segments in text. In Proceedings of CHI PLAY, Amsterdam. find
  33. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics, 3:405–418. find our card
  34. A. Nedoluzhko, J. Mirokvský, and P. Pajas. 2009. The coding scheme for annotating extended nominal coreference and bridging anaphora in the prague dependency treebank. In Proceedings of the Linguistic Annotation Workshop, pages 108–111. find
  35. Rebecca J. Passonneau, Vikas Bhardwaj, Ansaf Salleb-Aouissi, and Nancy Ide. 2012. Multiplicity and word sense: evaluating and learning from multiply labeled word sense annotations. Language Resources and Evaluation, 46(2):219–252. find
  36. Rebecca J. Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2:311–326. find
  37. Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018a. Comparing bayesian models of annotation. Transactions of the Association for Computational Linguistics. find
  38. Silviu Paun, Jon Chamberlain, Udo Kruschwitz, Juntao Yu, and Massimo Poesio. 2018b. A probabilistic annotation model for crowdsourcing coreference. In Proceedings of EMNLP, pages 1926–1937, Brussels, Belgium. Association for Computational Linguistics. find
  39. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL. find
  40. Slav Petrov and Dan Klein. 2007. Improved inference for unlexicalized parsing. In Proceedings of HLT-NAACL. find
  41. Barbara Plank, Dirk Hovy, and Anders Sogaard. 2014. Linguistically debatable or just plain wrong? In Proceedings of EACL. find
  42. Massimo Poesio and Ron Artstein. 2005a. Annotating (anaphoric) ambiguity. In Proc. of the Corpus Linguistics Conference, Birmingham. find
  43. Massimo Poesio and Ron Artstein. 2005b. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of the ACL Workshop on Frontiers in Corpus Annotation, pages 76–83. find
  44. Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Livio Robaldo, and Luca Ducceschi. 2013. Phrase Detectives: Utilizing collective intelligence for internet-scale language resource creation. ACM Transactions on Interactive Intelligent Systems, 3(1):1–44. find
  45. Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolution with the arrau corpus. In Proc. of the NAACL Worskhop on Computational Models of Reference, Anaphora and Coreference (CRAC), pages 11–22, New Orleans. find
  46. Massimo Poesio, Sameer Pradhan, Marta Recasens, Kepa Rodriguez, and Yannick Versley. 2016. Annotated corpora and annotation tools. In M. Poesio, R. Stuckardt, and Y. Versley, editors, Anaphora Resolution: Algorithms, Resources and Applications, chapter 4. Springer. find
  47. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea. find our card
  48. Nguyen Quoc Viet Hung, Nguyen Thanh Tam, Lam Ngoc Tran, and Karl Aberer. 2013. An evaluation of aggregation techniques in crowdsourcing. In Web Information Systems Engineering – WISE 2013, pages 1–15, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  49. Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bo-goni, and Linda Moy. 2010. Learning from crowds. Journal of Machine Learning Research, 11:1297– 1322. find
  50. Marta Recasens, Ed Hovy, and M. Antònia Martı́. 2011. Identity, non-identity, and near-identity: Addressing the complexity of coreference. Lingua, 121(6):1138–1152. find
  51. Marta Recasens and M. Antònia Martı́. 2010. AnCora-CO: Coreferentially annotated corpora for Spanish and Catalan. Language Resources and Evaluation, 44(4):315–345. find
  52. Marta Recasens, M. Antonia Martı́, and Constantin Orasan. 2012. Annotating near-identity from coreference disagreements. In Proceedings of LREC. find
  53. Aashish Sheshadri and Matthew Lease. 2013. SQUARE: A benchmark for research on computing crowd consensus. In Proceedings of the 1st AAAI Conference on Human Computation (HCOMP), pages 156–164. find
  54. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP, pages 254–263. Olga Uryupina, Ron Artstein, Antonella Bristot, Federica Cavicchio, Francesca Delogu, Kepa J. Rodriguez, and Massimo Poesio. To Appear. Annotating a broad range of anaphoric phenomena, in a variety of genres: the ARRAU corpus. Journal of Natural Language Engineering. find
  55. Noortje Venhuizen, Valerio Basile, Kilian Evang, and Johan Bos. 2013. Gamification for word sense labeling. In Proceedings of the 10th International Conference on Computational Semantics (IWCS’13). find
  56. Yannick Versley. 2008. Vagueness and referential ambiguity in a large-scale annotated corpus. Research on Language and Computation, 6:333–353. find
  57. Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. 2009. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22, pages 2035–2043. Curran Associates, Inc. find
  58. Billy T. M. Wong and Sophia Y. M. Lee. 2013. Annotating legitimate disagreement in corpus construction. In Proceedings of IJCNLP, pages 51–57, Nagoya, Japan. find