Elamur Research⌕
☰
home / papers / 2018 / A Probabilistic Annotation Model for Crowdsourcing Coreference

A Probabilistic Annotation Model for Crowdsourcing Coreference

Summary

Large coreference corpora are costly to build with expert annotators. Crowdsourcing is a cheaper option, but existing aggregation models assume a fixed label set shared across all items. In anaphoric annotation, the possible labels (antecedents and chains) differ in every document, so those models do not apply. The authors present MPA (Mention Pair Annotations), a Bayesian model for aggregating crowd judgments of anaphora. Each mention's annotations become yes/no decisions for every candidate label, using the binary relevance method. Each annotator gets a sensitivity and specificity per general class (e.g., DO, DN), and parameters are estimated with Variational Inference. The model picks the most probable label for each mention, and chains are formed by following the inferred links. Experiments used Phrase Detectives data and simulated datasets. MPA beat majority voting and the Stanford deterministic resolver. When used to train the Lee et al. (2017) system, its silver chains came closest to gold-chain training. This suggests crowd-built coreference corpora can approach expert quality.

Key points

  • Phrase Detectives contains 843 documents, about 1.2 million tokens and 392,741 markables. Of these, 563 documents (roughly 360,000 tokens) are currently annotated.
  • MPA follows the two-step mention pair framework: an unsupervised step infers coreferent pairs, and the pairs are then grouped into chains.
  • When Lee et al. (2017) was trained on MPA silver chains, the best score was only about 1 percentage point below training on gold chains.
  • Even plain majority-vote chains produced better training data than chains from the Stanford system (Lee et al., 2011).
  • On the standard Snow et al. (2008) benchmarks, MPA matched Bayesian Dawid–Skene and MACE: 93 on RTE, 94 on TEMP and 99 on WSD (accuracy, %). Majority vote scored 90, 93 and 99.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1218

@inproceedings{paun2018probabilistic,
  title = {{A Probabilistic Annotation Model for Crowdsourcing Coreference}},
  author = {Paun, Silviu and Chamberlain, Jon and Kruschwitz, Udo and Yu, Juntao and Poesio, Massimo},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1218},
  url = {https://doi.org/10.18653/v1/d18-1218}
}

References

52 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. An early version of this paper has been circulating since 2005 as “Kappa 3 = Alpha (or Beta)”. This version is still available from the ARRAU website. find
  2. Breck Baldwin and Thomas S. Morton. 1998. Dynamic coreference-based summarization. In Proceedings of the Third Conference on Empirical Methods for Natural Language Processing, pages 1– 6. find our card
  3. Anders Björkelund and Jonas Kuhn. 2014. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 47–57. find our card
  4. David M. Blei and Michael I. Jordan. 2006. Variational inference for Dirichlet process mixtures. Bayesian Analysis, 1(1):121–143. find
  5. David M. Blei, Alp Kucukelbir, and Jon D. McAuliffe. 2017. Variational inference: A review for statisticians. Journal of the American Statistical Association, 112(518):859–877. find
  6. Matthew D. Hoffman, David M. Blei, Chong Wang, and John Paisley. 2013. Stochastic variational inference. The Journal of Machine Learning Research, 14(1):1303–1347. find
  7. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet allocation. The Journal of Machine Learning Research, 3:993–1022. find our card
  8. Veronique Hoste. 2016. The mention-pair model. In Anaphora Resolution: Algorithms, Resources and Applications. Springer. find
  9. Johan Bos, Valerio Basile, Kilian Evang, Noortje J. Venhuizen, and Johannes Bjerva. 2017. The Groningen Meaning Bank. In The Handbook of Linguistic Annotation, chapter 18, pages 463–496. Springer. find
  10. Dirk Hovy, Taylor Berg-Kirkpatrick, Ashish Vaswani, and Eduard Hovy. 2013. Learning whom to trust with MACE. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1120–1130. find our card
  11. Bob Carpenter. 2008. Multilevel Bayesian models of categorical data annotation. Available at http://lingpipe-blog.com/lingpipe-white-papers. ↗
  12. Jon Chamberlain. 2016. Harnessing Collective Intelligence on Social Networks. Ph.D. thesis, University of Essex, School of Computer Science and Electronic Engineering. find
  13. Jon Chamberlain, Massimo Poesio, and Udo Kruschwitz. 2016. Phrase detectives corpus 1.0: Crowdsourced anaphoric coreference. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2016), Portoroz, Slovenia. find our card
  14. Nancy A. Chinchor. 1998. Overview of MUC-7/MET-2. In Proceedings of the 7 th Message Understanding Conference (MUC7). find
  15. Kevin Clark and Christopher D. Manning. 2016. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). find our card
  16. Alexander Philip Dawid and Allan M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28(1):20–28. find
  17. Paul Felt, Robbie Haertel, Eric K. Ringger, and Kevin D. Seppi. 2014. MOMRESP: A Bayesian model for multi-annotator document labeling. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2014), Reykjavik. find
  18. Ralph Grishman and Beth Sundheim. 1995. Design of the MUC-6 evaluation. In Proceedings of the 6th Conference on Message Understanding, MUC6 ’95, pages 1–11, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  19. Anupam Guha, Mohit Iyyer, Danny Bouman, and Jordan Boyd-Graber. 2015. Removing the training wheels: A coreference dataset that entertains humans and challenges computers. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Langu… find our card
  20. Kevin Humphreys, Robert Gaizauskas, and Saliha Azzam. Event coreference for information extraction. In Proceedings of a Workshop on Operational Factors in Practical, Robust Anaphora Resolution for Unrestricted Texts. find our card
  21. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, CONLL Sha… find our card
  22. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197. find our card
  23. Gjorgji Madjarov, Dragi Kocev, Dejan Gjorgjevikj, and SašO Deroski. 2012. An extensive experimental comparison of methods for multi-label learning. Pattern Recognition, 45(9):3084–3104. find
  24. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. find our card
  25. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association of Computational Linguistics, 3(1):405–418. find our card
  26. Thomas S. Morton. 1999. Using coreference for question answering. In Proceedings of the Workshop on Coreference and Its Applications, CorefApp ’99, pages 85–89, Stroudsburg, PA, USA. Association for Computational Linguistics. find our card
  27. Malvina Nissim, Shipra Dingare, Jean Carletta, and Mark Steedman. 2004. An annotation scheme for information status in dialogue. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2004). find
  28. Rebecca J. Passonneau. 2004. Computing reliability for coreference annotation. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2004), Lisbon. find
  29. Rebecca J. Passonneau and Bob Carpenter. 2014. The benefits of a model of annotation. Transactions of the Association for Computational Linguistics, 2:311–326. find
  30. Barbara Plank, Dirk Hovy, and Anders Sogaard. 2014. Linguistically debatable or just plain wrong? In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). find
  31. Massimo Poesio and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of ACL Workshop on Frontiers in Corpus Annotation, pages 76–83. find our card
  32. Massimo Poesio, Jon Chamberlain, and Udo Kruschwitz. 2017. Crowdsourcing. In Handbook of Linguistic Annotation. Springer. find
  33. Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Livio Robaldo, and Luca Ducceschi. 2013. Phrase Detectives: Utilizing collective intelligence for internet-scale language resource creation. ACM Transactions on Intelligent Interactive Systems, 3(1). find
  34. Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolution with the ARRAU corpus. In Proceedings of the NAACL Workshop on Computational Models of Reference,… find our card
  35. Massimo Poesio, Sameer Pradhan, Marta Recasens, Kepa Rodriguez, and Yannick Versley. 2016a. Annotated corpora and annotation tools. In Anaphora Resolution: Algorithms, Resources and Applications. Springer. find
  36. Massimo Poesio, Roland Stuckardt, and Yannick Versley. 2016b. Anaphora Resolution: Algorithms, Resources and Applications. Springer, Berlin. find
  37. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea. find our card
  38. Sameer S. Pradhan, Eduard Hovy, Mitch Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2007. Ontonotes: A unified relational semantic representation. In International Conference on Semantic Computing (ICSC 2007), pages 517–526. find
  39. Nguyen Quoc Viet Hung, Nguyen Thanh Tam, Lam Ngoc Tran, and Karl Aberer. 2013. An evaluation of aggregation techniques in crowdsourcing. In Web Information Systems Engineering – WISE 2013, pages 1–15, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  40. Vikas C. Raykar, Shipeng Yu, Linda H. Zhao, Gerardo Hermosillo Valadez, Charles Florin, Luca Bo-goni, and Linda Moy. 2010. Learning from crowds. Journal of Machine Learning Research, 11:1297– 1322. find
  41. Arndt Riester, David Lorenz, and Nina Seeman. 2010. A recursive annotation scheme for referential information status. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2010), pages 717–722. find
  42. Aashish Sheshadri and Matthew Lease. 2013. SQUARE: A Benchmark for Research on Computing Crowd Consensus. In Proceedings of the 1st AAAI Conference on Human Computation (HCOMP), pages 156–164. find
  43. Padhraic Smyth, Usama Fayyad, Michael Burl, Pietro Perona, and Pierre Baldi. 1994. Inferring ground truth from subjective labelling of Venus images. In Proceedings of the 7th International Conference on Neural Information Processing Systems, NIPS’94, pages 1085–1092, Cambridge, MA, USA. MIT Press. find
  44. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 254–263. find
  45. Wee Meng Soon, Hwee Tou Ng, and Daniel Chung Yong Lim. 2001. A machine learning approach to coreference resolution of noun phrases. Computational Linguistics, 27(4):521–544. find our card
  46. Josef Steinberger, Massimo Poesio, Mijail Kabadjov, and Karel Jezek. 2007. Two uses of anaphora resolution in summarization. Information Processing and Management, 43(6):1663–1680. Special issue on Summarization. find
  47. Grigorios Tsoumakas and Ioannis Katakis. 2007. Multi-label classification: An overview. International Journal of Data Warehousing and Mining (IJDWM), 2007:1–13. find
  48. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. Ontonotes: A large training corpus for enhanced processing. Handbook of Natural Language Processing and Machine Translation. Springer. find
  49. Jacob Whitehill, Ting-fan Wu, Jacob Bergsma, Javier R. Movellan, and Paul L. Ruvolo. 2009. find
  50. Whose vote should count more: Optimal integration of labels from labelers of unknown expertise. In Advances in Neural Information Processing Systems 22, pages 2035–2043. Curran Associates, Inc. find
  51. Zhiping Zheng. 2002. Answerbus question answering system. In Proceedings of the Second International Conference on Human Language Technology Research, HLT ’02, pages 399–404, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc. find
  52. A Similar steps were taken to derive the variational parameters associated with the sensitivity α and specificity β. In Equation (12) we derive the complete conditional associated with the positive outcome of the true label indicator: p(c i,m = 1\|...) ∝ p(c i,m = 1\|π z i,m )× × find

↑ back to top

This page in Markdown →