Elamur Research⌕
☰
home / papers / 2023 / Aggregating Crowdsourced and Automatic Judgments to Scale Up a Corpus of Anaphoric Reference for Fiction and Wikipedia Texts

Aggregating Crowdsourced and Automatic Judgments to Scale Up a Corpus of Anaphoric Reference for Fiction and Wikipedia Texts

Paper · 2023 · Juntao Yu, Silviu Paun, Maris Camilleri et al.

Summary

Existing coreference corpora are mostly mid-sized, concentrated on news or science, short-document, and often skip singletons and non-referring expressions. The authors release Phrase Detectives 3.0, an English anaphora corpus labelled mainly through the Phrase Detectives game-with-a-purpose, covering fiction and Wikipedia. Players fully annotated only about 70% of documents, so the authors propose an iterative resolve-and-aggregate procedure. An automatic anaphoric resolver labels every mention, including those humans never reached. Its output is then merged with the crowd judgments by a probabilistic aggregation method built for anaphora. The enlarged dataset is used to retrain the resolver, and the cycle repeats. Models trained on the completed corpus beat models trained only on fully player-annotated documents and models trained on the uncompleted full corpus. The resulting resource matches ONTONOTES in token count, has roughly twice as many markables, and contains many long documents. This gives the field training and test data for underrepresented genres and for long-document coreference.

Key points

  • Phrase Detectives 3.0 has about 1.37M tokens and roughly twice the markables of ONTONOTES; the counts of non-singleton markables are similar.
  • The annotation scheme is a simplified ARRAU scheme. Every noun phrase is marked as referring or non-referring, and singletons are included.
  • In the game, each markable gets at least 8 player judgments (20 on average), and each competing interpretation is checked by at least four other players in a validation game.
  • Document length reaches 14K tokens. Other corpora have much shorter documents on average: PRECO averages 329 tokens, ONTONOTES 467, ARRAU 630, and earlier Phrase Detectives 753.
  • With gold mentions and singletons excluded, LONGFORMER outperforms ROBERTA on the corpus by 0.9% CoNLL F1 on short documents and 2% on long ones.

Interesting ideas

  • I1. When human coverage of a dataset is incomplete, a model can supply labels for the gaps, and model labels can be pooled with human ones through a principled aggregator; the improved data then retrains the model in a loop.
  • I2. Annotation is split into two roles: one person proposes a reading, and others only agree or disagree with each competing reading, which is cheaper and more reliable than open labelling.
  • I3. Annotators work against gold data until they reach sufficient accuracy, and only then are allowed onto unlabelled items.
  • I4. A test set that separates long and short documents can reveal architectural differences that a short-document benchmark hides.
  • I5. Explicitly marking non-referring expressions and singletons, rather than leaving them unannotated, makes the referential status of every noun phrase part of the data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data could treat human corrections and the witness parsers' outputs as votes in a single aggregation step, then retrain and re-aggregate, rather than trusting any single source.
  • I2 (learn): When parsers or LLM teacher tiers disagree on an arc or a coreference link, the alternatives could go to a yes/no validator tier instead of asking for a fresh analysis.
  • I3 (learn): New witness parsers or teacher prompts could have to pass a gold-checked qualification set before their votes count toward silver data.
  • I4 (frames): Evaluation of coreference in crate prag should report scores by document length, so that failures in long-range tracking across whole tales are not averaged away.
  • I5 (world): The world builder could decide per noun phrase whether to create an entity at all, which avoids phantom entities from expletive or predicative phrases.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢AmazonSeattle · United States112 works · 40,565 citations🎓University of EssexColchester · United Kingdom34 works · 3,537 citations🎓Queen Mary University of LondonLondon · United Kingdom54 works · 3,542 citations🎓University of RegensburgRegensburg · Germany5 works · 14 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksCoreference resolution · 574 works

Cite

Cite as Juntao Yu, Silviu Paun, Maris Camilleri, Paloma Garcia, Jon Chamberlain, Udo Kruschwitz and Massimo Poesio. 2023. Aggregating Crowdsourced and Automatic Judgments to Scale Up a Corpus of Anaphoric Reference for Fiction and Wikipedia Texts. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.eacl-main.54 · arXiv 2210.05581

@inproceedings{yu2023aggregating,
  title = {{Aggregating Crowdsourced and Automatic Judgments to Scale Up a Corpus of Anaphoric Reference for Fiction and Wikipedia Texts}},
  author = {Juntao Yu, Silviu Paun, Maris Camilleri, Paloma Garcia, Jon Chamberlain, Udo Kruschwitz and Massimo Poesio},
  booktitle = {Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.eacl-main.54},
  eprint = {2210.05581},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.eacl-main.54}
}

References

55 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. K Bretonnel Cohen, Arrick Lanfranchi, Miji Joo-young Choi, Michael Bada, William A Baumgartner, Natalya Panteleyeva, Karin Verspoor, Martha Palmer, and Lawrence E Hunter. 2017. Coreference annotation and resolution in the colorado richly annotated full text (craft) corpus of biomedical journal articles. BMC bioinformatics, 18(1):1–14. ↗
  2. Seth Cooper, Firsas Khatib, Adrien Treuille, Janos Barbero, Jeehyung Lee, Michael Beenen, Andrew Leaver-Fay, David Baker, Zoran Popovic, and the Foldit Players. 2010. Predicting protein structures with a multiplayer online game. Nature, 466:756– 760. find
  3. Alexander P. Dawid and Allan M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Applied Statistics, 28(1):20– 28. ↗
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. ↗
  5. Anca Dumitrache, Lora Aroyo, Chris Welty, Robert-Jan Sips, and Anthony Levas. 2013. Dr. detective: combining gamification techniques and crowdsourcing to create a gold standard in medical text. In Proc. of CrowdSem. find
  6. Dagmara Dziedzic and Wojciech Włodarczyk. 2017. Making nlp games with a purpose fun to play using free to play mechanics: RoboCorp case study. In Proc. of the Games4NLP Symposium. find
  7. Ali Emami, Paul Trichelair, Adam Trischler, Kaheer Suleman, Hannes Schulz, and Jackie Chi Kit Cheung. 2019. The knowref coreference corpus: Removing gender and number cues for difficult pronominal anaphora resolution. In Proc. of the ACL.
  8. Eraldo R. Fernandes, Cícero N. dos Santos, and Ruy L. Milidiú. 2014. Latent trees for coreference resolution. Computational Linguistics, 40(4):801–835. ↗
  9. Karen Fort, Bruno Guillaume, and H. Chastant. 2014. Creating Zombilingo, a game with a purpose for dependency syntax annotation. In Proceedings of the 1st International Workshop on Gamification for Information Retrieval (GamifIR’14), pages 2–6. ACM. find
  10. Luke Gessler, Siyao Peng, Yang Liu, Yilun Zhu, Shabnam Behzad, and Amir Zeldes. 2020. AMALGUM – a free, balanced, multilayer english web corpus. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC), page 5267–5275. European Language Resources Association. ↗
  11. Abbas Ghaddar and Phillippe Langlais. 2016. Wiki-Coref: An English coreference-annotated corpus of Wikipedia articles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 136–142, Portorož, Slovenia. European Language Resources Association (ELRA).
  12. Iris Hendrickx, Gosse Bouma, Frederik Coppens, Walter Daelemans, Veronique Hoste, Geert Kloosterman, Anne-Marie Mineur, Joeri Van Der Vloet, and Jean-Luc Verschelde. 2008. A coreference corpus and resolution system for dutch. In Proc. LREC.
  13. Erhard W. Hinrichs, Sandra Kübler, and Karin Naumann. 2005. A unified representation for morphological, syntactic, semantic and referential annotations. In Proc. of the ACL Workshop on Frontiers in Corpus Annotation II: Pie in the Sky, Ann Arbor, Michigan. find
  14. Barbora Hladká, Jiří Mírovský, and Pavel Schlesinger. 2009. Designing a language game for collecting coreference annotation. In Proceedings of the Third Linguistic Annotation Workshop (LAW III), pages 52– 55, Singapore. find
  15. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77.
  16. Alain Joubert, Mathieu Lafourcade, and Nathalie Le Brun. 2018. The jeuxdemots project is 10 years old: What we have learned. In Proceedings of the 2018 LREC Workshop on Games and Gamification for Natural Language Processing (Games4NLP), pages 22–26. find
  17. David Jurgens and Roberto Navigli. 2014. It’s all fun and games until someone annotates: Video games with a purpose for linguistic annotation. Transactions of the ACL. ↗
  18. Doruk Kicikoglu, Richard Bartle, Jon Chamberlain, and Massimo Poesio. 2019. Wormingo: a’true gamification’ approach to anaphoric annotation. In FDG’19: Proc. of the GAMNLP Workshop at the 14th International Conference on the Foundations of Digital Games, pages 1–7, San Luis Obispo. find
  19. Hyun-Chul Kim and Zoubin Ghahramani. 2012. Bayesian classifier combination. In Proceedings of the Fifteenth International Conference on Artificial Intelligence and Statistics, volume 22 of Proceedings of Machine Learning Research, pages 619–627, La Palma, Canary Islands. PMLR. find
  20. Markus Krause, Aneta Takhtamysheva, Marion Witt-stock, and Rainer Malaka. 2010. Frontiers of a paradigm: exploring human computation with digital games. In Proceedings of HCOMP - the ACM SIGKDD Workshop on Human Computation, pages 22–25. find
  21. Mathieu Lafourcade. 2007. Making people play for lexical acquisition with the JeuxDeMots prototype. In Proc. of SNLP. find
  22. Mathieu Lafourcade, Alain Joubert, and Nathalie Le Brun. 2015. Games with a Purpose (GWAPs). Wiley. ↗
  23. Fred Landragin. 2016. Description, modéelisation et détection automatique des chaînes de réeférence (democrat). Bulletin de l’Association Française pour l’Intelligence Artificielle, 92:11–15. find
  24. Florian Laws, Florian Heimerl, and Hinrich Schütze. 2012. Active learning for coreference resolution. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 508–512. Association for Computational Linguistics.
  25. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of EMNLP.
  26. Kenton Lee, Luheng He, and Luke S. Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of ACL.
  27. Belinda Z. Li, Gabriel Stanovsky, and Luke Zettlemoyer. 2020. Active learning for coreference resolution using discrete annotation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, page 8320–8331. Association for Computational Linguistics.
  28. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  29. Proceedings of EMNLP, pages 1926–1937, Brussels, Belgium. Association for Computational Linguistics. find
  30. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke S. Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL.
  31. Massimo Poesio, Jon Chamberlain, and Udo Kruschwitz. 2017. Crowdsourcing. In N. Ide and J. Pustejovsky, editors, The Handbook of Annotation, pages 277–295. Springer. find
  32. Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Silviu Paun, Alexandra Uma, and Juntao Yu. 2019. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proc. of NAACL, page 1778–1789, Minneapolis. Association for Computational Linguistics (ACL). ↗
  33. Chris Madge, Richard Bartle, Jon Chamberlain, Udo Kruschwitz, and Massimo Poesio. 2019. Incremental game mechanics applied to text annotation. In CHI PLAY 2019 - Proceedings of the Annual Symposium on Computer-Human Interaction in Play, pages 545– 558, Barcelona. find
  34. Massimo Poesio, Jon Chamberlain, Udo Kruschwitz, Livio Robaldo, and Luca Ducceschi. 2013. Phrase Detectives: Utilizing collective intelligence for internet-scale language resource creation. ACM Transactions on Interactive Intelligent Systems, 3(1):1–44. find
  35. Chris Madge, Jussi Brightmore, Doruk Kicikoglu, Fatima Althani, Richard Bartle, Jon Chamberlain, Udo Kruschwitz, and Massimo Poesio. 2022. Lingotowns: A virtual world for natural language annotation and language learning. In CHI PLAY 2022 - Proceedings of the Annual Symposium on Computer-Human Interaction in Play, Demo session, Bremen. find
  36. Massimo Poesio, Sameer Pradhan, Marta Recasens, Kepa Rodriguez, and Yannick Versley. 2016. Annotated corpora and annotation tools. In M. Poesio, R. Stuckardt, and Y. Versley, editors, Anaphora Resolution: Algorithms, Resources and Applications, chapter 4. Springer. ↗
  37. Chris Madge, Jon Chamberlain, Udo Kruschwitz, and Massimo Poesio. 2017. Experiment-driven development of a gwap for marking segments in text. In Proceedings of CHI PLAY, Amsterdam. find
  38. Judith Muzerelle, Anaïs Lefeuvre, Emmanuel Schang, Jean-Yves Antoine, Aurore Pelletier, Denis Maurel, Iris Eshkol, and Jeanne Villaneau. 2014. Ancor_centre, a large free spoken french coreference corpus. In Proc. of LREC. find
  39. Anna Nedoluzhko, Michal Novák, Silvie Cinková, Marie Mikulová, and Jir̂í Mírovský. 2016. Coreference in prague czech-english dependency treebank. In Proc. of LREC, Portoroz. ELRA.
  40. Silviu Paun, Ron Artstein, and Massimo Poesio. 2022. Statistical methods for annotation analysis. Synthesis Lectures on Human Language Technologies, 15(1):1– 217. ↗
  41. Silviu Paun, Bob Carpenter, Jon Chamberlain, Dirk Hovy, Udo Kruschwitz, and Massimo Poesio. 2018a. Comparing bayesian models of annotation. Transactions of the Association for Computational Linguistics. ↗
  42. Silviu Paun, Jon Chamberlain, Udo Kruschwitz, Juntao Yu, and Massimo Poesio. 2018b. A probabilistic annotation model for crowdsourcing coreference. In Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea.
  43. Ellen F. Prince. 1992. The ZPG letter: subjects, definiteness, and information status. In S. Thompson and W. Mann, editors, Discourse description: diverse analyses of a fund-raising text, pages 295–325. John Benjamins. find
  44. Marta Recasens and M. Antònia Martí. 2010. AnCora-CO: Coreferentially annotated corpora for Spanish and Catalan. Language Resources and Evaluation, 44(4):315–345. ↗
  45. Rion Snow, Brendan O’Connor, Daniel Jurafsky, and Andrew Y. Ng. 2008. Cheap and fast - but is it good? Evaluating non-expert annotations for natural language tasks. In Proceedings of EMNLP, pages 254– 263. ↗
  46. Olga Uryupina, Ron Artstein, Antonella Bristot, Federica Cavicchio, Francesca Delogu, Kepa J. Rodriguez, and Massimo Poesio. 2020. Annotating a broad range of anaphoric phenomena, in a variety of genres: the ARRAU corpus. Journal of Natural Language Engineering. ↗
  47. Luis von Ahn. 2006. Games with a purpose. Computer, 39(6):92–94. ↗
  48. Luis von Ahn and Laura Dabbish. 2008. General techniques for designing games with a purpose. Communications of the ACM, pages 58–67. find
  49. Sam J. Wiseman, Alexander M. Rush, Stuart M. Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Proc. of the ACL, Bejing.
  50. Patrick Xia and Benjamin Van Durme. 2021. Moving on from ontonotes: Coreference resolution model transfer. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, page 5241–5256. Association for Computational Linguistics. find
  51. Juntao Yu, Sopan Khosla, Ramesh Manuvinakurike, Lori Levin, Vincent Ng, Massimo Poesio, Michael Strube, and Massimo Poesio. 2022a. The CODI/CRAC 2022 shared task on anaphora resolution, bridging and discourse deixis in dialogue. In Proc. of CODI/CRAC Shared Task. find
  52. Juntao Yu, Sopan Khosla, Nafise Sadat Moosavi, Silviu Paun, Sameer Pradhan, and Massimo Poesio. 2022b. The universal anaphora scorer. In Proceedings of the Language Resources and Evaluation Conference, pages 4873–4883, Marseille, France. European Language Resources Association.
  53. Juntao Yu, Alexandra Uma, and Massimo Poesio. 2020. A cluster ranking model for full anaphora resolution. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 11–20, Marseille, France. European Language Resources Association. ↗
  54. Michelle Yuan, Patrick Xia, Chandler May, Benjamin Van Durme, and Jordan Boyd-Graber. 2022. Adapting coreference resolution models through active learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, page 7533–7549. Association for Computational Linguistics.
  55. Amir Zeldes. 2020. Multilayer corpus studies. Routledge. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2023-aggregating-crowdsourced-automatic-judgments-scale/