Elamur Research⌕
☰
home / papers / 2024 / The ARRAU 3.0 Corpus

The ARRAU 3.0 Corpus

Paper · 2024 · Massimo Poesio, Maris Camilleri, Paloma Carretero Garcia et al.

Summary

ARRAU is a long-running effort, started in 2004, to build an English corpus whose anaphoric annotation goes well beyond plain identity coreference. Release 2 (2013) served as the core data for the CRAC 2018 shared task and as extra material for the CODI-CRAC 2021 and 2022 dialogue tasks. Those tasks exposed weak spots in the guidelines, notably discourse deixis, genericity and reference in dialogue, as well as faults in tokenization and markup. The authors re-annotated and cleaned the corpus, wrote a new annotation manual, and redesigned tokenization. They also repaired errors that had crept in when the Penn Treebank–derived RST portion was converted. The result, ARRAU 3, adds only a little material over ARRAU 2. However, it contains far more annotated generic mentions, bridging references and ambiguous markables. It covers news, medical and art-history text, spoken narratives and task-oriented dialogue. The corpus is distributed in MMAX XML and Universal Anaphora formats, which makes it a broad benchmark for anaphora phenomena that most coreference datasets ignore.

Key points

  • Subcorpora: RST (the Penn Treebank documents also present in the RST treebank), TRAINS (TRAINS-91 and TRAINS-93 dialogues), PEAR (Pear Stories spoken narratives) and GNOME (medical and art-history texts).
  • Size compared with ARRAU 2:
    • documents: 558 vs 552
    • tokens: 359,500 vs 348,072
    • markables: 106,700 vs 99,582
  • Annotated phenomena:
    • Every NP counts as a markable, including singletons.
    • Non-referring NPs are split into expletive, predicative and quantificational.
    • Discontinuous markables are allowed.
    • Generic premodifiers (e.g. "US", "exchange-rate") are marked.
    • Split-antecedent plurals, discourse deixis and bridging are covered.
  • Change in complex phenomena: discourse deixis and split-antecedent counts barely moved. Generics, bridging and ambiguous markables rose sharply; for example, RST generics went from 1438 to 6870.
  • Availability: the full corpus is distributed via LDC. The copyright-free parts (GNOME, PEAR, TRAINS-91) are also available in the Universal Anaphora repository. A new ARRAU 3 Annotation Manual is provided.

Interesting ideas

  • I1. The new tokenizer breaks words at internal non-word characters such as hyphens, but records how many whitespace characters precede each token, so the original text can be rebuilt exactly.
  • I2. Conversion from treebank bracketing silently broke tokens. Comma-separated numbers such as 4,393,237 shrank to "4", and "11 1/2" became "11 1".
  • I3. Marking non-referring NPs (expletives, predicatives, quantifiers) and singletons is treated as part of anaphora resolution, rather than leaving them out of the annotation.
  • I4. Plural pronouns can be linked to several separate antecedents at once (split antecedents). Genericity is annotated because whether two bare plurals corefer depends on whether they denote a kind.
  • I5. Using the corpus in shared tasks was the main way guideline problems surfaced. Cases annotators could not resolve are explicitly flagged as ambiguous rather than forced into one reading.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tokens): The crate en tokenizer could keep a leading-whitespace count per token, so that splitting compounds never prevents lossless reconstruction of the source string.
  • I2 (tree): When converting treebanks between UD dialects, we could check every converted sentence against its raw text, which would catch lost numerals and fractions before they contaminate training data.
  • I3 (frames): The prag coreference step could first decide whether a mention refers at all. This would keep "it" in "It is four o'clock" or "a busy place" out of coreference chains.
  • I4 (world): The text world could support set-valued entities built from earlier individuals, and could keep kind-level entities apart from specific instances.
  • I5 (learn): Silver data could carry an explicit ambiguity label wherever witness parsers disagree, and evaluation errors could be fed back into guideline or rule revisions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Queen Mary University of LondonLondon · United Kingdom54 works · 3,542 citations🔬Leibniz Institute for the German LanguageMannheim · Germany5 works · 22 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksCoreference resolution · 574 works

Cite

Cite as Massimo Poesio, Maris Camilleri, Paloma Carretero Garcia, Juntao Yu and Mark-Christoph Müller. 2024. The ARRAU 3.0 Corpus. Proceedings of the 5th Workshop on Computational Approaches to Discourse (CODI 2024).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.codi-1.12

@inproceedings{poesio2024arrau,
  title = {{The ARRAU 3.0 Corpus}},
  author = {Massimo Poesio, Maris Camilleri, Paloma Carretero Garcia, Juntao Yu and Mark-Christoph Müller},
  booktitle = {Proceedings of the 5th Workshop on Computational Approaches to Discourse (CODI 2024)},
  year = {2024},
  doi = {10.18653/v1/2024.codi-1.12},
  url = {https://doi.org/10.18653/v1/2024.codi-1.12}
}

References

62 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Loïc Grobol. 2020. Coreference resolution for spoken French. Ph.D. thesis, Université Sorbonne Nouvelle. find
  2. Ron Artstein and Massimo Poesio. 2006. Identifying reference to abstract objects in dialogue. In Proc. of BRANDIAL, Potsdam. find
  3. Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics. Computational Linguistics, 34(4):555–596. ↗
  4. David Bamman, Olivia Lewke, and Anya Mansoor. 2020. An annotated dataset of coreference in english literature. In Proc. of LREC. European Language Resources Association (ELRA), Association for Computational Linguistics (ACL). ↗
  5. Donna Byron. 2002. Resolving pronominal references to abstract entities. In Proc. of the ACL, pages 80–87. find
  6. Donna Byron and James Allen. 1998. Resolving demonstrative anaphora in the TRAINS-93 corpus. In Proc. of the Second Colloquium on Discourse, Anaphora and Reference Resolution. University of Lancaster. find
  7. Greg N. Carlson and Francis J. Pelletier, editors. 1995. The Generic Book. University of Chicago Press. find
  8. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In J. Kuppevelt and R. Smith, editors, Current Directions in Discourse and Dialogue, pages 85–112. Kluwer. ↗
  9. Peter A. Heeman and James F. Allen. 1995. The TRAINS-93 dialogues. TRAINS Technical Note TN 94-2, University of Rochester, Dept. of Computer Science, Rochester, NY. ↗
  10. Peter A. Heeman and Graeme Hirst. 1995. Collaborating on referring expressions. Computational Linguistics, 21(3):351–382. ↗
  11. Erhard W. Hinrichs, Sandra Kübler, Karin Naumann, Heike Telljohann, and Julia Trushkin. 2004. Recent developments in linguistic annotations of the TüBa-D/Z treebank. In Proceedings of the Third Workshop on Treebanks and Linguistic Theories, Tübingen, Germany. find
  12. Yufang Hou, Katja Markert, and Michael Strube. 2014. A rule-based system for unrestricted bridging resolution: Recognizing bridging anaphora and finding links to antecedents. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2082–2093, Doha, Qatar. Association for Computational Linguistics.
  13. Yufang Hou, Katja Markert, and Michael Strube. 2018. Unrestricted bridging resolution. Computational Linguistics, 44(2):237–284. ↗
  14. Wallace L. Chafe. 1980. The Pear Stories: Cognitive, Cultural and Linguistic Aspects of Narrative Production. Ablex, Norwood, NJ. find
  15. Hans Kamp and Uwe Reyle. 1993. From Discourse to Logic. D. Reidel, Dordrecht. ↗
  16. Herbert H. Clark. 1977. Bridging. In P. N. Johnson-Laird and P.C. Wason, editors, Thinking: Readings in Cognitive Science, pages 411–420. Cambridge University Press, London and New York. find
  17. Lauri Karttunen. 1976. Discourse referents. In J. Mc-Cawley, editor, Syntax and Semantics 7 - Notes from the Linguistic Underground, pages 363–385. Academic Press, New York. find
  18. Herbert H. Clark and Susan E. Brennan. 1990. Grounding in communication. In L. B. Resnick, J. Levine, and S. D. Behrend, editors, Perspectives on Socially Shared Cognition. APA. ↗
  19. Sopan Khosla, Juntao Yu, Ramesh Manuvinakurike, Vincent Ng, Massimo Poesio, Michael Strube, and Carolyn Rosé. 2021. The codi-crac 2021 shared task on anaphora, bridging, and discourse deixis in dialogue. In Proc. of the CODI/CRAC Shared Task Workshop.
  20. Kevin Bretonnel Cohen, Arrick Lanfranchi, Miji Jooyoung Choi, Michael Bada, William A. Baumgartner Jr., Natalya Panteleyeva, Karin Verspoor, Martha Palmer, and Lawrence E. Hunter. 2017. Coreference annotation and resolution in the Colorado Richly Annotated Full Text (CRAFT) corpus of biomedical journal articles. BMC Bioinformatics, 18(372). ↗
  21. Miriam Eckert and Michael Strube. 2001. Dialogue acts, synchronising units and anaphora resolution. Journal of Semantics. find
  22. Carola Eschenbach, Christopher Habel, Michael Herweg, and Klaus Rehkämper. 1989. Remarks on plural anaphora. In Proceedings of the fourth conference on European chapter of the Association for Computational Linguistics, pages 161–167. Association for Computational Linguistics. ↗
  23. Hideo Kobayashi and Vincent Ng. 2020. Bridging resolution: A survey of the state of the art. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3708–3721, Barcelona, Spain (Online). International Committee on Computational Linguistics. ↗
  24. Varada Kolhatkar and Graeme Hirst. 2014. Resolving shell nouns. In Proc. of EMNLP, pages 499–510, Doha, Qatar. find
  25. Varada Kolhatkar, Adam Roussel, Stefanie Dipper, and Heike Zinsmeister. 2018. Anaphora with non-nominal antecedents in computational linguistics: a Survey. Computational Linguistics, 44(3):547–612. ↗
  26. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: the Penn Treebank. Computational Linguistics, 19(2):313–330. ↗
  27. Katja Markert, Yufang Hou, and Michael Strube. 2012. Collective classification for fine-grained information status. In Proc. of the ACL, Juju island, Korea.
  28. Natalia N. Modjeska. 2003. Resolving other anaphors. Ph.D. thesis, University of Edinburgh. find
  29. Mark-Christoph Müller. 2008. Fully Automatic Resolution of It, This And That in Unrestricted Multy-Party Dialog. Ph.D. thesis, Universität Tübingen. find
  30. Mark-Christoph Müller. 2020. pyMMAX2: Deep access to MMAX2 projects from python. In Proceedings of the 14th Linguistic Annotation Workshop, pages 167–173, Barcelona, Spain. Association for Computational Linguistics. ↗
  31. Mark-Christoph Müller and Michael Strube. 2006. Multi-level annotation of linguistic data with MMAX2. In S. Braun, K. Kohn, and J. Mukherjee, editors, Corpus Technology and Language Pedagogy. New Resources, New Tools, New Methods, volume 3 of English Corpus Linguistics, pages 197–214. Peter Lang. find
  32. Judith Muzerelle, Anaïs Lefeuvre, Emmanuel Schang, Jean-Yves Antoine, Aurore Pelletier, Denis Maurel, Iris Eshkol, and Jeanne Villaneau. 2014. Ancor_centre, a large free spoken french coreference corpus. In Proc. of LREC. find
  33. Costanza Navarretta. 2000. Abstract anaphora resolution in Danish. In Proc. of the 1st SIGdial Workshop on Discourse and Dialogue, pages 56–65. ACL.
  34. Anna Nedoluzhko. 2013. Generic noun phrases and annotation of coreference and bridging relations in the prague dependency treebank. In Proc. of LAW, pages 103–111. find
  35. Anna Nedoluzhko, Michal Novák, Martin Popel, Zdenĕk Z̆abokrtský, Amir Zeldes, and Daniel Zeman. 2022. Corefud 1.0: Coreference meets universal dependencies. In Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC), page 4859–4872. European Language Resources Association.
  36. Barbara Hall Partee. 1972. Opacity, coreference, and pronouns. In D. Davidson and G. Harman, editors, Semantics for Natural Language, pages 415–441. D. Reidel, Dordrecht, Holland. ↗
  37. Massimo Poesio. 2004. Discourse annotation and semantic annotation in the GNOME corpus. In Proc. of the ACL Workshop on Discourse Annotation, pages 72–79, Barcelona. ↗
  38. Massimo Poesio. 2016. Linguistic and cognitive evidence about anaphora. In M. Poesio, R. Stuckardt, and Y. Versley, editors, Anaphora Resolution: Algorithms, Resources and Applications, chapter 2. Springer. ↗
  39. Massimo Poesio and Ron Artstein. 2005a. Annotating (anaphoric) ambiguity. In Proc. of the Corpus Linguistics Conference, Birmingham. find
  40. Massimo Poesio and Ron Artstein. 2005b. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proc. of ACL Workshop on Frontiers in Corpus Annotation, pages 76–83.
  41. Massimo Poesio and Ron Artstein. 2008. Anaphoric annotation in the ARRAU corpus. In Proc. of LREC, Marrakesh. ↗
  42. Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolution with the ARRAU corpus. In Proceedings of the First Workshop on Computational Models of Reference, Anaphora and Coreference, pages 11–22, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  43. Massimo Poesio, Rahul Mehta, Alex Maroudas, and Janet Hitzeman. 2004. Learning to solve bridging references. In Proc. of ACL, pages 143–150, Barcelona. find
  44. Massimo Poesio, Sameer Pradhan, Marta Recasens, Kepa Rodriguez, and Yannick Versley. 2016. Annotated corpora and annotation tools. In M. Poesio, R. Stuckardt, and Y. Versley, editors, Anaphora Resolution: Algorithms, Resources and Applications, chapter 4. Springer. ↗
  45. Massimo Poesio and Hannes Rieser. 2010. Completions, coordination, and alignment in dialogue. Dialogue and Discourse, 1(1):1–89. ↗
  46. Massimo Poesio and Renata Vieira. 1998. A corpus-based investigation of definite description use. Computational Linguistics, 24(2):183–216. ↗
  47. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 30–35, Baltimore, Maryland. Association for Computational Linguistics. ↗
  48. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea.
  49. Sameer S. Pradhan, Lance Ramshaw, Ralph Weischedel, Jessica MacBride, and Linnea Micciulla. 2007. Unrestricted coreference: Indentifying entities and events in ontonotes. In Proc. IEEE International Conference on Semantic Computing (ICSC), Irvine, CA. find
  50. Ellen F. Prince. 1981. Toward a taxonomy of given-new information. In P. Cole, editor, Radical Pragmatics, pages 223–256. Academic Press, New York. find
  51. Altaf Rahman and Vincent Ng. 2012. Resolving complex cases of definite pronouns: The Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 777–789, Jeju Island, Korea. Association for Computational Linguistics.
  52. Marta Recasens and M. Antònia Martí. 2010. AnCora-CO: Coreferentially annotated corpora for Spanish and Catalan. Language Resources and Evaluation, 44(4):315–345. ↗
  53. Ina Roesiger. 2018. Rule- and learning-based methods for bridging resolution in the ARRAU corpus. In Proc. of CRAC.
  54. Ina Roesiger, Arndt Riester, and Jonas Kuhn. 2018. Bridging resolution: Task definition, corpus resources and rule-based experiments. In Proceedings of the 27th International Conference on Computational Linguistics, pages 3516–3528, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  55. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34:8732–8740. ↗
  56. Olga Uryupina, Ron Artstein, Antonella Bristot, Federica Cavicchio, Francesca Delogu, Kepa J. Rodriguez, and Massimo Poesio. 2020. Annotating a broad range of anaphoric phenomena, in a variety of genres: the ARRAU corpus. Journal of Natural Language Engineering. ↗
  57. Bonnie L. Webber. 1991. Structure and ostension in the interpretation of discourse deixis. Language and Cognitive Processes, 6(2):107–135. ↗
  58. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617. ↗
  59. Juntao Yu, Sopan Khosla, Ramesh Manuvinakurike, Lori Levin, Vincent Ng, Massimo Poesio, Michael Strube, and Massimo Poesio. 2022a. The CODI/CRAC 2022 shared task on anaphora resolution, bridging and discourse deixis in dialogue. In Proc. of CODI/CRAC Shared Task. find
  60. Juntao Yu, Sopan Khosla, Nafise Moosavi, Silviu Paun, Sameer Pradhan, and Massimo Poesio. 2022b. The universal anaphora scorer 1.0. In Proc. of LREC.
  61. Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. ↗
  62. Amir Zeldes. 2022. Can we fix the scope for coreference? Dialogue and Discourse, 13(1):41–62. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/poesio-2024-arrau-3-0-corpus/