Elamur Research⌕
☰
home / papers / 2023 / The Universal Anaphora Scorer 2.0

The Universal Anaphora Scorer 2.0

Summary

The Universal Anaphora (UA) initiative wants evaluation tools that cover more than plain coreference. Version 1.0 of its scorer already handled identity anaphora, split antecedents, bridging and discourse deixis. However, it could only score contiguous mentions, and it could evaluate zero anaphora only when the zeros were given in the test data. A separate fork built for CorefUD had added support for discontinuous mentions. This paper presents release 2.0, a Python tool that merges the two branches. It reads both the CoNLL-UA and CorefUD markup formats, scores discontinuous markables, and adds a new method for the basic case of zero anaphora, where systems must predict unrealized arguments themselves.

The authors validated the tool by re-scoring system outputs from two earlier shared tasks. Under exact mention matching, its results agree with the original official scorers. Small differences under partial matching come from deliberate design choices. The release gives multilingual, multi-genre anaphora research one consistent evaluation tool that is compatible with UD.

Key points

  • The authors tested the scorer on the winning CorPipe submission to the CRAC 2022 multilingual task. Strict-matching scores were identical to those of the CorefUD scorer 1.0 (e.g., CoNLL F1 33.18).
  • Under partial matching on the same submission, UA 2.0 scores came out slightly lower (CoNLL F1 70.49 vs 70.72). The cause is that UA 2.0 enforces one-to-one alignment between overlapping mentions. The MOR overlap score also differs between the two scorers because they map mentions differently.
  • The scorer was also checked against the CRAFT-CR 2019 scorer, using output from the system of Lu and Poesio (2021). Strict scores matched exactly (CoNLL F1 47.67), and partial scores were nearly the same (50.23 vs 50.22). The original scorer gave slightly different results across repeated runs, while UA 2.0 was deterministic.
  • Zero anaphora appears in Arabic/Chinese OntoNotes and in several CorefUD datasets. Scoring predicted zeros requires handling tokens that the system inserts.
  • The publicly available CorefUD 1.1 contains 17 datasets in 12 languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 5 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yu2023universal,
  title = {{The Universal Anaphora Scorer 2.0}},
  author = {Yu, Juntao and Novák, Michal and Aloraini, Abdulrahman and Moosavi, Nafise Sadat and Paun, Silviu and Pradhan, Sameer and Poesio, Massimo},
  booktitle = {Proceedings of the 15th International Conference on Computational Semantics},
  year = {2023},
  url = {https://aclanthology.org/2023.iwcs-1.19/}
}

References

47 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdulrahman Aloraini, Sameer Pradhan, and Massimo Poesio. 2022. Joint coreference resolu-12 https://github.com/bill-baumgartner/ reference-coreference-scorers tion for zeros and non-zeros in Arabic. In Proceedings of the The Seventh Arabic Natural Language Processing Workshop (WANLP), pages 11– 21, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. ↗
  2. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In The First International Conference on Language Resources and Evaluation (LREC) - Workshop on linguistics coreference, volume 1, pages 563– 566. ACL. find our card
  3. William Baumgartner, Michael Bada, Sampo Pyysalo, Manuel R. Ciosici, Negacy Hailu, Harrison Pielke-Lombardo, Michael Regan, and Lawrence Hunter. 2019. CRAFT shared tasks 2019 overview — integrated structure, semantics, and coreference. In Proceedings of the 5th Workshop on BioNLP Open Shared Tasks, pages 174–184, Hong Kong, China. Association for Computational Linguistics. find
  4. Herbert H. Clark. 1977. Bridging. In P. N. Johnson-Laird and P.C. Wason, editors, Thinking: Readings in Cognitive Science, pages 411–420. Cambridge University Press, London and New York. find
  5. Kevin Bretonnel Cohen, Arrick Lanfranchi, Miji Joo-young Choi, Michael Bada, William A. Baumgartner Jr., Natalya Panteleyeva, Karin Verspoor, Martha Palmer, and Lawrence E. Hunter. 2017. Coreference annotation and resolution in the Colorado Richly Annotated Full Text (CRAFT) corpus of biomedical journal articles. BMC Bioinformatics, 18(372). ↗
  6. Pascal Denis and Jason Baldridge. 2009. Global joint models for coreference resolution and named entity classification. Procesamiento del Lenguaje Natural, 42:87–96. find
  7. Carola Eschenbach, Christopher Habel, Michael Herweg, and Klaus Rehkämper. 1989. Remarks on plural anaphora. In Proceedings of the fourth conference on European chapter of the Association for Computational Linguistics, pages 161– 167. Association for Computational Linguistics. find our card
  8. Jan Hajič, Eduard Bejček, Jaroslava Hlaváčová, Marie Mikulová, Milan Straka, Jan Štěpánek, and Barbora Štěpá nková. 2020. Prague Dependency Treebank - Consolidated 1.0. In Proceedings of the 12th International Conference on Language Resources and Evaluation (LREC 2020), pages 5208–5218, Marseille, France. European Language Resources Association. find
  9. John A. Hawkins. 1978. Definiteness and Indefiniteness. Croom Helm, London. find
  10. Yufang Hou. 2020. Bridging anaphora resolution as question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1428–1438, Online. Association for Computational Linguistics. ↗ our card
  11. Yufang Hou, Katja Markert, and Michael Strube. 2018. Unrestricted bridging resolution. Computational Linguistics, 44(2):237–284. ↗ our card
  12. Hans Kamp and Uwe Reyle. 1993. From Discourse to Logic. D. Reidel, Dordrecht. ↗
  13. Sopan Khosla, Juntao Yu, Ramesh Manuvinakurike, Vincent Ng, Massimo Poesio, Michael Strube, and Carolyn Rosé. 2021. The codi-crac 2021 shared task on anaphora, bridging, and discourse deixis in dialogue. In Proc. of the CODI/CRAC Shared Task Workshop. find our card
  14. Hideo Kobayashi and Vincent Ng. 2021. Bridging resolution: Making sense of the state of the art. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1652–1659, Online. Association for Computational Linguistics. find our card
  15. Varada Kolhatkar, Adam Roussel, Stefanie Dipper, and Heike Zinsmeister. 2018. Anaphora with non-nominal antecedents in computational linguistics: a Survey. Computational Linguistics, 44(3):547–612. ↗ our card
  16. Harold W Kuhn. 1955. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83–97. ↗
  17. Pengcheng Lu and Massimo Poesio. 2021. Coreference resolution for the biomedical domain: A survey. In Proc. of the CRAC Workshop. ↗ our card
  18. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25–32, Vancouver, British Columbia, Canada. Association for Computational Linguistics. ↗ our card
  19. Xiaoqiang Luo, Sameer Pradhan, Marta Recasens, and Eduard Hovy. 2014. An extension of BLANC to system mentions. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 24–29, Baltimore, Maryland. Association for Computational Linguistics. find
  20. Ana Marasović, Leo Born, Juri Opitz, and Anette Frank. 2017. A mention-ranking model for abstract anaphora resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 221–232, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  21. Nafise S. Moosavi and Michael Strube. 2016. A proposal for a link-based entity aware metric. In Proc. of ACL, pages 632–642, Berlin. find
  22. James Munkres. 1957. Algorithms for the assignment and transportation problems. Journal of the society for industrial and applied mathematics, 5(1):32–38. ↗
  23. Anna Nedoluzhko, Michal Novák, Martin Popel, Zdeněk Žabokrtský, Amir Zeldes, and Daniel Zeman. 2022. CorefUD 1.0: Coreference meets Universal Dependencies. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 4859–4872, Marseille, France. European Language Resources Association. ↗ our card
  24. Anna Nedoluzhko, Michal Novák, Martin Popel, Zdenĕk Z̆abokrtský, and Daniel Zeman. 2021. Coreference meets universal dependencies – a pilot experiment on harmonizing coreference datasets for 11 languages. ÚFAL Technical Report TR-2021-66, Charles University, Prague. find our card
  25. Michal Novák, Martin Popel, Zdeněk Žabokrtský, Daniel Zeman, Anna Nedoluzhko, Kutay Acar, Peter Bourgonje, Silvie Cinková, Gülşen Ce-biroğlu Eryiğit, Jan Hajič, Christian Hardmeier, Dag Haug, Tollef Jørgensen, Andre Kåsen, Pauline Krielke, Frédéric Landragin, Ekaterina Lapshinova-Koltunski, Petter Mæhlum, M. Antònia Martı́, Marie Mikulová, Anders Nøklestad, Maciej Ogrodniczuk, Lilja Øvrelid, Tuğba Pamay Arslan, Marta Recasens, Per Erik Solberg, Manfred Stede, Milan Straka, Svetlana Toldova, Noémi Vadász, Erik Velldal, Veronika Vincze, Amir Zeldes, and Voldemaras Žitkus. 2023. Coreference in universal dependencies 1.1 (CorefUD 1.1). LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University. find
  26. Tim O’Gorman, Michael Regan, Kira Griffitt, Ulf Hermjakob, Kevin Knight, and Martha Palmer. 2018. Amr beyond the sentence: the multi-sentence amr corpus. In Proc. of COLING, pages 3693–3702, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  27. Silviu Paun, Juntao Yu, Nafise Moosavi, and Massimo Poesio. 2023. Scoring coreference chains with split-antecedent anaphors and other entities constructed from a discourse model. Dialogue and Discourse. find our card
  28. Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolution with the ARRAU corpus. In Proceedings of the First Workshop on Computational Models of Reference, Anaphora and Coreference, pages 11–22, New Orleans, Louisiana. Association for Computational Linguistics. find our card
  29. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 30–35, Baltimore, Maryland. Association for Computational Linguistics. ↗ our card
  30. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea. find our card
  31. Ellen F. Prince. 1981. Toward a taxonomy of given-new information. In P. Cole, editor, Radical Pragmatics, pages 223–256. Academic Press, New York. find
  32. Marta Recasens and Ed Hovy. 2011. Blanc: Implementing the rand index for coreference evaluation. Natural Language Engineering. ↗
  33. Marta Recasens and M. Antònia Martı́. 2010. AnCora-CO: Coreferentially annotated corpora for Spanish and Catalan. Language Resources and Evaluation, 44(4):315–345. ↗
  34. Milan Straka and Jana Straková. 2022. ÚFAL Cor-Pipe at CRAC 2022: Effectivity of multilingual models for coreference resolution. In Proceedings of the CRAC 2022 Shared Task on Multilingual Coreference Resolution, pages 28–37, Gyeongju, Republic of Korea. Association for Computational Linguistics. find
  35. Don Tuggener. 2014. Coreference resolution evaluation for higher level applications. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, volume 2: Short Papers, pages 231– 235, Gothenburg, Sweden. Association for Computational Linguistics. find our card
  36. Olga Uryupina, Ron Artstein, Antonella Bristot, Federica Cavicchio, Francesca Delogu, Kepa J. Rodriguez, and Massimo Poesio. 2020. Annotating a broad range of anaphoric phenomena, in a variety of genres: the ARRAU corpus. Journal of Natural Language Engineering. ↗
  37. Hardik Vala, Andrew Piper, and Derek Ruths. 2016. The more antecedents, the merrier: Resolving multi-antecedent anaphors. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2287–2296, Berlin, Germany. Association for Computational Linguistics. find our card
  38. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland, November 6-8, 1995. ↗ our card
  39. Bonnie L. Webber. 1991. Structure and ostension in the interpretation of discourse deixis. Language and Cognitive Processes, 6(2):107–135. ↗
  40. Juntao Yu, Sopan Khosla, Ramesh Manuvinakurike, Lori Levin, Vincent Ng, Massimo Poesio, Michael Strube, and Massimo Poesio. 2022a. The CODI/CRAC 2022 shared task on anaphora resolution, bridging and discourse deixis in dialogue. In Proc. of CODI/CRAC Shared Task. find
  41. Juntao Yu, Sopan Khosla, Nafise Moosavi, Silviu Paun, Sameer Pradhan, and Massimo Poesio. 2022b. The universal anaphora scorer 1.0. In Proc. of LREC. find our card
  42. Juntao Yu, Nafise Sadat Moosavi, Silviu Paun, and Massimo Poesio. 2020. Free the plural: Unrestricted split-antecedent anaphora resolution. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6113– 6125, Barcelona, Spain (Online). International Committee on Computational Linguistics. find our card
  43. Juntao Yu, Nafise Sadat Moosavi, Silviu Paun, and Massimo Poesio. 2021. Stay together: A system for single and split-antecedent anaphora resolution. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. find our card
  44. Juntao Yu and Massimo Poesio. 2020. Multitask learning based neural bridging reference resolution. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3534–3546, Barcelona, Spain (Online). International Committee on Computational Linguistics. ↗ our card
  45. Zdeněk Žabokrtský, Miloslav Konopı́k, Anna Nedoluzhko, Michal Novák, Maciej Ogrodniczuk, Martin Popel, Ondřej Pražák, Jakub Sido, Daniel Zeman, and Yilun Zhu. 2022. Findings of the shared task on multilingual coreference resolution. In Proceedings of the CRAC 2022 Shared Task on Multilingual Coreference Resolution, pages 1–17, Gyeongju, Republic of Korea. Association for Computational Linguistics. ↗
  46. Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. ↗
  47. Ethan Zhou and Jinho D. Choi. 2018. They exist! introducing plural mentions to coreference resolution and entity linking. In Proceedings of the 27th International Conference on Computational Linguistics, pages 24–34, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2023-universal-anaphora-scorer-2-0/