Elamur Research⌕
☰
home / papers / 2022 / The Universal Anaphora Scorer

The Universal Anaphora Scorer

Summary

Coreference models now perform well on the identity anaphora annotated in ONTONOTES, so research is moving toward harder phenomena such as bridging, discourse deixis and plurals with split antecedents. Evaluation for these has relied on scattered partial tools, or on no tools at all. The authors work within the Universal Anaphora (UA) initiative, an effort modelled on Universal Dependencies to standardise anaphora annotation across languages. They present the UA scorer, a Python tool built on the Reference Coreference Scorer and on Moosavi's Generalized Coreference Scorer.

The scorer reads CONLL-UA, a CONLL-U extension with separate layers for identity, bridging and discourse deixis. It combines identity and bridging evaluation in one package. It also adds the first evaluation for split-antecedent anaphora, using an entirely new method. Discourse deixis becomes scorable too: UA encodes its antecedents like identity markables, so the usual coreference metrics apply. The scorer was the official evaluation tool of the CODI-CRAC 2021 shared task on anaphora in dialogue. It gives the field a common benchmark for anaphoric interpretation beyond plain coreference.

Key points

  • The UA scorer builds on the reference scorer for coreference released by Pradhan et al. (2014) and on coval, the CRAC 2018 Generalized Coreference Scorer. It covers identity anaphora, bridging, split-antecedent anaphora and discourse deixis.
  • Split-antecedent evaluation is a novel proposal. Discourse deixis reuses identity metrics such as MUC, B3, CEAF, BLANC, LEA and the CONLL average of MUC, B3 and CEAF F1.
  • CONLL-UA exists in two inter-convertible variants. The "compact" variant stores everything in the CONLL-U Misc column and is fully UD-compatible. The "exploded" variant uses separate columns.
  • The scorer natively handles the exploded format. A compact-format extension was built for the 2022 CRAC-CorefUD shared task on multilingual coreference.
  • The scorer was tested in the CODI-CRAC 2021 shared task and was planned for its 2022 edition.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Discourse analysis · Coreference resolution · Universal Dependencies

Cited by

14 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{yu2022universal,
  title = {{The Universal Anaphora Scorer}},
  author = {Yu, Juntao and Khosla, Sopan and Moosavi, Nafise Sadat and Paun, Silviu and Pradhan, Sameer and Poesio, Massimo},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference},
  year = {2022},
  doi = {10.63317/4uneskjaxipa},
  url = {https://doi.org/10.63317/4uneskjaxipa}
}

References

58 references, 30 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bagga, A. and Baldwin, B. (1998). Algorithms for scoring coreference chains. In The First International Conference on Language Resources and Evaluation (LREC) - Workshop on linguistics coreference, volume 1, pages 563–566. ACL. find our card
  2. Bamman, D., Lewke, O., and Mansoor, A. (2020). An annotated dataset of coreference in english literature. In Proc. of LREC. European Language Resources Association (ELRA), Association for Computational Linguistics (ACL). ↗ our card
  3. Clark, H. H. (1977). Bridging. In P. N. Johnson-Laird et al., editors, Thinking: Readings in Cognitive Science, pages 411–420. Cambridge University Press, London and New York. find
  4. Cohen, K. B., Lanfranchi, A., Choi, M. J.-y., Bada, M., Baumgartner Jr., W. A., Panteleyeva, N., Verspoor, K., Palmer, M., and Hunter, L. E. (2017). Coreference annotation and resolution in the Colorado Richly Annotated Full Text (CRAFT) corpus of biomedical journal articles. BMC Bioinformatics, 18(372). ↗
  5. Denis, P. and Baldridge, J. (2009). Global joint models for coreference resolution and named entity classification. Procesamiento del Lenguaje Natural, 42:87– 96. find
  6. Eschenbach, C., Habel, C., Herweg, M., and Rehkämper, K. (1989). Remarks on plural anaphora. In Proceedings of the fourth conference on European chapter of the Association for Computational Linguistics, pages 161–167. Association for Computational Linguistics. find our card
  7. Hawkins, J. A. (1978). Definiteness and Indefiniteness. Croom Helm, London. find
  8. Hou, Y., Markert, K., and Strube, M. (2018). Unrestricted bridging resolution. Computational Linguistics, 44(2):237–284. ↗ our card
  9. Hou, Y. (2020). Bridging anaphora resolution as question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1428–1438, Online, July. Association for Computational Linguistics. find our card
  10. Joshi, M., Chen, D., Liu, Y., Weld, D. S., Zettlemoyer, L., and Levy, O. (2020). Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. ↗ our card
  11. Kamp, H. and Reyle, U. (1993). From Discourse to Logic. D. Reidel, Dordrecht. ↗
  12. Kantor, B. and Globerson, A. (2019). Coreference resolution with entity equalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 673–677, Florence, Italy, July. Association for Computational Linguistics. find our card
  13. Khosla, S., Yu, J., Manuvinakurike, R., Ng, V., Poesio, M., Strube, M., and Rosé, C. (2021). The codi-crac 2021 shared task on anaphora, bridging, and discourse deixis in dialogue. In Proc. of the CODI/CRAC Shared Task Workshop. find our card
  14. Kobayashi, H. and Ng, V. (2021). Bridging resolution: Making sense of the state of the art. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1652–1659, Online, June. Association for Computational Linguistics. find our card
  15. Kolhatkar, V. and Hirst, G. (2014). Resolving shell nouns. In Proc. of EMNLP, pages 499–510, Doha, Qatar. find
  16. Kolhatkar, V., Roussel, A., Dipper, S., and Zinsmeister, H. (2018). Anaphora with non-nominal antecedents in computational linguistics: a Survey. Computational Linguistics, 44(3):547–612. ↗ our card
  17. Kuhn, H. W. (1955). The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83–97. ↗
  18. Lee, K., He, L., Lewis, M., and Zettlemoyer, L. (2017). End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188– 197, Copenhagen, Denmark, September. Association for Computational Linguistics. find our card
  19. Lee, K., He, L., and Zettlemoyer, L. (2018). Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–692. find our card
  20. Liu, Q., Jiang, H., Evdokimov, A., Ling, Z.-H., Zhu, X., Wei, S., and Hu, Y. (2017). Cause-effect knowledge acquisition and neural association model for solving a set of winograd schema problems. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 2344–2350. find
  21. Lu, J. and Ng, V. (2018). Event coreference resolution: A survey of two decades of research. In Proc. of IJCAI, pages 5479–5486. find
  22. Luo, X. and Pradhan, S. (2016). Evaluation metrics. In Massimo Poesio, et al., editors, Anaphora Resolution: Algorithms, Resources and Applications, pages 147–170. Springer. find
  23. Luo, X., Pradhan, S., Recasens, M., and Hovy, E. (2014). An extension of BLANC to system mentions. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 24–29, Baltimore, Maryland, June. Association for Computational Linguistics. find
  24. Luo, X. (2005). On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25–32, Vancouver, British Columbia, Canada, October. Association for Computational Linguistics. find our card
  25. Marasović, A., Born, L., Opitz, J., and Frank, A. (2017). A mention-ranking model for abstract anaphora resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 221–232, Copenhagen, Denmark, September. Association for Computational Linguistics. find our card
  26. Moosavi, N. S. and Strube, M. (2016). A proposal for a link-based entity aware metric. In Proc. of ACL, pages 632–642, Berlin. find
  27. Müller, M.-C. and Strube, M. (2006). Multi-level annotation of linguistic data with mmax2. In S. Braun, et al., editors, Corpus Technology and Language Pedagogy. New Resources, New Tools, New Methods, volume 3 of English Corpus Linguistics, pages 197–214. Peter Lang. find
  28. Munkres, J. (1957). Algorithms for the assignment and transportation problems. Journal of the society for industrial and applied mathematics, 5(1):32–38. ↗
  29. Muzerelle, J., Lefeuvre, A., Schang, E., Antoine, J.-Y., Pelletier, A., Maurel, D., Eshkol, I., and Villaneau, J. (2014). Ancor centre, a large free spoken french coreference corpus. In Proc. of LREC. find
  30. Nedoluzhko, A. (2013). Generic noun phrases and annotation of coreference and bridging relations in the prague dependency treebank. In Proc. of LAW, pages 103–111. find
  31. O’Gorman, T., Regan, M., Griffitt, K., Hermjakob, U., Knight, K., and Palmer, M. (2018). Amr beyond the sentence: the multi-sentence amr corpus. In Proc. of COLING, pages 3693–3702, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  32. Paun, S., Yu, J., Moosavi, N., and Poesio, M. (2021). Scoring coreference chains with split-antecedent anaphors and other entities constructed from a discourse model. Submitted. find our card
  33. Poesio, M. and Artstein, R. (2005). The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In A. Meyers, editor, Proc. of ACL Workshop on Frontiers in Corpus Annotation, pages 76–83, June. find our card
  34. Poesio, M. and Rieser, H. (2010). Completions, coordination, and alignment in dialogue. Dialogue and Discourse, 1(1):1–89. ↗
  35. Poesio, M., Mehta, R., Maroudas, A., and Hitzeman, J. (2004). Learning to solve bridging references. In Proc. of ACL, pages 143–150, Barcelona, July. find
  36. Poesio, M., Grishina, Y., Kolhatkar, V., Moosavi, N., Roesiger, I., Roussel, A., Simonjetz, F., Uma, A., Uryupina, O., Yu, J., and Zinsmeister, H. (2018). Anaphora resolution with the ARRAU corpus. In Proceedings of the First Workshop on Computational Models of Reference, Anaphora and Coreference, pages 11–22, New Orleans, Louisiana, June. Association for Computational Linguistics. find our card
  37. Poesio, M., Chamberlain, J., Kruschwitz, U., Paun, S., Uma, A., and Yu, J. (2019). A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proc. of NAACL, page 1778–1789, Minneapolis. Association for Computational Linguistics (ACL). ↗ our card
  38. Poesio, M. (2016). Linguistic and cognitive evidence about anaphora. In M. Poesio, et al., editors, Anaphora Resolution: Algorithms, Resources and Applications, chapter 2. Springer. ↗
  39. Pradhan, S., Moschitti, A., Xue, N., Uryupina, O., and Zhang, Y. (2012). CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the Sixteenth Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea. find our card
  40. Pradhan, S., Luo, X., Recasens, M., Hovy, E., Ng, V., and Strube, M. (2014). Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 30–35, Baltimore, Maryland, June. Association for Computational Linguistics. find our card
  41. Prince, E. F. (1981). Toward a taxonomy of given-new information. In P. Cole, editor, Radical Pragmatics, pages 223–256. Academic Press, New York. find
  42. Rahman, A. and Ng, V. (2012). Resolving complex cases of definite pronouns: The Winograd schema challenge. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 777–789, Jeju Island, Korea, July. Association for Computational Linguistics. find our card
  43. Recasens, M. and Hovy, E. (2011). Blanc: Implementing the rand index for coreference evaluation. Natural Language Engineering. ↗
  44. Recasens, M. and Martı́, M. A. (2010). AnCora-CO: Coreferentially annotated corpora for Spanish and Catalan. Language Resources and Evaluation, 44(4):315–345. ↗
  45. Recasens, M., Màrquez, L., Sapena, E., Martı́, M. A., Taulé, M., Hoste, V., Poesio, M., and Versley, Y. (2010). Semeval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 1–8, Uppsala, Sweden, July. find our card
  46. Sakaguchi, K., Le Bras, R., Bhagavatula, C., and Choi, Y. (2020). Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34:8732–8740, Apr. find
  47. Uryupina, O., Artstein, R., Bristot, A., Cavicchio, F., Delogu, F., Rodriguez, K. J., and Poesio, M. (2020). Annotating a broad range of anaphoric phenomena, in a variety of genres: the ARRAU corpus. Journal of Natural Language Engineering. ↗
  48. Vala, H., Piper, A., and Ruths, D. (2016). The more antecedents, the merrier: Resolving multi-antecedent anaphors. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2287–2296, Berlin, Germany, August. Association for Computational Linguistics. find our card
  49. Vieira, R. and Poesio, M. (2000). An empirically-based system for processing definite descriptions. Computational Linguistics, 26(4):539–593, December. ↗
  50. Vilain, M., Burger, J., Aberdeen, J., Connolly, D., and Hirschman, L. (1995). A model-theoretic coreference scoring scheme. In Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland, November 6-8, 1995. find our card
  51. Webber, B. L. (1991). Structure and ostension in the interpretation of discourse deixis. Language and Cognitive Processes, 6(2):107–135. ↗
  52. Webster, K., Recasens, M., Axelrod, V., and Baldridge, J. (2018). Mind the GAP: A balanced corpus of gendered ambiguous pronouns. Transactions of the Association for Computational Linguistics, 6:605–617. ↗ our card
  53. Wiseman, S., Rush, A. M., Shieber, S., and Weston, J. (2015). Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1416–1426, Beijing, China, July. Association for Computational Linguistics. find our card
  54. Yu, J. and Poesio, M. (2020). Multitask learning based neural bridging reference resolution. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3534–3546, Barcelona, Spain (Online), December. International Committee on Computational Linguistics. ↗ our card
  55. Yu, J., Moosavi, N. S., Paun, S., and Poesio, M. (2020). Free the plural: Unrestricted split-antecedent anaphora resolution. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6113–6125, Barcelona, Spain (Online), December. International Committee on Computational Linguistics. find our card
  56. Yu, J., Moosavi, N. S., Paun, S., and Poesio, M. (2021). Stay together: A system for single and split-antecedent anaphora resolution. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics. find our card
  57. Zeldes, A. (2017). The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. ↗
  58. Zhou, E. and Choi, J. D. (2018). They exist! introducing plural mentions to coreference resolution and entity linking. In Proceedings of the 27th International Conference on Computational Linguistics, pages 24–34, Santa Fe, New Mexico, USA, August. Association for Computational Linguistics. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/yu-2022-universal-anaphora-scorer/