Elamur Research⌕
☰
home / papers / 2023 / How to Evaluate Coreference in Literary Texts?

How to Evaluate Coreference in Literary Texts?

Paper · 2023 · Ana-Isabel Duron-Tejedor, Pascal Amsili and Thierry Poibeau

Summary

The paper argues that standard coreference metrics are poorly suited to long fiction and that a single aggregate number hides more than it reveals. The authors review the main scoring schemes, both link-based and mention-based. Each has documented blind spots, and on the same input they often disagree in ways nobody can easily interpret. They then compare OntoNotes, the usual benchmark, with a full French novel, Manon Lescaut. The novel has very few huge chains, and one character dominates the mentions, so benchmark results say little about performance on novels. Their proposal is to split the evaluation by the kind of chain:

  • long chains anchored by a named entity, which are usually the main characters;
  • shorter chains, mostly secondary or background figures;
  • singletons, treated as a separate question of how many characters a novel contains.

Existing metrics can then be computed inside each group. Comparing systems with one number becomes harder, but the authors note that such comparisons were already unreliable. The payoff is diagnostic output that digital humanists can actually act on.

Key points

  • Metrics surveyed: MUC, BLANC, B3, CEAF (CEAFe and CEAFm) and LEA. Each has its own blind spot:
    • MUC ignores singletons.
    • BLANC rises with extra non-coreferent links.
    • CEAF's one-to-one alignment discards correct decisions outside the mapping.
    • LEA fails on "triangular inequality" configurations.
  • On Pradhan et al.'s toy example, the metrics give F-scores of 0.4 (MUC), 0.46 (B3), 0.52 (CEAF) and 0.36 (BLANC), a spread of 0.16.
  • Average OntoNotes document (over 3,493 documents): 56 mentions, 13 chains, about 4.1 mentions per chain, no singletons, 466 tokens.
  • Manon Lescaut: 77,232 tokens, 11,888 mentions, 143 chains and 56 singletons. That gives 83 mentions per chain without singletons and 59 with them. Nearly half of all mentions refer to the Chevalier Des Grieux.
  • The authors reject the CoNLL score (a mean of MUC, B3 and CEAF) as averaging incommensurable quantities. On French LitBank, a breakdown of BookNLP output showed that long chains were being split into fragments.

Interesting ideas

  • I1. Instead of reporting one global coreference score, compute the usual metrics separately for main-character chains, minor-character chains and singletons, so each error type becomes visible on its own.
  • I2. Averaging metrics that measure different things produces a number with no clear meaning; separate, interpretable indicators are more useful for deciding what to fix.
  • I3. Counting a text's distinct characters (its "population") is treated as a task separate from linking mentions, and it is accepted that some cases, such as repeated unnamed "coachman" mentions, cannot be decided.
  • I4. Before trusting benchmark scores, compare simple statistics such as chain length, mentions per chain and singleton share between the benchmark and the target texts.
  • I5. Analysing the errors of a specific chain type (long chains breaking into pieces) pointed directly to the component that needed strengthening, namely entity clustering.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): The coreference module could report accuracy per stratum, especially protagonist chains versus one-off mentions, so we can tell whether pronoun resolution fails on heroes or on passers-by.
  • I2 (learn): Evaluation dashboards should keep per-metric and per-category figures apart rather than collapsing them into a composite score, which matches our emphasis on explainable steps.
  • I3 (world): The entity inventory of a text's world could be scored on its own, with an explicit "possibly the same entity" status for underdetermined descriptions instead of forcing a merge or a split.
  • I4 (answer): For fable and tale QA, we could check whether our coreference test data resembles stories built around a few recurring characters, since benchmark numbers may not transfer otherwise.
  • I5 (world): Detecting fragmented protagonist chains could trigger a merge or repair pass in entity tracking, for example linking a named character with later definite descriptions of them.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Centre National de la Recherche ScientifiqueParis · France168 works · 62,369 citations🔬École Normale Supérieure - PSLParis · France23 works · 4,357 citations🎓Université Paris Sciences et LettresParis · France21 works · 3,290 citations🎓Université Sorbonne NouvelleParis · France9 works · 569 citations🔬Langues, Textes, Traitements Informatiques, CognitionMontrouge · France7 works · 397 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksLiterature and Literary Theory · 519 works

🧭 Topics 1Research subjects and languages the work is about.Coreference resolution · 574 works

Cite

Cite as Ana-Isabel Duron-Tejedor, Pascal Amsili and Thierry Poibeau. 2023. How to Evaluate Coreference in Literary Texts?. arXiv:2401.00238.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.00238 · arXiv 2401.00238

@misc{durontejedor2023how,
  title = {{How to Evaluate Coreference in Literary Texts?}},
  author = {Ana-Isabel Duron-Tejedor, Pascal Amsili and Thierry Poibeau},
  year = {2023},
  doi = {10.48550/arxiv.2401.00238},
  eprint = {2401.00238},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.00238}
}

References

2 references, 2 of them with a card of ours. ↗ leads to the work.

  1. M. Poesio, J. Yu, S. Paun, A. Aloraini, P. Lu, J. Haber, D. Cokal, Computational models of anaphora, Annual Review of Linguistics 9 (2023) 561–587. URL: https://qmro.qmul.ac.uk/ xmlui/handle/123456789/83033. doi: 10.1146/annurev-linguistics-031120-111653 . D. Bamman, T. Underwood, N. A. Smith, A Bayesian mixed effects model of literary character, in: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Baltimore, Maryland, 2014, pp. 370–379. URL: https://aclanthology.org/P14-1035. doi: 10.3115/v1/P14-1035 . R. Grishman, B. Sundheim, Message understanding conference-6: A brief history, in: Proceedings of the 16th Conference on Computational Linguistics - Volume 1, COLING ’96, Association for Computational Linguistics, Stroudsburg, PA, USA, 1996, pp. 466–471. URL: http://dx.doi. org/10.3115/992628.992709. doi: 10.3115/992628.992709 . M. Borovikova, L. Grobol, A. Halftermeyer, S. Billot, A methodology for the comparison of human judgments with metrics for coreference resolution, in: Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 16–23. URL: https://aclanthology.org/2022.humeval-1.2. doi: 10.18653/v1/2022.humeval-1.2 . A. Lion-Bouton, L. Grobol, J.-Y. Antoine, S. Billot, A. Lefeuvre-Halftermeyer, Comment arpenter sans mètre : les scores de résolution de chaînes de coréférences sont-ils des métriques ? (do the standard scores of evaluation of coreference resolution constitute metrics ?), in: Actes de la 6e conférence conjointe Journées d’Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), ATALA et AFCP, Nancy, France, 2020, pp. 10–18. URL: https://aclanthology.org/2020.jeptalnrecital-eternal. N. S. Moosavi, M. Strube, Which coreference evaluation metric do you trust? a proposal for a link-based entity aware metric, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Berlin, Germany, 2016, pp. 632–642. URL: https://aclanthology.org/P16-1060. doi: 10.18653/v1/P16-1060 . N. S. Moosavi, Robustness in Coreference Resolution, University of Heidelberg, 2020. URL: http://www.ub.uni-heidelberg.de/archiv/27919. doi: 10.11588/heidok.00027919 . M. Vilain, J. Burger, J. Aberdeen, D. Connolly, L. Hirschman, A model-theoretic coreference scoring scheme, in: Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland, November 6-8, 1995, 1995, pp. 45–52. URL: https://aclanthology.org/M95-1005. M. Recasens, E. Hovy, Blanc: Implementing the rand index for coreference evaluation, Nat. ↗
  2. Lang. Eng. 17 (2011) 485–510. URL: https://doi.org/10.1017/S135132491000029X. doi: 10.1017/ S135132491000029X . X. Luo, S. Pradhan, M. Recasens, E. Hovy, An extension of BLANC to system mentions, in: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Baltimore, Maryland, 2014, pp. 24–29. URL: https://aclanthology.org/P14-2005. doi: 10.3115/v1/P14-2005 . A. Bagga, B. Baldwin, Algorithms for scoring coreference chains, in: The first international conference on language resources and evaluation workshop on linguistics coreference, volume 1, 1998, pp. 563–566. X. Luo, On coreference resolution performance metrics, in: Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Vancouver, British Columbia, Canada, 2005, pp. 25–32. URL: https://aclanthology.org/H05-1004. E. Hovy, M. Marcus, M. Palmer, L. Ramshaw, R. Weischedel, OntoNotes: The 90% solution, in: Proceedings of the Human Language Technology Conference of the NAACL, Companion Volume: Short Papers, Association for Computational Linguistics, New York City, USA, 2006, pp. 57–60. URL: https://aclanthology.org/N06-2015. S. Pradhan, X. Luo, M. Recasens, E. Hovy, V. Ng, M. Strube, Scoring coreference partitions of predicted mentions: A reference implementation, in: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Baltimore, Maryland, 2014, pp. 30–35. URL: https://aclanthology. org/P14-2006. doi: 10.3115/v1/P14-2006 . D. Bamman, O. Lewke, A. Mansoor, An annotated dataset of coreference in English literature, in: Proceedings of the Twelfth Language Resources and Evaluation Conference, European Language Resources Association, Marseille, France, 2020, pp. 44–54. URL: https://aclanthology. org/2020.lrec-1.6. P. Denis, J. Baldridge, Global joint models for coreference resolution and named entity classification, Procesamiento del lenguaje natural 42 (2009). S. Pradhan, A. Moschitti, N. Xue, O. Uryupina, Y. Zhang, CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes, in: Joint Conference on EMNLP and CoNLL - Shared Task, Association for Computational Linguistics, Jeju Island, Korea, 2012, pp. 1–40. URL: https://aclanthology.org/W12-4501. A. Woloch, The one and the Many, Minor characters and the Space in the Novel, Princeton University Press, Princeton, 2004. F. Lavocat, Les personnages rêvent aussi, Hermann, coll. "Fictions pensantes", Paris, 2020. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/duron-tejedor-2023-evaluate-coreference-literary-texts/