Elamur Research
☰
home / papers / 2023 / AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference

AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference

What it is about

The paper asks whether coreference models react to pronoun ambiguity as people do. It builds AmbiCoref, a corpus of minimal sentence pairs: an ambiguous and an unambiguous sentence differ only in one verb phrase (for example "Abby told Brittney that she upset Courtney" against "hugged"). The templates come from four psycholinguistic effects (experiencer constraints for objects and subjects, implicit causality, transfer of possession) and use VerbNet classes; the corpus has over 96,000 sentences. In a crowdsourced study of 625 sentences with three judgements each, unambiguous items get the intended referent (likely or definitely) for 83.2%, 91.9%, 85.8% and 68.3% of cases in the four types, while ambiguous items show a clear shift toward uncertainty. Five models (three CoreNLP variants, SpanBERT, NeuralCoref 4.0) are compared by the Earth Mover's Distance between their cluster-case distributions for matched pairs. Four of the five hardly change (mean EMD 0.6 to 4.0); SpanBERT changes most (11.7).

Ideas

For Mova

Code: coref/src/sieve.rs (case output and abstention), coref/src/eval.rs (the minimal-pair measure), and prag for a clarification response. Chapters: languages/en/35-reference-and-coreference.md, languages/en/33-implicature-and-presupposition.md. Measure: the EMD-style shift between ambiguous and unambiguous pairs, run twice, with a shuffled-verb negative control that should show no shift.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.findings-eacl.75.pdf

Summary

Coreference systems typically assign every pronoun to an antecedent, even when people would find the reference unclear and ask for clarification. The authors test whether current models notice such ambiguity. They built AmbiCoref, a synthetic English diagnostic corpus of minimal sentence pairs. In each pair, swapping only the verb phrase turns a pronoun with a clear antecedent into one with an unclear antecedent. The templates draw on psycholinguistic findings about experiencer verbs, implicit causality and transfer of possession. Verbs were chosen manually from VerbNet classes, and the templates were filled with names and noun phrases.

Crowdworkers resolved the unambiguous items as intended and were noticeably less confident on the ambiguous ones. Five coreference systems trained on CoNLL 2012 mostly produced the same output for both versions. Only SpanBERT changed its predictions to any measurable degree. The results suggest that models need finer-grained annotation or better calibration to respond to ambiguity the way humans do.

Key points

  • AmbiCoref contains more than 96 thousand sentences. They come from four template families (ECO, ECS, IC, TOP), which make up six ambiguous/unambiguous template pairs.
  • Template slots were filled as follows:
    • Verbs come from VerbNet classes 31 (psych verbs), 13 (change of possession) and 37 (communication).
    • Names were sampled from US popular-name lists.
    • Gendered noun phrases come from WinoBias.
  • In the Mechanical Turk study, each instance received 3 judgments. On unambiguous items, annotators picked the intended referent 83.2% (ECO), 91.9% (ECS), 85.8% (IC) and 68.3% (TOP) of the time. Ambiguous items shifted toward low-confidence or "ambiguous" answers.
  • Five models were evaluated: SpanBERT, NeuralCoref 4.0, and the deterministic, statistical and neural CoreNLP systems. A template was excluded for a model if the model resolved fewer than 40% of its unambiguous cases correctly. TOP was frequently excluded under this rule.
  • The authors measured how much each model's output distribution changed between ambiguous and unambiguous versions, using mean Earth Mover's Distance:
ModelMean EMD
SpanBERT11.7%
NeuralCoref 4.04.0%
CoreNLP Neural3.5%
CoreNLP Statistical1.2%
CoreNLP Deterministic0.6%

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-eacl.75 · arXiv 2302.00762

@inproceedings{yuan2023ambicoref,
  title = {{AmbiCoref: Evaluating Human and Model Sensitivity to Ambiguous Coreference}},
  author = {Yuan, Yuewei and Malaviya, Chaitanya and Yatskar, Mark},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-eacl.75},
  eprint = {2302.00762},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.findings-eacl.75}
}

References

28 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Laura Aina and Tal Linzen. 2021. The language model understood the prompt was ambiguous: Probing syntactic uncertainty through generation. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 42– 57, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  2. Alfonso Caramazza, Ellen Grober, Catherine Garvey, and Jack Yates. 1977. Comprehension of anaphoric pronouns. Journal of Verbal Learning and Verbal Behavior, 16(5):601–609. find
  3. Kevin Clark and Christopher D Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1405–1415. find our card
  4. Kevin Clark and Christopher D. Manning. 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256–2262, Austin, Texas. Association for Computational Linguistics. find our card
  5. Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 295–302, Online. Association for Computational Linguistics. find
  6. Anca Dumitrache, Lora Aroyo, and Chris Welty. 2019. A crowdsourced frame disambiguation corpus with ambiguity. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2164–2170, Minneapolis, Minnesota. Association for Computational Linguistics. find
  7. Meiqi Guo, Mingda Zhang, Siva Reddy, and Malihe Alikhani. 2021. Abg-coqa: Clarifying ambiguity in conversational question answering. In 3rd Conference on Automated Knowledge Base Construction. find
  8. Sarah L Haywood, Martin J Pickering, and Holly P Branigan. 2005. Do speakers avoid ambiguities during dialogue? Psychological Science, 16(5):362– 366. find
  9. Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021. How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering. Transactions of the Association for Computational Linguistics, 9:962–977. find
  10. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. find our card
  11. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803–5808, Hong Kong, China. Association for Computational Linguistics. ↗ our card
  12. H. Rohde and A. Kehler. 2014. Grammatical and information-structural influences on pronoun production. Language, Cognition and Neuroscience, 29(8):912–927. find
  13. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885–916. find our card
  14. Hinrich Schutze. 1995. Ambiguity in language learning: computational and cognitive models. Stanford University. find
  15. Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55–60, Baltimore, Maryland. Association for Computational Linguistics. ↗ our card
  16. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783– 5797, Online. Association for Computational Linguistics. ↗
  17. Ofir Pele and Michael Werman. 2009. Fast and robust earth mover’s distances. In 2009 IEEE 12th International Conference on Computer Vision, pages 460–467. IEEE. find
  18. Steven T. Piantadosi, Harry Tily, and Edward Gibson. 2012. The communicative function of ambiguity in language. Cognition, 122(3):280–291. find
  19. Massimo Poesio and Ron Artstein. 2005. The reliability of anaphoric annotation, reconsidered: Taking ambiguity into account. In Proceedings of the Workshop on Frontiers in Corpus Annotations II: Pie in the Sky, pages 76–83, Ann Arbor, Michigan. Association for Computational Linguistics. find
  20. Massimo Poesio, Jon Chamberlain, Silviu Paun, Juntao Yu, Alexandra Uma, and Udo Kruschwitz. 2019. A crowdsourced corpus of multiple judgments and disagreement on anaphoric interpretation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1778–1789, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  21. Karin Kipper Schuler. 2005. VerbNet: A broad-coverage, comprehensive verb lexicon. University of Pennsylvania. find
  22. F. Springston. 1976. Verb-derived constraints in the comprehension of anaphoric pronouns. Paper presented at the Eastern Psychological Association (1976). find
  23. Shiva Upadhye, Leon Bergen, and Andrew Kehler. 2020. Predicting reference: What do language models learn about discourse models? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 977–982, Online. Association for Computational Linguistics. find
  24. Thomas Wasow, Amy Perfors, and David Beaver. 2003. The puzzle of ambiguity. find
  25. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. Linguistic Data Consortium, Philadelphia, PA, 23. find
  26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtow-icz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45, Online. Association for Computational Linguistics. ↗ our card
  27. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. find our card
  28. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. Association for Computational Linguistics. find our card