Elamur ResearchβŒ•
☰
home / papers / 2021 / Understanding Mention Detector-Linker Interaction in Neural Coreference Resolution

Understanding Mention Detector-Linker Interaction in Neural Coreference Resolution

Summary

Wu and Gardner analyze why strong neural coreference systems still fall short of human performance. They study SpanBERT + c2f-coref, the leading member of the end-to-end model family, and separate its two parts: a mention detector that proposes candidate spans and a linker that chooses antecedents. Using the CoNLL-2012 and PreCo development sets, they build oracle detectors that add missing gold mentions, remove spurious ones, or do both. They also build an oracle linker that always picks the correct antecedent. Errors are categorized using the Kummerfeld and Klein (2013) taxonomy. Detectors are usually designed to favor recall, but the experiments show that precision matters more. Noisy candidates make the linker less confident even about correct links. However, a precise detector is hard to build because the detector cannot reliably judge anaphoricity. With the original candidates, an ideal linker would score almost perfectly, and most remaining linker mistakes involve pronouns. The authors suggest dedicated anaphoricity modules and better contextual reasoning as next steps.

Key points

  • Baseline: The SpanBERT-large + c2f-coref system (L=30, Ξ»=0.4, K=50, no speaker or genre metadata) reaches 79.17 coreference F1 on CoNLL-12 and 85.04 on PreCo. Its anaphoric-mention recall exceeds 96%, but precision is only 28.37% (CoNLL-12) and 39.23% (PreCo).
  • Precision vs. recall oracles (CoNLL-12): The comparison baseline is 78.28 F1.
    • Giving the linker perfect anaphoric precision raises F1 to 86.02 post-hoc and 89.98 with re-training.
    • Perfect recall yields only about 79.4–79.7 F1.
    • Increasing max span width L or Ξ» does not improve on the original setting.
  • Anaphoricity: A classifier trained explicitly to separate anaphoric mentions from singletons has a confusion index of 0.81, where 1 corresponds to random guessing. When the same text appears as both types within a document, the index rises to 0.997.
  • Linker ceiling: An oracle linker using the original candidates scores 97.07 F1 (CoNLL-12) and 98.69 (PreCo). Restricting it to coarse-pruned antecedents still gives 96.61 and 98.65.
  • Remaining errors: The authors manually inspected 150 conflated-entity errors on CoNLL-12, assuming a perfect detector. Of these, 109 involve pronouns, most often a pronoun linked to the wrong nominal (43 cases).

Our summary of the paper, in our own words β€” not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

5 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.crac-1.16 Β· arXiv 2009.09363

@inproceedings{wu2021understanding,
  title = {{Understanding Mention Detector-Linker Interaction in Neural Coreference Resolution}},
  author = {Wu, Zhaofeng and Gardner, Matt},
  booktitle = {Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference},
  year = {2021},
  doi = {10.18653/v1/2021.crac-1.16},
  eprint = {2009.09363},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.crac-1.16}
}

References

27 references, 22 of them with a card of ours. β†— leads to the work, Β«findΒ» searches for one we have no id for yet.

  1. Saliha Azzam, Kevin Humphreys, and Robert Gaizauskas. 1999. Using coreference chains for text summarization. In Coreference and Its Applications. find our card
  2. Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A large-scale dataset in preschool vocabulary for coreference resolution. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 172–181, Brussels, Belgium. Association for Computational Linguistics. Downloaded from https://preschool-lab.github.io/PreCo. β†— our card
  3. Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5925–5932, Hong Kong, China. Association for Computational Linguistics. β†— our card
  4. Luheng He, Kenton Lee, Omer Levy, and Luke Zettlemoyer. 2018. Jointly predicting predicates and arguments in neural semantic role labeling. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 364–369, Melbourne, Australia. Association for Computational Linguistics. find our card
  5. Zhengbao Jiang, Wei Xu, Jun Araki, and Graham Neubig. 2020. Generalizing natural language analysis through span-relation representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2120–2133, Online. Association for Computational Linguistics. find
  6. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Spanbert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. β†— our card
  7. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803–5808, Hong Kong, China. Association for Computational Linguistics. β†— our card
  8. Nikolaos Kolitsas, Octavian-Eugen Ganea, and Thomas Hofmann. 2018. End-to-end neural entity linking. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 519–529, Brussels, Belgium. Association for Computational Linguistics. β†—
  9. Jonathan K. Kummerfeld and Dan Klein. 2013. Error-driven analysis of challenges in coreference resolution. In Proceedings of the 2013 Conference on Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the conll-2011 shared task. In Proceedings of the fifteenth conference on computational natural language learning: Shared task, pages 28–34. Association for Computational Linguistics. find our card
  10. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. Association for Computational Linguistics. β†— our card
  11. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687–692, New Orleans, Louisiana. Association for Computational Linguistics. β†— our card
  12. Jing Lu and Vincent Ng. 2020. Conundrums in entity coreference resolution: Making sense of the state of the art. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6620–6631, Online. Association for Computational Linguistics. β†— our card
  13. Yi Luan, Dave Wadden, Luheng He, Amy Shah, Mari Ostendorf, and Hannaneh Hajishirzi. 2019. A general framework for information extraction using dynamic span graphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3036–3046, Minneapolis, Minnesota. Association for Computational Linguistics. β†— our card
  14. Sebastian Martschat and Michael Strube. 2015. Latent structures for coreference resolution. Transactions of the Association for Computational Linguistics, 3:405–418. β†— our card
  15. Nafise Sadat Moosavi and Michael Strube. 2016. Search space pruning: A simple solution for better coreference resolvers. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1005–1011, San Diego, California. Association for Computational Linguistics. find our card
  16. Haoruo Peng, Daniel Khashabi, and Dan Roth. 2015. Solving hard coreference problems. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 809– 819, Denver, Colorado. Association for Computational Linguistics. Australia. Association for Computational Linguistics. find our card
  17. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 1– 40. Association for Computational Linguistics. find our card
  18. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 15–20, New Orleans, Louisiana. Association for Computational Linguistics. β†— our card
  19. Marta Recasens, Marie-Catherine de Marneffe, and Christopher Potts. 2013. The life and death of discourse entities: Identifying singleton mentions. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 627–633, Atlanta, Georgia. Association for Computational Linguistics. find
  20. Sanjay Subramanian and Dan Roth. 2019. Improving generalization in coreference resolution via adversarial training. In Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019), pages 192–197, Minneapolis, Minnesota. Association for Computational Linguistics. β†— our card
  21. Zexuan Zhong and Danqi Chen. 2021. A frustratingly easy approach for entity and relation extraction. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 50–61, Online. Association for Computational Linguistics. β†— our card
  22. Olga Uryupina. 2009. Detecting anaphoricity and antecedenthood for coreference resolution. Procesamiento del Lenguaje Natural, 42. find
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  24. Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016. Learning global features for coreference resolution. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 994–1004, San Diego, California. Association for Computational Linguistics. find our card
  25. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6953–6963, Online. Association for Computational Linguistics. β†— our card
  26. Liyan Xu and Jinho D. Choi. 2020. Revealing the myth of higher-order inference in coreference resolution. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8527–8533, Online. Association for Computational Linguistics. β†— our card
  27. Rui Zhang, CΓ­cero Nogueira dos Santos, Michihiro Yasunaga, Bing Xiang, and Dragomir Radev. 2018. Neural coreference resolution with deep biaffine attention by joint mention detection and mention clustering. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 102–107, Melbourne, *Figure 1: The PreCo anaphoric mention classifier confusion index (Β§5.1) on spans with different widths.* β†— our card

↑ back to top

This page in Markdown β†’ https://md.research.elamur.ai/free/papers/wu-2021-understanding-mention-detector-linker-interaction/