Elamur Research⌕
☰
home / papers / 2021 / OntoGUM: Evaluating Contextualized SOTA Coreference Resolution on 12 More Genres

OntoGUM: Evaluating Contextualized SOTA Coreference Resolution on 12 More Genres

Summary

Neural coreference systems score highly on OntoNotes, but there has been little data in the same annotation scheme to test whether they work on other kinds of text. Earlier out-of-domain tests used WikiCoref and GAP, which are small, limited to Wikipedia, and in GAP's case cover only name–pronoun links. The authors built OntoGUM by automatically converting the 12-genre GUM corpus into the OntoNotes scheme. The conversion uses deterministic rules that draw on GUM's gold syntax and other annotation layers. It removes relation and markable types that OntoNotes excludes, such as singletons, bridging, compounds and indefinite anaphors, and it adjusts appositive and verbal spans. An agreement study found the converted data about as consistent as human annotation in OntoNotes. Two systems were trained or developed on OntoNotes and tested on OntoGUM: the rule-based Stanford dcoref and e2e+SpanBERT-large. Both lost substantial accuracy. Performance depended strongly on genre, with pronoun-heavy texts that have gold speaker information scoring best. This suggests contextualized LMs do not remove covert lexical overfitting, and OntoGUM offers a more realistic benchmark.

Key points

  • OntoGUM contains 168 documents (~150K tokens) in 12 genres, with 19,378 mentions and 4,471 chains. It is the largest open gold dataset that follows OntoNotes, and the data and code are on GitHub (yilunzhu/ontogum).
  • Conversion quality was checked on 3 documents (2,500 tokens, 371 mentions):
    • span detection scored ~96 and the CoNLL score was ~92;
    • only 8 conversion errors were found, plus 2 inherited from GUM.
  • e2e+SpanBERT dropped about 15 CoNLL points (19%) on OntoGUM, reaching an average of 64.6. dcoref dropped by more than 18 points (30%).
  • Spoken and pronoun-rich genres with gold speakers (vl, it, cn, sp, fc) scored highest. Even the SOTA model struggled with nw, wh and tx, so recent Wikinews articles were hard for it despite the newswire content in OntoNotes.
  • Results use the official CoNLL-2012 scorer, which averages MUC, B3 and CEAFφ4 F1.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

20 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 9 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.acl-short.59 · arXiv 2106.00933

@inproceedings{zhu2021ontogum,
  title = {{OntoGUM: Evaluating Contextualized SOTA Coreference Resolution on 12 More Genres}},
  author = {Zhu, Yilun and Pradhan, Sameer and Zeldes, Amir},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)},
  year = {2021},
  doi = {10.18653/v1/2021.acl-short.59},
  eprint = {2106.00933},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2021.acl-short.59}
}

References

28 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A large-scale dataset in preschool vocabulary for coreference resolution. In Proceedings of EMNLP 2018, pages 172–181, Brussels. find our card
  2. Kevin Clark and Christopher D. Manning. 2015. Entity-centric coreference resolution with model stacking. In Proceedings of ACL-IJCNLP 2015, Long Papers, pages 1405–1415, Beijing, China. find our card
  3. Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256–2262, Austin, Texas. find our card
  4. Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of ACL 2016, Long Papers, pages 643–653, Berlin. find our card
  5. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL 2019, pages 4171–4186, Minneapolis, MN. find
  6. Greg Durrett and Dan Klein. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of EMNLP 2013, pages 1971–1982, Seattle, WA. find our card
  7. Greg Durrett and Dan Klein. 2014. A joint model for entity analysis: Coreference, typing, and linking. TACL, 2:477–490. ↗ our card
  8. Hongliang Fei, Xu Li, Dingcheng Li, and Ping Li. 2019. End-to-end deep reinforcement learning based coreference resolution. In Proceedings of ACL 2019, pages 660–665, Florence, Italy. find our card
  9. Abbas Ghaddar and Phillippe Langlais. 2016. Wiki-Coref: An English coreference-annotated corpus of Wikipedia articles. In Proceedings of LREC 2016), pages 136–142, Portorož, Slovenia. find our card
  10. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019a. SpanBERT: Improving pre-training by representing and predicting spans. CoRR, abs/1907.10529. ↗ our card
  11. Mandar Joshi, Omer Levy, Daniel S. Weld, and Luke Zettlemoyer. 2019b. BERT for coreference resolution: Baselines and analysis. In Proceedings of EMNLP-IJCNLP 2019, pages 5803–5808, Hong Kong, China. find our card
  12. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885–916. ↗ our card
  13. Nafise Sadat Moosavi and Michael Strube. 2017. Lexical features in coreference resolution: To be used with caution. In Proceedings of ACL 2017, Short Papers, pages 14–19, Vancouver, Canada. find our card
  14. Nafise Sadat Moosavi and Michael Strube. 2018. Using linguistic features to improve the generalization capability of neural coreference resolvers. In Proceedings of EMNLP 2018, pages 193–203, Brussels. find our card
  15. Massimo Poesio, Yulia Grishina, Varada Kolhatkar, Nafise Moosavi, Ina Roesiger, Adam Roussel, Fabian Simonjetz, Alexandra Uma, Olga Uryupina, Juntao Yu, and Heike Zinsmeister. 2018. Anaphora resolution with the ARRAU corpus. In Proceedings of CRAC 2018, pages 11–22, New Orleans, LA. find our card
  16. Massimo Poesio, Roland Stuckardt, and Yannick Versley. 2016. Anaphora resolution. Springer. find
  17. Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of ACL 2014, pages 30–35, Baltimore, MD. find our card
  18. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Proceedings of CoNLL 2013, pages 143–152, Sofia. find
  19. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40, Jeju Island, Korea. find our card
  20. Marta Recasens, Marie-Catherine de Marneffe, and Christopher Potts. 2013. The life and death of discourse entities: Identifying singleton mentions. In Proceedings of NAACL 2013, pages 627–633, Atlanta, Georgia. find
  21. Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018. Mind the GAP: A balanced corpus of gendered ambiguous pronouns. TACL, pages 605–617. ↗ our card
  22. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of NAACL 2018, Short Papers, pages 687–692, New Orleans, LA. find our card
  23. Ralph Weischedel, Eduard Hovy, Mitchell Marcus, Martha Palmer, Robert Belvin, Sameer Pradhan, Lance Ramshaw, and Nianwen Xue. 2011. OntoNotes: A large training corpus for enhanced processing. In Joseph Olive, Caitlin Christianson, and John McCary, editors, Handbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation. Springer. find
  24. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL 2014 System Demonstrations, pages 55–60. find our card
  25. Sam Wiseman, Alexander M. Rush, Stuart Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of ACL-IJCNLP 2015, Long Papers, pages 1416–1426, Beijing, China. find our card
  26. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of EMNLP 2017, pages 188– 197, Copenhagen, Denmark. find our card
  27. Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016. Learning global features for coreference resolution. In Proceedings of NAACL 2016, pages 994–1004, San Diego, CA. find our card
  28. Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581–612. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/zhu-2021-ontogum-evaluating-contextualized-sota-coreference/