Elamur Research⌕
☰
home / papers / 2022 / Adapting Coreference Resolution Models through Active Learning

Adapting Coreference Resolution Models through Active Learning

Summary

Neural coreference resolution (CR) systems trained on OntoNotes 5.0 often perform poorly in new domains, where text style, entity meanings and annotation guidelines differ. Creating large in-domain training sets is expensive. The authors study how active learning can cut this cost when adapting a source model to a target domain. They use ICoref, an incremental, constant-memory variant of C2F-COREF, together with discrete annotations, where an annotator marks only the first antecedent of a selected span. They extend the clustered entropy strategy of Li et al. (2020) to also capture uncertainty about whether a span is a mention at all. They also examine the trade-off between reading new documents and labeling more spans within an already-read document. Both simulations on PreCo and QBCoref and a real-time user study with a custom interface show the same pattern: concentrating annotations in the same document is more effective than spreading them across many documents. The work offers practical guidance for building CR models quickly in low-resource domains.

Key points

  • The paper compares several uncertainty sampling strategies: ment-ent (mention detection entropy), clust-ent (clustering entropy), cond-ent (conditional entropy), joint-ent (joint entropy), and li-clust-ent (the original clustered entropy). Random and random-ment serve as baselines.
  • The best strategy depends on the domain:
    • On PreCo, with long documents and many singletons, mention-detection entropy (ment-ent) helps most. This is surprising, because that uncertainty is not directly tied to the training objective.
    • On QBCoref, where linking is hard, linking-oriented strategies work better.
  • Pure random sampling performs worst everywhere, while random-ment does much better, which underlines how important the mention detector is.
  • Some strategies suffer from sampling near-duplicates. li-clust-ent tends to pick nested spans, and cond-ent mostly selects pronouns that often belong to one cluster. For reference, ICoref trained on the full QBCoref training set reaches 0.795 avg F1.
  • Restricting annotation to one document raises labeling throughput and makes training less volatile. Code is available at github.com/forest-snow/incremental-coref.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

17 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 12 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.519 · arXiv 2104.07611

@inproceedings{yuan2022adapting,
  title = {{Adapting Coreference Resolution Models through Active Learning}},
  author = {Yuan, Michelle and Xia, Patrick and May, Chandler and Durme, Benjamin Van and Boyd-Graber, Jordan},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.519},
  eprint = {2104.07611},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.acl-long.519}
}

References

39 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jordan T. Ash and Ryan P Adams. 2020. On warm-starting neural network training. In Proceedings of Advances in Neural Information Processing Systems. find
  2. Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In The first international conference on language resources and evaluation workshop on linguistics coreference. find our card
  3. David Bamman, Olivia Lewke, and Anya Mansoor. 2020. An annotated dataset of coreference in English literature. In Proceedings of the Language Resources and Evaluation Conference. find our card
  4. Hong Chen, Zhenhua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A large-scale dataset in preschool vocabulary for coreference resolution. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  5. Caroline Gasperin. 2009. Active learning for anaphora resolution. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing. find our card
  6. Anupam Guha, Mohit Iyyer, Danny Bouman, and Jordan Boyd-Graber. 2015. Removing the training wheels: A coreference dataset that entertains humans and challenges computers. In Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  7. Ajay Gupta, Devendra Verma, Sachin Pawar, Sangameshwar Patil, Swapnil Hingmire, Girish K Palshikar, and Pushpak Bhattacharyya. 2018. Identifying participant mentions and resolving their coreferences in legal court judgements. In International Conference on Text, Speech, and Dialogue. find
  8. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. find our card
  9. Siddharth Karamcheti, Ranjay Krishna, Li Fei-Fei, and Christopher Manning. 2021. Mind your outliers! investigating the negative impact of outliers on active learning for visual question answering. In Proceedings of the Association for Computational Linguistics. find
  10. Jin-Dong Kim, Ngan Nguyen, Yue Wang, Jun’ichi Tsujii, Toshihisa Takagi, and Akinori Yonezawa. 2012. The genia event and protein coreference tasks of the BioNLP shared task 2011. In BMC bioinformatics. find
  11. Jan-Christoph Klie, Richard Eckart de Castilho, and Iryna Gurevych. 2020. From zero to hero: Human-in-the-loop entity linking in low resource domains. In Proceedings of the Association for Computational Linguistics. find
  12. Jonathan K. Kummerfeld and Dan Klein. 2013. Error-driven analysis of challenges in coreference resolution. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  13. Ji-Ung Lee, Christian M. Meyer, and Iryna Gurevych. 2020. Empowering Active Learning to Jointly Optimize System and User Demands. In Proceedings of the Association for Computational Linguistics. find
  14. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  15. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Conference of the North American Chapter of the Association for Computational Linguistics. find our card
  16. David D. Lewis and William A. Gale. 1994. A sequential algorithm for training text classifiers. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. find
  17. Belinda Z Li, Gabriel Stanovsky, and Luke Zettlemoyer. 2020. Active learning for coreference resolution using discrete annotation. In Proceedings of the Association for Computational Linguistics. find our card
  18. Bing Liu, Harrisen Scells, Guido Zuccon, Wen Hua, and Genghong Zhao. 2021. ActiveEA: Active learning for neural entity alignment. In Proceedings of named entity recognition in clinical text. Journal of the American Medical Informatics Association, 26:1314–1322. find
  19. Empirical Methods in Natural Language Processing. find
  20. Ming Liu, Wray Buntine, and Gholamreza Haffari. 2018. Learning to actively learn neural machine translation. In Conference on Computational Natural Language Learning. find
  21. David Lowell, Zachary C. Lipton, and Byron C. Wallace. 2019. Practical obstacles to deploying active learning. In Proceedings of Empirical Methods in Natural Language Processing. find
  22. Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing. find our card
  23. Katerina Margatina, Giorgos Vernikos, Loïc Barrault, and Nikolaos Aletras. 2021. Active learning by acquiring contrastive examples. In Proceedings of Empirical Methods in Natural Language Processing. find
  24. Timothy Miller, Dmitriy Dligach, and Guergana Savova. 2012. Active learning for coreference resolution. In BioNLP: Proceedings of the 2012 Workshop on Biomedical Natural Language Processing. find our card
  25. Corbèn Poot and Andreas van Cranenburgh. 2020. A benchmark of rule-based and neural coreference resolution in Dutch novels and news. In Proceedings of the Third Workshop on Computational Models of Reference, Anaphora and Coreference. find our card
  26. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using OntoNotes. In Conference on Computational Natural Language Learning. find
  27. Piyush Rai, Avishek Saha, Hal Daumé III, and Suresh Venkatasubramanian. 2010. Domain adaptation meets active learning. In Conference of the North American Chapter of the Association for Computational Linguistics. find
  28. Mrinmaya Sachan, Eduard Hovy, and Eric P Xing. 2015. An active learning approach to coreference resolution. In International Joint Conference on Artificial Intelligence. find
  29. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of the Association for Computational Linguistics. find our card
  30. Zhaofeng Wu and Matt Gardner. 2021. Understanding mention detector-linker interaction for neural coreference resolution. In Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference. find
  31. Patrick Xia, João Sedoc, and Benjamin Van Durme. 2020. Incremental neural coreference resolution in constant memory. In Proceedings of Empirical Methods in Natural Language Processing. find our card
  32. Patrick Xia and Benjamin Van Durme. 2021. Moving on from OntoNotes: Coreference resolution model transfer. In Proceedings of Empirical Methods in Natural Language Processing. find
  33. Michelle Yuan, Hsuan-Tien Lin, and Jordan Boyd-Graber. 2020. Cold-start active learning through self-supervised language modeling. In Proceedings of Empirical Methods in Natural Language Processing. find
  34. Ye Zhang, Matthew Lease, and Byron C. Wallace. 2017. Active discriminative text representation learning. In Association for the Advancement of Artificial Intelligence. find
  35. Shanheng Zhao and Hwee Tou Ng. 2014. Domain adaptation with active learning for coreference resolution. In Proceedings of the 5th International Workshop on Health Text Mining and Information Analysis (Louhi). find our card
  36. Jingbo Zhu, Huizhen Wang, Tianshun Yao, and Benjamin K. Tsou. 2008. Active learning with sampling by uncertainty and density for word sense disambiguation and text classification. In Proceedings of International Conference on Computational Linguistics. find
  37. Burr Settles. 2009. Active learning literature survey. Technical report, University of Wisconsin-Madison Department of Computer Sciences. find our card
  38. Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Sixth Message Understanding Conference (MUC-6): Proceedings of a Conference Held in Columbia, Maryland. find our card
  39. Qiang Wei, Yukun Chen, Mandana Salimi, Joshua C Denny, Qiaozhu Mei, Thomas A Lasko, Qingxia Chen, Stephen Wu, Amy Franklin, Trevor Cohen, et al. 2019. Cost-aware active learning for find

↑ back to top

This page in Markdown →