Elamur Research⌕
☰
home / papers / 2020 / A Benchmark of Rule-Based and Neural Coreference Resolution in Dutch Novels and News

A Benchmark of Rule-Based and Neural Coreference Resolution in Dutch Novels and News

Summary

Dutch coreference resolution has lagged behind English, where end-to-end neural models now lead. Dutch tools have so far been rule-based or used classic machine learning. The authors compare two systems. The first is dutchcoref, a Dutch adaptation of the Stanford rule-based system (Lee et al., 2013). The second is e2e-Dutch, a variant of the Lee et al. (2018) end-to-end model that uses BERT token representations. Both are evaluated on SoNaR-1 (mostly news and Wikipedia, about one million words) and RiddleCoref (fragments of contemporary novels). For each corpus the authors created new splits that reserve 70% for training, making supervised learning possible.

The neural model performs best on news/Wikipedia, while the rule-based system does better on fiction. Follow-up analyses point to several confounding factors. The neural model is hurt by the small novel training set and by very long documents. The rule-based system is penalized by SoNaR-1's annotation conventions. The work shows how strongly domain, document length and annotation scheme affect such comparisons, and it argues for harmonizing Dutch coreference corpora.

Key points

  • The training data differ sharply in size: RiddleCoref has only 23 training documents, while SoNaR-1 has 581. Novel documents are much longer, averaging 295.78 sentences versus 64.28.
  • The two corpora use different parse inputs. For SoNaR-1, the gold Lassy Small treebank trees were aligned to the coreference annotations and enriched with gold NER and Alpino features. RiddleCoref relies on automatically predicted Alpino parses.
  • In a truncation experiment on the novels, e2e-Dutch scores fall as documents grow longer, under both CoNLL and LEA. dutchcoref shows no such trend.
  • Excluding singletons or supplying gold mentions shifts scores considerably, but the ranking of the two systems stays the same.
  • A manual inspection of SoNaR-1 found unlinked exact-match mentions, missing mentions and questionable markable boundaries. The code and models are released at github.com/andreasvc/crac2020.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

11 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2011.01615

@inproceedings{poot2020benchmark,
  title = {{A Benchmark of Rule-Based and Neural Coreference Resolution in Dutch Novels and News}},
  author = {Poot, Corbèn and Cranenburgh, Andreas van},
  booktitle = {Proceedings of the Third Workshop on Computational Models of Reference, Anaphora and Coreference},
  year = {2020},
  eprint = {2011.01615},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2011.01615}
}

References

38 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Bamman. 2017. Natural language processing for the long tail. In Proceedings of Digital Humanities. find
  2. David Bamman, Olivia Lewke, and Anya Mansoor. 2020. An annotated dataset of coreference in English literature. In Proceedings of The 12th Language Resources and Evaluation Conference, pages 44–54. ↗ our card
  3. Gosse Bouma, Walter Daelemans, Iris Hendrickx, Véronique Hoste, and Anne-Marie Mineur. 2007. The COREA-project: manual for the annotation of coreference in Dutch texts. Technical report, University of Groningen. find
  4. Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of EMNLP, pages 2256–2262. find our card
  5. Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of ACL, pages 643–653. find our card
  6. Andreas van Cranenburgh. 2019. A Dutch coreference resolution system with an evaluation on literary fiction. Computational Linguistics in the Netherlands Journal, 9:27–54. find
  7. Orphée De Clercq, Véronique Hoste, and Iris Hendrickx. 2011. Cross-domain Dutch coreference resolution. In Proceedings of RANLP, pages 186–193. find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, pages 4171–4186. find
  9. Eraldo Fernandes, Cı́cero dos Santos, and Ruy Milidiú. 2012. Latent structure perceptron with feature induction for unrestricted coreference resolution. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 41–48. find our card
  10. Rob van der Goot, Hessel Haagsma, and Dieke Oele. 2015. GroRef: Rule-based coreference resolution for Dutch. In CLIN26 shared task. find
  11. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of LREC. find our card
  12. Iris Hendrickx, Gosse Bouma, Frederik Coppens, Walter Daelemans, Veronique Hoste, Geert Kloosterman, Anne-Marie Mineur, Joeri van der Vloet, and Jean-Luc Verschelde. 2008a. A coreference corpus and resolution system for Dutch. In Proceedings of LREC. find our card
  13. Iris Hendrickx, Veronique Hoste, and Walter Daelemans. 2008b. Semantic and syntactic features for Dutch coreference resolution. In Computational Linguistics and Intelligent Text Processing, pages 351–361, Berlin, Heidelberg. Springer Berlin Heidelberg. find
  14. Véronique Hoste. 2005. Optimization issues in machine learning of coreference resolution. Ph.D. thesis, Universiteit Antwerpen. Faculteit Letteren en Wijsbegeerte. find
  15. Eduard Hovy, Mitchell Marcus, Martha Palmer, Lance Ramshaw, and Ralph Weischedel. 2006. OntoNotes: The 90% solution. In Proceedings of NAACL, pages 57–60. find our card
  16. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64–77. find our card
  17. Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of EMNLP-IJCNLP, pages 5807–5812. find our card
  18. Corina Koolen, Karina van Dalen-Oskam, Andreas van Cranenburgh, and Erica Nagelhout. 2020. Literary quality in the eye of the Dutch reader: The national reader survey. Poetics. ↗
  19. Jonathan K. Kummerfeld and Dan Klein. 2013. Error-driven analysis of challenges in coreference resolution. In Proceedings of EMNLP, pages 265–277. find our card
  20. Heeyoung Lee, Angel Chang, Yves Peirsman, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885–916. ↗ our card
  21. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In Proceedings of CoNLL, pages 28–34. find our card
  22. Heeyoung Lee, Mihai Surdeanu, and Dan Jurafsky. 2017a. A scaffolding approach to coreference resolution integrating statistical and rule-based models. Natural Language Engineering, 23(5):733– 762. ↗
  23. Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017b. End-to-end neural coreference resolution. In Proceedings of EMNLP, pages 188–197. find our card
  24. Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of NAACL, pages 687–692. find our card
  25. Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? A proposal for a link-based entity aware metric. In Proceedings of ACL, pages 632–642. find our card
  26. Gertjan van Noord. 2006. At last parsing is now operational. In TALN06. Verbum Ex Machina. Actes de la 13e conference sur le traitement automatique des langues naturelles, pages 20–42. find
  27. Gertjan van Noord, Ineke Schuurman, and Vincent Vandeghinste. 2006. Syntactic annotation of large corpora in STEVIN. In Proceedings of LREC. find
  28. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of EMNLP, pages 1532–1543. find
  29. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL, pages 2227–2237. find
  30. Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL - Shared Task, pages 1–40. find our card
  31. Sameer Pradhan, Lance Ramshaw, Mitchell Marcus, Martha Palmer, Ralph Weischedel, and Nianwen Xue. 2011. CoNLL-2011 shared task: Modeling unrestricted coreference in OntoNotes. In Proceedings of CoNLL, pages 1–27. find our card
  32. Marta Recasens, Lluı́s Màrquez, Emili Sapena, M. Antònia Martı́, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010. SemEval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of SemEval, pages 1–8. find our card
  33. Ineke Schuurman, Véronique Hoste, and Paola Monachesi. 2010. Interacting semantic layers of annotation in SoNaR, a reference corpus of contemporary written Dutch. In Proc. of LREC, pages 2471–2477. find
  34. Gertjan Van Noord. 2009. Huge parsed corpora in Lassy. In Proceedings of TLT7, Groningen, The Netherlands. LOT. find
  35. Wietse de Vries, Andreas van Cranenburgh, and Malvina Nissim. 2020. What’s so special about BERT’s layers? A closer look at the NLP pipeline in monolingual and multilingual models. In Findings of EMNLP. find
  36. Wietse de Vries, Andreas van Cranenburgh, Arianna Bisazza, Tommaso Caselli, Gertjan van Noord, and Malvina Nissim. 2019. BERTje: A Dutch BERT model. arXiv:1912.09582. ↗ our card
  37. Sam Wiseman, Alexander M. Rush, Stuart Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of ACL, pages 1416–1426. find our card
  38. Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. CorefQA: Coreference resolution as query-based span prediction. In Proceedings of ACL, pages 6953–6963. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/poot-2020-benchmark-rule-based-neural-coreference/