Using Linguistic Features to Improve the Generalization Capability of Neural Coreference Resolvers

What it is about

Neural coreference resolvers that win on CoNLL-2012 lose their ranking on WikiCoref, which uses the same guidelines. The authors add classic linguistic features to deep-coref (mention type and fine type, named-entity type, gender, number, animacy, dependency relation, POS; pairwise agreement, string match, compatibility of pre- and post-modifiers, salience-based discourse features combining distance and subject/object role). Added wholesale they help little. Selecting only the informative (feature, value) combinations helps a lot: their EPM algorithm stores mention-pair samples in a frequent-pattern tree and keeps patterns that pass a G² likelihood-ratio test (p < 0.01), add information over their sub-patterns, and are frequent enough. The model with these patterns gains in-domain and gives the best out-of-domain results reported at the time on WikiCoref and other sets, on par with a system built specifically for WikiCoref.

Ideas

I1. Use EPM-style mining to choose feature conjunctions for prag's perceptron, which prag/README.md lists as a next step (pairs such as aux1×sperson, rform×you): enumerate slot=value pairs and triples over the silver, keep those whose association with a label passes a G² test and adds over their parts, and add only these as conjunction features. Measure act and indirect macro-F1 in prag eval with the shuffled-label control; the mined list itself is an explanation of what the model may use.
I2. For coreference, mine discriminative patterns over mention-pair features (mention type, agreement, Hobbs rank, sieve that fired, distance bucket) to find conjunctions that predict wrong links of the current sieves, e.g. «it + candidate is a time/number + distance > 1 sentence». Mining must run on licence-clean data or the teacher's silver; on GUM it may only diagnose, not supply rules (GUM is NC and dev-only). Turn each pattern that is linguistically sensible into a hand-written blocker in coref/src/sieve.rs and measure its precision on GUM dev.
I3. Prefer the paper's four carefully built pairwise features (agreement, modifier compatibility, salience-weighted distance) over raw lexical ones in any future ranker; they carry across domains.

For Mova

languages/en/35-reference-and-coreference.md §8–9; 31-speech-acts.md §9 (act features). Code: prag/src/feats.rs and prag/src/model.rs (mined conjunctions), coref/src/sieve.rs (blockers). Measures: prag eval macro-F1; sieve precision and CoNLL F1 on GUM dev.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1018.pdf

Summary

Coreference resolvers are often applied to texts and domains that lack annotated coreference data, so out-of-domain robustness matters. Yet recent neural systems tuned on CoNLL transfer poorly. Their gains may partly reflect memorizing mentions shared between the CoNLL training and evaluation splits. Moosavi and Strube test whether linguistic features can restore generalization in a neural resolver, using deep-coref as the base system. Simply appending a full set of features extracted with Stanford CoreNLP barely changes results. The features cover string match, syntax, shallow semantics and discourse. The authors instead keep only informative (feature, value) pairs. They select these with EPM, a new discriminative pattern mining algorithm that matches competing miners in quality and scales better to large data. With this selection, the model trained only on CoNLL equals a WikiCoref-specific system on WikiCoref. It also beats the e2e-coref ensemble when whole CoNLL genres are held out. The main lesson is that linguistic knowledge helps neural models only if it is filtered for informativeness. Otherwise, strong lexical features overshadow it.

Key points

Adding all linguistic features to deep-coref on the CoNLL development set has little effect. CoNLL F1 changes from 66.09 to 66.17 for "ranking" and from 65.82 to 66.32 for "top-pairs".
deep-coref was chosen over e2e-coref because features such as head match need explicit mention heads. e2e-coref instead derives heads through attention.
On WikiCoref, after training on CoNLL, "+EPM" scores 55.30 CoNLL F1, compared with 52.52 for top-pairs and 53.63 for the e2e-coref ensemble. It roughly ties G&L (55.11), a system that relies on Wikipedia markup and Freebase links.
In genre hold-out tests, "+EPM" reaches 68.14 on pt (Bible) versus 65.45 for the e2e ensemble, and 61.52 on wb (weblog) versus 60.99.
In the ablation, pairwise feature-values matter most and POS/NER tags matter least. The EPM code is available at github.com/ns-moosavi/epm.

Topics: Coreference resolution

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1018 · arXiv 1708.00160

@inproceedings{moosavi2018using,
  title = {{Using Linguistic Features to Improve the Generalization Capability of Neural Coreference Resolvers}},
  author = {Moosavi, Nafise Sadat and Strube, Michael},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1018},
  eprint = {1708.00160},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/d18-1018}
}

References

41 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Micha Elsner and Eugene Charniak. 2008. Coreference-inspired coherence modeling. In Proceedings ACL-HLT 2008 Conference Short Papers, Columbus, Ohio, 15–20 June 2008, pages 41–44. find
Alan Agresti. 2007. An Introduction to Categorical Data Analysis. John Wiley & Sons. find
Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of the 1st International Conference on Language Resources and Evaluation, Granada, Spain, 28–30 May 1998, pages 563–566. find our card
Rong-En Fan, Kai-Wei Chang, Cho-Jui Hsieh, Xiang-Rui Wang, and Chih-Jen Lin. 2008. LIBLINEAR: A library for large linear classification. The Journal of Machine Learning Research, 9:1871–1874. find
Iyad Batal and Milos Hauskrecht. 2010. Constructing classification features using minimal predictive patterns. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management, pages 869–878. find
Eraldo R Fernandes and Ruy L Milidiú. 2012. Entropy-guided feature generation for structured learning of Portuguese dependency parsing. In Proceedings of the International Conference on Computational Processing of the Portuguese Language, pages 146–156. Springer. find
Eric Bengtson and Dan Roth. 2008. Understanding the value of features for coreference resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, Waikiki, Honolulu, Hawaii, 25–27 October 2008, pages 294–303. find our card
Eraldo Rezende Fernandes, Cı́cero Nogueira dos Santos, and Ruy Luiz Milidiú. 2012. Latent structure perceptron with feature induction for unrestricted coreference resolution. In Proceedings of the Shared Task of the 16th Conference on Computational Natural Language Learning, Jeju Island, Korea, 12–14 July 2012, pages 41–48. find our card
Anders Björkelund and Richárd Farkas. 2012. Data-driven multilingual coreference resolution using resolver stacking. In Proceedings of the Shared Task of the 16th Conference on Computational Natural Language Learning, Jeju Island, Korea, 12–14 July 2012, pages 49–55. find
Abbas Ghaddar and Philippe Langlais. 2016a. Coreference in Wikipedia: Main concept resolution. In Proceedings of the 20th Conference on Computational Natural Language Learning, Berlin, Germany, 7–11 August 2016, pages 229–238. find
Hong Cheng, Xifeng Yan, Jiawei Han, and Chih-Wei Hsu. 2007. Discriminative frequent pattern analysis for effective classification. In Proceedings of the IEEE 23rd International Conference on Data Engineering (ICDE 2007), pages 716–725. find
Abbas Ghaddar and Philippe Langlais. 2016b. Wiki-Coref: An English coreference-annotated corpus of Wikipedia articles. In Proceedings of the 10th International Conference on Language Resources and Evaluation, Portorož, Slovenia, 23–28 May 2016. find our card
Hong Cheng, Xifeng Yan, Jiawei Han, and Philip S Yu. 2008. Direct discriminative pattern mining for effective classification. In Proceedings of the IEEE 24th International Conference on Data Engineering (ICDE 2008), pages 169–178. find
Jiawei Han, Jian Pei, Yiwen Yin, and Runying Mao. 2004. Mining frequent patterns without candidate generation: A frequent-pattern tree approach. Data Mining and Knowledge Discovery, 8(1-5):53–87. find
Kevin Clark and Christopher D. Manning. 2016a. Improving coreference resolution by learning entity-level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, 7–12 August 2016. find our card
Mirella Lapata and Regina Barzilay. 2005. Automatic evaluation of text coherence: Models and representations. In Proceedings of the 19th International Joint Conference on Artificial Intelligence, Edinburgh, Scotland, 30 July – 5 August 2005, pages 1085–1090. find
Kevin Clark and Christopher D. Manning. 2016b. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Tex., 1–5 November 2016, pages 2256–2262. find our card
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188–197, Copenhagen, Denmark. find our card
Pascal Denis and Jason Baldridge. 2008. Specialized models and ranking for coreference resolution. In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing, Waikiki, Honolulu, Hawaii, 25–27 October 2008, pages 660– 669. find
M. Lichman. 2013. UCI machine learning repository. find
Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of the Human Language Technology Conference and the 2005 Conference on Empirical Methods in Natural Language Processing, Vancouver, B.C., Canada, 6–8 October 2005, pages 25–32. find our card
Ted Dunning. 1993. Accurate methods for the statistics of surprise and coincidence. Computational Linguistics, 19(1):61–74. find our card
Marc Segond and Christian Borgelt. 2011. Item set mining based on cover similarity. Advances in Knowledge Discovery and Data Mining, pages 493– 505. find
Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. find our card
Richard S. Sutton and Andrew G. Barto. 1998. Reinforcement learning: An introduction, volume 1. MIT press Cambridge. find
Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? A proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, 7–12 August 2016, pages 632–642. find our card
Olga Uryupina. 2007. Knowledge acquisition for coreference resolution. Ph.D. thesis, Saarland University. find
Olga Uryupina and Alessandro Moschitti. 2015. A state-of-the-art mention-pair model for coreference resolution. In Proceedings of STARSEM 2015: The Fourth Joint Conference on Lexical and Computational Semantics, Denver, Col., 4–5 June 2015, pages 289–298. find
Nafise Sadat Moosavi and Michael Strube. 2017a. Lexical features in coreference resolution: To be used with caution. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Vancouver, B.C., Canada, 30 July –4 August 2017. find our card
Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the 6th Message Understanding Conference (MUC-6), pages 45–52, San Mateo, Cal. Morgan Kaufmann. find our card
Nafise Sadat Moosavi and Michael Strube. 2017b. Use generalized representations, but do not forget surface features. In Proceedings of the 2nd Workshop on Coreference Resolution Beyond OntoNotes (CORBON 2017), pages 1–7, Valencia, Spain. find
Geoffrey I. Webb. 2006. Discovering significant patterns. Machine Learning, 68(1):1–39. find
Vincent Ng and Claire Cardie. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, Penn., 7–12 July 2002, pages 104–111. find our card
Eric W. Noreen. 1989. Computer Intensive Methods for Hypothesis Testing: An Introduction. Wiley, New York, N.Y. find
Sam Wiseman, Alexander M. Rush, Stuart Shieber, and Jason Weston. 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Beijing, China, 26–31 July 2015, pages 1416–1426. find our card
Simone Paolo Ponzetto and Michael Strube. 2006. Exploiting semantic role labeling, WordNet and Wikipedia for coreference resolution. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, New York, N.Y., 4–9 June 2006, pages 192–199. find
Xiaofeng Yang, Jian Su, Guodung Zhou, and Chew Lim Tan. 2004. Improving pronoun resolution by incorporating coreferential information of candidates. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics, Barcelona, Spain, 21–26 July 2004, pages 128– 135. find
Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Baltimore, Md., 22–27 June 2014, pages 30– 35. find our card
Dian Yu and Heng Ji. 2016. Unsupervised person slot filling based on graph mining. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Berlin, Germany, 7–12 August 2016, pages 44–53. find
Marta Recasens and Eduard Hovy. 2009. A deeper look into features for coreference resolution. In Proceedings of the 7th Discourse Anaphora and Anaphor Resolution, pages 29–42. find
Sebastian Schuster and Christopher D. Manning. 2016. Enhanced English universal dependencies: An improved representation for natural language understanding tasks. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France. find our card