Elamur Research⌕
☰
home / papers / 2011 / Multi-metric optimization for coreference: The UniTN / IITP / Essex submission to the 2011 CONLL Shared Task

Multi-metric optimization for coreference: The UniTN / IITP / Essex submission to the 2011 CONLL Shared Task

Summary

Coreference resolution has no single agreed-upon evaluation metric, so the CoNLL 2011 Shared Task ranked systems by a combination of measures. The authors respond by tuning their system against several metrics at once. They use multi-objective optimization (MOO) with the genetic algorithm NSGA-II. Each binary chromosome decides which of 42 features to keep and whether to use one classifier for all anaphors or separate classifiers for pronouns and other nominals. The objectives are the F-scores of MUC and two CEAF variants. The base system is BART with a mention-pair model and Weka decision trees, and its mention detector was adapted to the OntoNotes annotation. Using every feature performed worse than simple Soon et al. and Ng & Cardie baselines on the development set. The MOO-selected configuration beat all baselines on every metric, raising the FINAL score by 2–5 points. The authors conclude that feature-rich coreference systems need automatic feature selection to benefit from their features.

Key points

  • The search space was 43 binary genes: 42 features (string match, agreement, syntax/binding, salience, Web patterns, etc.) plus one split/single classifier gene. NSGA-II settings were population 20, 20 generations, mutation 0.1 and crossover 0.9.
  • Because of the size of OntoNotes, optimization ran separately on 3 sub-samples. Pareto-front solutions were then rescored on the full data, and the best FINAL score was kept.
  • Mention detection merged parse-based NP chunks with Stanford NER entities. It reached 91.56% F under gold coreference, and 91.45% after discarding noise such as URLs, fillers and some time expressions.
  • On the development set, the optimized split model scored 55.15 FINAL (MUC 57.05, B3 67.46, CEAF-E 42.61). The all-features model scored 50.33 and the Soon et al. baseline 53.42.
  • On the official test evaluation (open track), the system scored FINAL 54.32, with MUC 57.53%, B3 65.18% and CEAF-E 40.16%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

7 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{uryupina2011multi,
  title = {{Multi-metric optimization for coreference: The UniTN / IITP / Essex submission to the 2011 CONLL Shared Task}},
  author = {Uryupina, Olga and Saha, Sriparna and Ekbal, Asif and Poesio, Massimo},
  booktitle = {Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task},
  year = {2011},
  url = {https://aclanthology.org/W11-1908/}
}

References

10 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Bagga and B. Baldwin. 1998. Algorithms for scoring coreference chains. In Proc. of the LREC workshop on Linguistic Coreference, pages 563–566, Granada. find our card
  2. Kalyanmoy Deb, Amrit Pratap, Sameer Agarwal, and T. Meyarivan. 2002. A fast and elitist multiobjective genetic algorithm: NSGA-II. IEEE Transactions on Evolutionary Computation, 6(2):181–197. find
  3. Kalyanmoy Deb. 2001. Multi-objective Optimization Using Evolutionary Algorithms. John Wiley and Sons, Ltd, England. find
  4. G. Doddington, A. Mitchell, M. Przybocki, L. Ramshaw, S. Strassell, and R. Weischedel. 2000. The automatic content extraction (ACE) program–tasks, data, and evaluation. In Proc. of LREC. find
  5. Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by Gibbs sampling. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics, pages 363–370. find
  6. D. E. Goldberg. 1989. Genetic Algorithms in Search, Optimization and Machine Learning. Addison-Wesley, New York. find
  7. Veronique Hoste. 2005. Optimization Issues in Machine Learning of Coreference Resolution. Ph.D. thesis, Antwerp University. find
  8. X. Luo. 2005. On coreference resolution performance metrics. In Proc. NAACL / EMNLP, Vancouver. find our card
  9. Art Munson, Claire Cardie, and Rich Caruana. 2005. Optimizing to arbitrary NLP metrics using ensemble selection. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing (HLT/EMNLP), pages 539–546. find
  10. Vincent Ng and Claire Cardie. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of the 40th Annual Meeting on Association for C omputational Linguistics, pages 104–111. find our card

↑ back to top

This page in Markdown →