Elamur Research⌕
☰
home / papers / 2012 / Active Learning for Coreference Resolution

Active Learning for Coreference Resolution

Summary

Supervised coreference resolution (CR) relies on costly hand-labeled data. Active learning (AL) can reduce that cost, but earlier attempts at AL for CR had failed. Uncertainty sampling performed no better than random selection, and the authors confirmed this in their own setup. They propose an AL scheme built on the mention-pair system SUCRE, which uses a decision-tree link classifier and best-first clustering. The scheme combines three ideas:

  • Query-by-committee selection: links are chosen by committee disagreement rather than by single-model uncertainty.
  • Bootstrapped neighborhood pooling: the committee's majority vote stands in for missing gold labels, which yields a class-balanced candidate pool in the style of Soon et al.
  • Neighborhood selection: whole groups of links around a markable are labeled together, so each markable gets both positive and negative examples.

The method was tested on the English portion of the SemEval-2010 CR data, a subset of OntoNotes 2.0. Neighborhood-based AL beat a strong random baseline that was given gold neighborhoods. It did so on MUC, B3, CEAF and their mean, needing fewer labels and reaching a higher ceiling. This is the first positive result showing that AL can make CR annotation cheaper.

Key points

  • Data and pool: the training set has about 96,000 words and the test set about 24,000. The candidate pool holds 1.7 million links within a 100-markable window, and only 1.5% of these are coreferent.
  • Committee: 10 copies of the SUCRE link classifier, each trained on a random subset of the links labeled so far. Links are ranked by vote entropy.
  • Results at 20,000 links: neighborhood-selection AL reached MUC 63.07, B3 86.94, CEAF 84.42 and mean F1 78.14. The gold N-pool random baseline scored 61.60, 85.00, 82.85 and 76.48. The gains are significant under a sign test (p < .05, 10 runs).
  • Link-level selection: selecting individual links helped B3 and CEAF but hurt MUC. It produced too many singleton entities, which lowered recall.
  • Results at 50,000 links: the advantage persisted, with a mean F1 of 78.42 versus 77.33 for the baseline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

10 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{laws2012active,
  title = {{Active Learning for Coreference Resolution}},
  author = {Laws, Florian and Heimerl, Florian and Schütze, Hinrich},
  booktitle = {Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2012},
  url = {https://aclanthology.org/N12-1055/}
}

References

16 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Argamon-Engelson and I. Dagan. 1999. Committee-based sample selection for probabilistic classifiers. JAIR, 11:335–360. find
  2. A. Culotta, M. Wick, R. Hall, and A. McCallum. 2007. First-order probabilistic models for coreference resolution. In HLT-NAACL 2007. find our card
  3. Pedro Domingos and Michael J. Pazzani. 1997. On the optimality of the simple bayesian classifier under zero-one loss. Mach. Learn., 29(2-3):103–130. find
  4. K. Dwyer and R. Holte. 2007. Decision tree instability and active learning. In ECML. find
  5. C. Gasperin. 2009. Active learning for anaphora resolution. In Proceedings of the NAACL HLT 2009 Workshop on Active Learning for Natural Language Processing. find our card
  6. E. Hovy, M. Marcus, M. Palmer, L. Ramshaw, and R. Weischedel. 2006. Ontonotes: The 90% solution. In HLT-NAACL. find
  7. H. Kobdani, H. Schütze, M. Schiehlen, and H. Kamp. 2011. Bootstrapping coreference resolution using word associations. In ACL. find our card
  8. D. Lewis and W. Gale. 1994. A sequential algorithm for training text classifiers. In SIGIR. find
  9. V. Ng and C. Cardie. 2002. Improving machine learning approaches to coreference resolution. In ACL. find our card
  10. M. Osborne and J. Baldridge. 2004. Ensemble-based active learning for parse selection. In HLT-NAACL. find
  11. S. Pradhan, L. Ramshaw, M. Marcus, M. Palmer, R. Weischedel, and N. Xue. 2011. Conll-2011 shared task: Modeling unrestricted coreference in ontonotes. In CoNLL. find our card
  12. M. Recasens, L. Màrquez, E. Sapena, M. A. Martı́, M. Taulé, V. Hoste, M. Poesio, and Y. Versley. 2010. Semeval-2010 task 1: Coreference resolution in multiple languages. In Proceedings of the 5th International Workshop on Semantic Evaluation. find our card
  13. E. Ringger, P. McClanahan, R. Haertel, G. Busby, M. Carmen, J. Carroll, K. Seppi, and D. Lonsdale. 2007. Active learning for part-of-speech tagging: Accelerating corpus annotation. In Linguistic Annotation Workshop at ACL-2007. find
  14. W. M. Soon, D. Chung, D. Chung Yong Lim, Y. Lim, and H. T. Ng. 2001. A machine learning approach to coreference resolution of noun phrases. Computational Linguistics, 27(4). find our card
  15. K. Tomanek, J. Wermter, and U. Hahn. 2007. An approach to text corpus construction which cuts annotation costs and maintains reusability of annotated data. In EMNLP-CoNLL. find
  16. S. Tong and D. Koller. 2002. Support vector machine active learning with applications to text classification. JMLR, 2:45–66. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/laws-2012-active-learning-coreference-resolution/