Elamur Research⌕
☰
home / papers / 2002 / Combining Sample Selection and Error-Driven Pruning for Machine Learning of Coreference Rules

Combining Sample Selection and Error-Driven Pruning for Machine Learning of Coreference Rules

Summary

The paper addresses noun phrase coreference resolution, which is usually treated as pairwise classification of NPs followed by clustering. Ng and Cardie identify three weaknesses of this setup. First, positive pairs are very rare. Second, many positive pairs drawn from a coreference chain are hard to learn from local cues. Third, pairwise decisions ignore transitivity, so better classifier accuracy need not improve clustering-level scores.

They build a baseline with RIPPER rule learning, 25 automatically computed features and best-first clustering. Each weakness gets its own fix:

  • NEG-SELECT removes many negative training pairs.
  • Automatic positive-instance selection keeps mainly the "easy" coreferent pairs.
  • RULE-SELECT is a greedy error-driven method. It drops classifier rules whenever doing so raises the MUC F-measure on a separate pruning corpus.

Evaluated on MUC-6 and MUC-7, the final system reaches F-measures of 69.5 and 63.4. This beats the best official MUC systems and the strongest earlier learning-based approach (Soon et al., 2001). The work matters because it links training decisions directly to the evaluation metric, without changing the learner itself.

Key points

  • Only about 2% of training pairs in the MUC-6/MUC-7 coreference data are positive. Negative sample selection raised F-measure by roughly 3–5%, and positive sample selection added a further 12–14%.
  • RULE-SELECT moved F-measure from 69.3 to 69.5 on MUC-6 and from 57.2 to 63.4 on MUC-7. On MUC-7, precision jumped from 55.1 to 73.6, while recall dipped from 59.5 to 54.2.
  • Each dataset's formal evaluation corpus served as the pruning corpus for the other. As a control, the authors simply trained on the training and pruning data combined, without pruning. This gave 67.6 on MUC-6 and 57.8 on MUC-7, so the gains come from pruning rather than from extra data.
  • The final system outperforms the best MUC entries by 4.6% (MUC-6) and 1.6% (MUC-7), and Soon et al. (2001) by 6.9% and 3.0%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

16 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 8 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1118693.1118701

@inproceedings{ng2002combining,
  title = {{Combining Sample Selection and Error-Driven Pruning for Machine Learning of Coreference Rules}},
  author = {Ng, Vincent and Cardie, Claire},
  booktitle = {Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002)},
  year = {2002},
  doi = {10.3115/1118693.1118701},
  url = {https://doi.org/10.3115/1118693.1118701}
}

References

18 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. Aone and S. W. Bennett. 1995. Evaluating Automated and Manual Acquisition of Anaphora Resolution Strategies. In Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics, pages 122–129. find our card
  2. A. Blum and P. Langley. 1997. Selection of relevant features and examples in machine learning. Artificial Intelligence, pages 245–271. find
  3. C. Cardie and N. Howe. 1997. Improving minority class prediction using case-specific feature weights. In Proceedings of the Fourteenth International Conference on Machine Learning, pages 57–65. find
  4. C. Cardie and K. Wagstaff. 1999. Noun Phrase Coreference as Clustering. In Joint SIGDAT Conference on Empirical Methods in Natural Language Processing and Very Large Corpora (EMNLP/VLC-99), pages 82– 89. find our card
  5. W. Cohen. 1995. Fast Effective Rule Induction. In Proceedings of the Twelfth International Conference on Machine Learning, San Francisco, CA. find
  6. A. Dempster. 1968. A generalization of Bayesian inference. Journal of the Royal Statistical Society, 30:205– 247. find
  7. T. Fawcett. 1996. Learning with skewed class distributions — summary of responses. Machine Learning List: Vol. 8, No. 20. find
  8. D. F. Gordon and D. Perlis. 1989. Explicitly biased generalization. Computational Intelligence, 5:67–81. find
  9. S. Harabagiu, R. Bunescu, and S. Maiorano. 2001. Text and Knowledge Mining for Coreference Resolution. In Proceedings of the Second Meeting of the North America Chapter of the Association for Computational Linguistics (NAACL-2001), pages 55–62. find
  10. A. Kehler. 1997. Probabilistic Coreference in Information Extraction. In Proceedings of the Second Conference on Empirical Methods in Natural Language Processing, pages 163–173. find our card
  11. M. Kubat and S. Matwin. 1997. Addressing the Curse of Imbalanced Training Sets: One-Sided Selection. In Proceedings of the 14th International Conference on Machine Learning (ICML-97), pages 179–186. find
  12. J. McCarthy and W. Lehnert. 1995. Using Decision Trees for Coreference Resolution. In Proceedings of the Fourteenth International Conference on Artificial Intelligence, pages 1050–1055. find
  13. MUC-6. 1995. Proceedings of the Sixth Message Understanding Conference (MUC-6). Morgan Kaufmann, San Francisco, CA. find
  14. MUC-7. 1998. Proceedings of the Seventh Message Understanding Conference (MUC-7). Morgan Kaufmann, San Francisco, CA. find
  15. V. Ng and C. Cardie. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics. find our card
  16. M. Pazzani, C. Merz, P. Murphy, K. Ali, T. Hume, and C. Brunk. 1994. Reducing Misclassification Costs. In Proceedings of the Eleventh International Conference on Machine Learning, pages 217–225. find
  17. W. M. Soon, H. T. Ng, and D. C. Y. Lim. 2001. A Machine Learning Approach to Coreference Resolution of Noun Phrases. Computational Linguistics, 27(4):521–544. find our card
  18. M. Vilain, J. Burger, J. Aberdeen, D. Connolly, and L. Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the Sixth Message Understanding Conference (MUC-6), pages 45– 52. Morgan Kaufmann. find our card

↑ back to top