Elamur Research⌕
☰
home / papers / 2001 / Evaluation Tool for Rule-based Anaphora Resolution Methods

Evaluation Tool for Rule-based Anaphora Resolution Methods

Summary

Comparing anaphora resolution systems has been unreliable because each one was tested on its own data, with its own pre-processing, and often with some manual correction of the input. Barbu and Mitkov address this with an evaluation workbench, an extensible environment where algorithms are plugged in and run on identical texts through identical pre-processing, fully automatically. Pre-processing uses Conexor's FDG Parser plus a custom NP extractor. Its output is stored in an XML format, so any shallow parser could be swapped in. The authors re-implemented three knowledge-poor pronoun resolvers: Kennedy and Boguraev's parser-free method, Baldwin's CogNiac (in its resolve-all variant) and Mitkov's knowledge-poor approach. They tested these on a coreference-annotated corpus of technical documents. Kennedy and Boguraev scored highest, followed by Mitkov and then CogNiac. All three performed well below their originally published figures. This supports the view that fully automatic resolution is harder than earlier evaluations implied. The authors also suggest that evaluation reports describe how difficult the test data is, for example the number of candidate antecedents and the referential distance.

Key points

  • Shared pipeline: the workbench runs Conexor's FDG Parser followed by an NP extractor and stores the result in XML. It has a graphical interface, reports per-method and per-file results, and plans to add a full-parser mode and WordNet access.
  • Test corpus: four technical texts (Beowulf HOW TO, Linux CD-Rom HOW TO, Access HOW TO and a Windows Help file).
    • It contains 28,272 words, 19,305 NPs and 422 pronouns, of which 362 are anaphoric.
    • It was annotated by two people with a slightly modified MUC scheme.
  • Overall success rate:
    • Kennedy & Boguraev: 61.6%
    • Mitkov: 56.9%
    • CogNiac: 49.72%
  • Overall precision (equal to recall, since all three methods are robust):
    • Kennedy & Boguraev: 52.84%
    • Mitkov: 48.81%
    • CogNiac: 42.65%
  • Proposed difficulty statistics for the corpus:
    • about 10.85 candidate antecedents per pronoun on average;
    • an average referential distance of roughly 1.3 sentences and 4.3 NPs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

11 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 4 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/1073012.1073018

@inproceedings{barbu2001evaluation,
  title = {{Evaluation Tool for Rule-based Anaphora Resolution Methods}},
  author = {Barbu, Catalina and Mitkov, Ruslan},
  booktitle = {Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics},
  year = {2001},
  doi = {10.3115/1073012.1073018},
  url = {https://doi.org/10.3115/1073012.1073018}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chinatsu Aone and Scot W. Bennett. 1995. Evaluating automated and manual acquisition of anaphora resolution rules. In Proceedings of the 33th Annual Meeting of the Association for Computational Linguistics (ACL ’95), pages 122– 129. find
  2. Breck Baldwin. 1997. Cogniac: High precision coreference with limited knowledge and linguistic resources. In R. Mitkov and B. Boguraev, editors, Operational factors in practical, robust anaphora resolution for unrestricted texts, pages 38 – 45. find our card
  3. Ido Dagan and Alon Itai. 1990. Automatic processing of large corpora for the resolution of anaphora references. In Proceedings of the 13th International Conference on Computational Linguistics (COLING’90), volume III, pages 1–3. find our card
  4. Ido Dagan and Alon Itai. 1991. A statistical filter for resolving pronoun references. In Y.A. Feldman and A. Bruckstein, editors, Artificial Intelligence and Computer Vision, pages 125 – 135. Elsevier Science Publishers B.V. find
  5. Antonio Ferrandez, Manolo Palomar, and L. Moreno. 1997. Slot unification grammar and anaphora resolution. In Proceedings of the International Conference on Recent Advances in Natural Language Proceeding (RANLP’97), pages 294– 299. find
  6. Robert Gaizauskas and Kevin Humphreys. 2000. Quantitative evaluation of coreference algorithms in an information extraction system. In Simon Botley and Antony Mark McEnery, editors, Corpus-based and Computational Approaches to Discourse Anaphora, Studies in Corpus Linguistics, chapter 8, pages 145 –… find
  7. Niyu Ge, J. Hale, and E. Charniak. 1998. A statistical approach to anaphora resolution. In Proceedings of the Sixth Workshop on Very Large Corpora, COLING-ACL ’98, pages 161 – 170, Montreal, Canada. find our card
  8. Jerry Hobbs. 1976. Pronoun resolution. Research report 76-1, City College, City University of New York. find
  9. Jerry Hobbs. 1978. Pronoun resolution. Lingua, 44:339–352. find
  10. Christopher Kennedy and Branimir Boguraev. 1996. Anaphora for everyone: pronominal anaphora resolution without a parser. In Proceedings of the 16th International Conference on Computational Linguistics (COLING’96), pages 113–118, Copenhagen, Denmark. find our card
  11. Shalom Lappin and H.J. Leass. 1994. An algorithm for pronominal anaphora resolution. Computational Linguistics, 20(4):535 – 562. find our card
  12. Ruslan Mitkov, R. Evans, C. Orasan, C. Barbu, L. Jones, and V. Sotirova. 2000. Coreference and anaphora: developing annotating tools, annotated resources and annotation strategies. In Proceedings of the Discourse, Anaphora and Reference Resolution Conference (DAARC2000), pages 49–58, Lancaster, UK. find
  13. Ruslan Mitkov. 1998a. Evaluating anaphora resolution approaches. In Proceedings of the Discourse Anaphora and Anaphora Resolution Colloquium (DAARC’2), pages 164 – 172, Lancaster, UK. find
  14. Ruslan Mitkov. 1998b. Robust pronoun resolution with limited knowledge. In Proceedings of the 18th International Conference on Computational Linguistics (COLING’98/ACL’98, pages 867 – 875. Morgan Kaufmann. find our card
  15. Ruslan Mitkov. 2000a. Towards a more consistent and comprehensive evaluation of anaphora resolution algorithms and systems. In Proceedings of the Discourse, Anaphora and Reference Resolution Conference (DAARC2000), pages 96 – 107, Lancaster, UK. find
  16. Ruslan Mitkov. 2000b. Towards more comprehensive evaluation in anaphora resolution. In Proceedings of the Second International Conference on Language Resources and Evaluation, volume III, pages 1309 – 1314, Athens, Greece. find
  17. Ruslan Mitkov. 2001. Outstanding issues in anaphora resolution. In Al. Gelbukh, editor, Computational Linguistics and Intelligent Text Processing, pages 110–125. Springer. find
  18. Constantin Orăsan, Richard Evans, and Ruslan Mitkov. 2000. Enhancing preference-based anaphora resolution with genetic algorithms. In Proceedings of Natural Language Processing - NLP2000, pages 185 – 195. Springer. find
  19. P. Tapanainen and T. Järvinen. 1997. A non-projective dependency parser. In Proceedings of the 5th Conference of Applied Natural Language Processing, pages 64 – 71, Washington D.C., USA. find
  20. Joel R. Tetreault. 1999. Analysis of syntax-based pronoun resolution methods. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics (ACL ’99), pages 602 – 605, Maryland, USA. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/barbu-2001-evaluation-tool-rule-based-anaphora/