Elamur Research⌕
☰
home / papers / 2008 / Anaphora Resolution Exercise: an Overview

Anaphora Resolution Exercise: an Overview

Summary

Approaches to anaphora and coreference resolution have been hard to compare because studies typically use different corpora and scoring schemes. To address this, the authors ran the first Anaphora Resolution Exercise (ARE), a shared evaluation held alongside DAARC2007 and limited to English pronouns and NP coreference. It was deliberately exploratory, meant to prepare a later full-scale campaign. ARE defined four tasks: pronoun resolution and coreference chain building, each run on texts with pre-marked entities and on raw texts. This setup separates evaluation of resolution algorithms from evaluation of complete end-to-end systems. The data came from a coreference-annotated collection of Reuters news articles, and each task had its own XML input format and scoring metric. Three institutions submitted runs, and the results were published anonymously. The organisers drew several conclusions: future editions should cover indirect anaphora and languages other than English, the MUC scoring scheme seems overly lenient, and their overlap-based measure needs improvement. ARE's main contribution is a shared testbed that makes comparison between systems more consistent.

Key points

  • Four tasks:
    • (1) Pronoun resolution with given candidate NPs.
    • (2) Coreference chains over pre-marked entities.
    • (3) Pronoun resolution on raw text.
    • (4) Coreference chains on raw text.
  • Data: The corpus of Hasler et al. (2006), built from Reuters articles, totals more than 55,000 words in five topic clusters. Four clusters were released for training and one was held out for testing.
    • Links tagged ucoref (uncertain) were discarded and only coref links kept.
    • The retained relations were IDENTITY, SYNONYMY, GENERALISATION and SPECIALISATION between NPs.
  • Scoring: Pronoun tasks were scored by success rate or by precision/recall with partial credit for overlap. Task 4 used a modified MUC-6 score.
  • Participation: The University of Karlsruhe, Alexandru Ioan Cuza University (Iasi) and the University of Wolverhampton took part. Each had 48 hours after downloading the test data to submit results.
  • Results: In Task 1, the baseline scored 33.16% success rate, compared with 43.07% and 71.55% for the two systems. Task 4 drew the most submissions (three), while Task 2 received only one.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

8 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{orasan2008anaphora,
  title = {{Anaphora Resolution Exercise: an Overview}},
  author = {Orăsan, Constantin and Cristea, Dan and Mitkov, Ruslan and Branco, António},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  doi = {10.63317/2v67ynrmx9tc},
  url = {https://doi.org/10.63317/2v67ynrmx9tc}
}

References

10 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Cătălina Barbu and Ruslan Mitkov. 2001. Evaluation tool for rule-based anaphora resolution methods. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 34 – 41, Toulouse, France, July 9 – 11. find our card
  2. Dan Cristea and Oana Postolache. 2006. Anaphora resolution: framework, creation of resources, and evaluation. In Svetla Koeva and Mila Dimitrova-Vulchanova, editors, Proceedings of the Fifth International Conference on Formal Approaches to South Slavic and Balkan Languages, pages 1 – 5, Sofia, Bulga… find
  3. Laura Hasler, Constantin Orăsan, and Karin Naumann. 2006. NPs for Events: Experiments in Coreference Annotation. In Proceedings of the 5th edition of the International Conference on Language Resources and Evaluation (LREC2006), pages 1167 – 1172, Genoa, Italy, 24 – 26 May. find our card
  4. Inderjeet Mani, Therese Firmin, David House, Michael Chrzanowski, Gary Klein, Lynette Hirshman, Beth Sundheim, and Leo Obrst. 1998. The TIPSTER SUMMAC text summarisation evaluation: Final report. Technical Report MTR 98W0000138, The MITRE Corporation. find
  5. Ruslan Mitkov. 2000. Towards more comprehensive evaluation in anaphora resolution. In Proceedings of the Second International Conference on Language Resources and Evaluation, volume III, pages 1309 – 1314, Athens, Greece. find
  6. Ruslan Mitkov. 2001. Outstanding issues in anaphora resolution. In Al. Gelbukh, editor, Computational Linguistics and Intelligent Text Processing, pages 110– 125. Springer. find
  7. Ruslan Mitkov. 2002. Anaphora resolution. Longman. find
  8. Tony G. Rose, Mark Stevenson, and Miles Whitehead. 2002. The Reuters Corpus Volume 1 - from Yesterday’s News to Tomorrow’s Language Resources. In Proceedings of the Third International Conference on Language Resources and Evaluation (LREC-2002), pages 827 – 833, Las Palmas de Gran Canaria, May. find
  9. TIDES. 2000. Translingual Information Detection, Extraction and Summarization (TIDES). http://www.darpa.mil/iao/TIDES.htm. ↗
  10. M. Vilain, J. Burger, J. Aberdeen, D. Connoly, and L. Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the 6th Message Understanding Conference (MUC-6), pages 45 – 52, San Francisco, California, USA. find our card

↑ back to top

This page in Markdown →