Elamur Research⌕
☰
home / papers / 2001 / A Corpus-Based Evaluation of Centering and Pronoun Resolution

A Corpus-Based Evaluation of Centering and Pronoun Resolution

Summary

This paper asks how leading pronoun resolution algorithms compare when they are implemented and run automatically on the same corpus. Earlier evaluations were done by hand on small text sets. The author introduces Left-Right Centering (LRC), a centering-based algorithm. It follows the constraints and rules of centering theory but processes utterances incrementally, unlike Brennan, Friedman, and Pollard's (BFP) method. LRC is compared with Hobbs's naive algorithm, Strube's S-list and BFP. Every system uses only syntactic information and is tested on parsed, coreference-annotated Penn Treebank material: New York Times articles and fiction texts. LRC is then used to check two psycholinguistic proposals for ranking the Cf-list. One gives prominence to entities in prepended phrases. The other places the possessed entity before the possessor in complex NPs. Neither change improves accuracy. The evidence instead supports a syntax-based ranking and Walker and Prince's left-to-right handling of complex NPs. The work shows that coded, corpus-scale evaluation can both benchmark resolvers and test findings from small psycholinguistic experiments.

Key points

  • Four algorithms (Hobbs, BFP, S-list, LRC and its variants) are compared in computer-encoded form on a shared, Penn Treebank-based corpus. The paper reports this kind of comparison had not been done before.
  • Following the proposal of Gordon, Grosz, and Gilliom (1993) to place prepended phrases high in the Cf-list ranking does not help resolution.
  • Ordering possessed entities before possessors is slightly worse than left-to-right ordering. On the New York Times data (1,691 pronouns), LRC-S resolves 1,362 pronouns correctly with left-to-right ordering versus 1,337 with full reversal, a gap of 25 pronouns.
  • Errors suggest that previous mention and the type of referring expression affect salience. This is consistent with Strube and Hahn's functional centering.
  • The author estimates that purely syntactic resolvers top out in the mid-80% range. Reaching 90% or more across domains would likely require semantic knowledge, such as sortal constraints.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

24 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 14 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@misc{tetreault2001corpus,
  title = {{A Corpus-Based Evaluation of Centering and Pronoun Resolution}},
  author = {Tetreault, Joel R.},
  year = {2001},
  url = {https://aclanthology.org/J01-4003/}
}

References

24 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Brennan, Susan E., Marilyn W. Friedman, and Carl J. Pollard. 1987. A centering approach to pronouns. In Proceedings o/the 25th Annual Meeting of the Association for Computational Linguistics, pages 155-162. find
  2. Byron, Donna K. and Joel R. Tetreault. 1999. A flexible architecture for reference resolution. In Proceedings of the 9th Conference of the European Chapter of the Association for Computational Linguistics, pages 229-232. find
  3. Carter, David M. 1987. Interpreting Anaphors in Natural Language Texts. Ellis Horwood, Chichester, UK. find
  4. Ge, Niyu, John Hale, and Eugene Charniak. 1998. A statistical approach to anaphora resolution. In Proceedings of the Sixth Workshop on Very Large Corpora, pages 161-170. find our card
  5. Gordon, Peter C., Barbara J. Grosz, and Laura Gilliom. 1993. Pronouns, names and the centering of attention in discourse. Cognitive Science, 17(3):311-348. find
  6. Gordon, Peter C., Randall Hendrick, Kerry Ledoux, and Chin Lung Yang. 1999. Processing of reference and the structure of language: An analysis of complex noun phrases. Language and Cognitive Processes, 14(4):353-379. find
  7. Grosz, Barbara J., Aravind K. Joshi, and Scott Weinstein. 1986. Towards a computational theory of discourse interpretation. Preliminary draft. find
  8. Grosz, Barbara J., Aravind K. Joshi, and Scott Weinstein. 1995. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21(2):203-225. find our card
  9. Grosz, Barbara J. and Candace L. Sidner. 1986. Attention, intentions, and the structure of discourse. Computational Linguistics, 12(3):175-204. find our card
  10. Hobbs, Jerry R. 1978. Resolving pronoun references. Lingua, 44:311-338. find our card
  11. Kameyama, Megumi. 1998. Intrasentential centering: A case study. In M. A. Walker, A. K. Joshi, and E. F. Prince, editors, Centering Theory in Discourse, 89-112. Oxford University Press. find
  12. Kehler, Andrew. 1997. Current theories of centering for pronoun interpretation: A critical evaluation. Computational Linguistics, 23(3):467-475. find
  13. Marcus, Mitchell P., Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330. find our card
  14. Mitkov, Ruslan. 1998. Robust pronoun resolution with limited knowledge. In Proceedings of the 36th Meeting of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics, pages 869-875. find our card
  15. Mitkov, Ruslan. 2000. Towards a more consistent and comprehensive evaluation of anaphora resolution algorithms and systems. In Proceedings qf the Discourse Anaphora and Reference Resolution Conference (DAARC2000), 96-107. find
  16. Prince, Ellen F. 1981. Towards a taxonomy of given-new information. In P. Cole, editor, Radical Pragmatics, 223-255. Academic Press. find
  17. Strube, Michael. 1998. Never look back: An alternative to centering. In Proceedings of the 36th Meeting of the Association for Computational Linguistics and the 17th International Conference on Computational Linguistics, volume 2, pages 1251-1257. find
  18. Strube, Michael and Udo Hahn. 1996. Functional centering. In Proceedings of the 34th Annual Meeting of the Association for Computational Linguistics, pages 270-277. find
  19. Strube, Michael and Udo Hatch. 1999. Functional centering: Grounding referential coherence in information structure. Computational Linguistics, 25(3):309-344. find our card
  20. Suri, Linda Z., Kathleen F. McCoy, and Jonathan D. DeCristofaro. 1999. A methodology for extending focusing frameworks. Computational Linguistics, 25(2):173-194. find
  21. Tetreault, Joel. 1999. Analysis of syntax-based pronoun resolution methods. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 602-605. find our card
  22. Walker, Marilyn A. 1989. Evaluating discourse processing algorithms. In Proceedings of the 27th Annual Meeting of the Association for Computational Linguistics, pages 251-261. find
  23. Walker, Marilyn A., Masayo Iida, and Sharon Cote. 1994. Japanese discourse and the process of centering. Computational Linguistics, 20(2):193-233. find
  24. Walker, Marilyn A. and Ellen F. Prince. 1996. A bilateral approach to givenness: A hearer-status algorithm and a centering algorithm. In T. Fretheim and J. K. Gundel, editors, Reference and Referent Accessibility, John Benjamins, Amsterdam, pages 291-306. find

↑ back to top