Elamur Research⌕
☰
home / papers / 2009 / SemEval-2010 Task 1: Coreference Resolution in Multiple Languages

SemEval-2010 Task 1: Coreference Resolution in Multiple Languages

Summary

This paper proposes a SemEval-2010 shared task that benchmarks automatic coreference resolution in Catalan, English and Spanish. The task asks three things: how well one system carries over to another language, how much each language needs its own tuning, and whether the choice of scoring metric changes how systems are ranked. There are two subtasks. The first is building complete coreference chains over named entities, pronouns and full NPs. The second is finding the antecedents of pronouns. Participants can work from gold-standard linguistic input, available only for Catalan and Spanish, or from automatically predicted input. They can also choose a closed setting (training data only) or an open one (external resources allowed). Systems are scored with B-cubed and CEAF, and MUC is added for comparison with earlier work. This lets the organisers also study the metrics themselves. The data is AnCora-CO, which is the AnCora corpora extended with identity coreference links. As a task description, the paper reports no system results. Its value lies in supplying rare non-English coreference data and a shared multilingual way to evaluate systems.

Key points

  • AnCora-CO contains about 400K words of Catalan, about 400K of Spanish and about 120K of English. The Catalan and Spanish parts are freely available for research.
  • The organisers chose B-cubed and CEAF because MUC has known flaws: it gives no credit for singleton entities and it favours systems that output fewer chains. They rejected the ACE value because it depends on entity and mention types.
  • Only identity links are scored. Appositions, predicative NPs, discourse deixis, bound anaphora and bridging are left out. Elliptical subjects in the pro-drop languages (Catalan and Spanish) are annotated as mentions.
  • Seven trained linguists annotated the corpus with the AnCoraPipe tool in two passes: a first coding pass, then a revision pass.
  • A pilot study on Spanish data, with eight annotators, gave inter-annotator agreement above 0.8 using Krippendorff's alpha.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution

Cited by

85 papers of our library cite it (of the 140673 whose bibliographies we read); 35 of them have a page here, 50 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{recasens2009semeval,
  title = {{SemEval-2010 Task 1: Coreference Resolution in Multiple Languages}},
  author = {Recasens, Marta and Martí, Toni and Taulé, Mariona and Màrquez, Lluís and Sapena, Emili},
  booktitle = {Proceedings of the Workshop on Semantic Evaluations: Recent Achievements and Future Directions (SEW-2009)},
  year = {2009},
  url = {https://aclanthology.org/W09-2411/}
}

References

13 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bagga, Amit and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In Proceedings of Language Resources and Evaluation Conference. find our card
  2. Bertran, Manuel, Oriol Borrega, Marta Recasens, and Bàrbara Soriano. 2008. AnCoraPipe: A tool for multilevel annotation, Procesamiento del Lenguaje Natural, n. 41: 291-292, SEPLN. find
  3. Denis, Pascal and Jason Baldridge. 2008. Specialized models and ranking for coreference resolution. Proceedings of the Empirical Methods in Natural Language Processing (EMNLP 2008). find our card
  4. Luo, Xiaoqiang. 2005. On coreference resolution performance metrics. Proceedings of HLT/NAACL 2005. find our card
  5. McCarthy Joseph and Wendy Lehnert. 1995. Using decision trees for coreference resolution. Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence. find
  6. Mitkov, Ruslan. 1998. Robust pronoun resolution with limited knowledge. Proceedings of the 36 th Annual Meeting of the Association for Computational Linguistics, and 17 th International Conference on Computational Linguistics (COLING-ACL98). find our card
  7. Morton, Thomas. 2000. Using coreference for question answering. Proceedings of the 8th Text REtrieval Conference (TREC-8). find our card
  8. Ng, Vincent. 2008. Unsupervised models for coreference resolution. Proceedings of the Empirical Methods in Natural Language Processing (EMNLP 2008). find our card
  9. NIST. 2003. In Proceedings of ACE 2003 workshop. Booklet, Alexandria, VA. find
  10. Recasens, Marta. 2008. Towards Coreference Resolution for Catalan and Spanish. Master Thesis. University of Barcelona. find
  11. Steinberger, Josef, Massimo Poesio, Mijail Kabadjov, and Karel Jezek. 2007. Two uses of anaphora resolution in summarization. Information Processing and Management, 43:1663–1680. find
  12. Taulé, Mariona, Antònia Martí, and Marta Recasens. 2008. AnCora: Multilevel corpora with coreference information for Spanish and Catalan. In Proceedings of the Language Resources and Evaluation Conference (LREC 2008). van Deemter, Kees and Rodger Kibble. 2000. Squibs and Discussions: On coreferring:… find
  13. Vilain, Marc, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of MUC-6. find our card

↑ back to top