Elamur Research
☰
home / papers / 2016 / WikiCoref: An English Coreference-annotated Corpus of Wikipedia Articles

WikiCoref: An English Coreference-annotated Corpus of Wikipedia Articles

What it is about

WikiCoref is an English coreference corpus of Wikipedia articles, a genre that earlier corpora covered little, since most of them are newswire. The annotation follows OntoNotes with some changes. Each markable has a coreference type, a mention type and, for each chain, the equivalent Freebase topic. The first release has 30 documents, 59,652 tokens and 2,229 sentences, from 209 to 9,869 tokens each. The mention extent is the maximal one, so it includes pre-modifiers, post-modifiers and relative clauses. Agreement between two annotators is a Cohen's kappa of 0.78 on mention identification, and a MUC F1 of 83.3%. The authors call both values fair agreement.

Ideas

For Mova

The chapter is languages/en/35-reference-and-coreference.md. The code is coref/src/mention.rs and coref/src/sieve.rs, with the entity links in world. The measure is MUC F1 and CoNLL coreference F1 on the held-out documents, run twice. The corpus is NC-licensed, so it is used for evaluation only.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/L16-1021.pdf

Summary

Most English coreference corpora, such as MUC, ACE and OntoNotes, are built mainly from news text. Systems trained on news are known to degrade on other genres, and Wikipedia, despite being heavily used as a knowledge source for coreference systems, had no dedicated annotated corpus. The authors address this gap with WikiCoref, a small, freely available English corpus made entirely of Wikipedia articles. Articles were sampled to balance length, out-link density and topic, using the Ratinov and Roth gazetteer lists.

Annotation was semi-automatic. The Stanford CoreNLP Dcoref sieve system was extended with Wikipedia and Freebase information (anchors, redirects, concept attributes) to propose extra mentions, merge chains and fix mention boundaries. A human annotator then corrected the output. The scheme largely follows OntoNotes. Each markable records its coreference type, its mention type and a linked Freebase topic. Agreement measured with Kappa and MUC was reported as fair. The resource is meant to support coreference resolution tailored to encyclopedic text.

Key points

  • The first release contains 30 Wikipedia documents with 7955 mentions grouped into 1785 coreference chains, annotated by a single annotator.
  • Articles were drawn from four size ranges: under 1000 words, 1000–2000, 2000–5000 and over 5000. Pages shorter than 200 words and list-like index pages were excluded.
  • Pre-annotation combined Dcoref with Wikipedia/Freebase entity tracking, using WikipediaMiner for article access. This improved both recall and precision of the candidate chains.
  • Half of the chains contain only two mentions, and about 5.7% contain ten or more. Each article's main concept averages 68 mentions per document, which is roughly 25% of all mentions.
  • Agreement was checked on three documents annotated by two people, using Kappa and MUC. The corpus is downloadable from the RALI site, and the work was funded by the Nuance Foundation.

Topics: Coreference resolution · Treebanks

Cite

The paper itself, not this card. BibTeX from the library's record of the work. DOI 10.63317/3yjtmzi2qxqk

@inproceedings{ghaddar2016wikicoref,
  title = {{WikiCoref: An English Coreference-annotated Corpus of Wikipedia Articles}},
  author = {Ghaddar, Abbas and Langlais, Phillippe},
  booktitle = {Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16)},
  year = {2016},
  doi = {10.63317/3yjtmzi2qxqk},
  url = {https://doi.org/10.63317/3yjtmzi2qxqk}
}

References

25 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bollacker, K., Evans, C., Paritosh, P., Sturge, T., and Taylor, J. (2008). Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247–1250. find
  2. Carletta, J. (1996). Assessing agreement on classification tasks: the kappa statistic. Computational linguistics, 22(2):249–254. find our card
  3. Castano, J., Zhang, J., and Pustejovsky, J. (2002). Anaphora resolution in biomedical literature. In International Symposium on Reference Resolution. find
  4. Cohen, K. B., Lanfranchi, A., Corvey, W., Baumgartner Jr, W. A., Roeder, C., Ogren, P. V., Palmer, M., and Hunter, L. (2010). Annotation of all coreference in biomedical text: Guideline selection and adaptation. In Proceedings of BioTxtM 2010, pages 37–41. find
  5. Doddington, G. R., Mitchell, A., Przybocki, M. A., Ramshaw, L. A., Strassel, S., and Weischedel, R. M. (2004). The automatic content extraction (ACE) program-tasks, data, and evaluation. In LREC, volume 2, page 1. find
  6. Haghighi, A. and Klein, D. (2009). Simple coreference resolution with rich syntactic and semantic features. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1152–1161. find our card
  7. Hendrickx, I. and Hoste, V. (2009). Coreference resolution on blogs and commented news. In Anaphora Processing and Applications, pages 43–53. find
  8. Hirshman, L. and Chinchor, N. (1998). MUC-7 coreference task definition. version 3.0. In Proceedings of the Seventh Message Understanding Conference (MUC-7). find
  9. Klaus, K. (1980). Content analysis: An introduction to its methodology. find
  10. Lee, H., Chang, A., Peirsman, Y., Chambers, N., Surdeanu, M., and Jurafsky, D. (2013). Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885–916. find our card
  11. Manning, C. D., Surdeanu, M., Bauer, J., Finkel, J. R., Bethard, S., and McClosky, D. (2014). The stanford CoreNLP natural language processing toolkit. In ACL (System Demonstrations), pages 55–60. find our card
  12. Milne, D. and Witten, I. H. (2008). Learning to link with wikipedia. In Proceedings of the 17th ACM conference on Information and knowledge management, pages 509– 518. find
  13. Müller, C. and Strube, M. (2006). Multi-level annotation of linguistic data with MMAX2. Corpus technology and language pedagogy: New resources, new tools, new methods, pages 197–214. find
  14. Nguyen, N., Kim, J. D., and Tsujii, J. (2011). Overview of bionlp 2011 protein coreference shared task. In Proceedings of BioNLP Shared Task 2011 Workshop, pages 74–82. find
  15. Nicolov, N., Salvetti, F., and Ivanova, S. (2008). Sentiment analysis: Does coreference matter. In AISB 2008 Convention Communication, Interaction and Social Intelligence, volume 1, page 37. find
  16. Poesio, M., Di Eugenio, B., and Keohane, G. (2002). Discourse structure and anaphora: An empirical study. Rapport Technique NLE Technical Note TN-02-02. University of Essex. find
  17. Poesio, M. (2004). Discourse annotation and semantic annotation in the GNOME corpus. In Proceedings of the 2004 ACL Workshop on Discourse Annotation, pages 72–79. find
  18. Pradhan, S. S., Ramshaw, L., Weischedel, R., MacBride, J., and Micciulla, L. (2007). Unrestricted coreference: Identifying entities and events in OntoNotes. In null, pages 446–453. find
  19. Pradhan, S., Ramshaw, L., Marcus, M., Palmer, M., Weischedel, R., and Xue, N. (2011). Conll-2011 shared task: Modeling unrestricted coreference in ontonotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–27. find our card
  20. Raghunathan, K., Lee, H., Rangarajan, S., Chambers, N., Surdeanu, M., Jurafsky, D., and Manning, C. (2010). A multi-pass sieve for coreference resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 492–501. find our card
  21. Ratinov, L. and Roth, D. (2009). Design challenges and misconceptions in named entity recognition. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning, pages 147–155. find our card
  22. Rodriguez, K. J., Delogu, F., Versley, Y., Stemle, E. W., and Poesio, M. (2010). Anaphoric annotation of wikipedia and blogs in the live memories corpus. In Proceedings of LREC, pages 157–163. find
  23. Schäfer, U., Spurk, C., and Steffen, J. (2012). A fully coreference-annotated corpus of scholarly papers from the acl anthology. In Proceedings of COLING 2012, pages 1059–1070. find
  24. Segura-Bedmar, I., Crespo, M., de Pablo-Sánchez, C., and Martı́nez, P. (2010). Resolving anaphoras for the extraction of drug-drug interactions in pharmacological documents. BMC bioinformatics, 11:1–11. find
  25. Vilain, M., Burger, J., Aberdeen, J., Connolly, D., and Hirschman, L. (1995). A model-theoretic coreference scoring scheme. In Proceedings of the 6th conference on Message understanding, pages 45–52. find our card