Elamur Research⌕
☰
home / papers / 2012 / A Fully Coreference-annotated Corpus of Scholarly Papers from the ACL Anthology

A Fully Coreference-annotated Corpus of Scholarly Papers from the ACL Anthology

Summary

Most coreference resolution (CR) research and training data, such as MUC and ACE, come from newswire. The authors found that systems trained on news perform worse on scientific writing, which relies more on definite descriptions, domain terminology and abstract entities than on person or organization names. To fill this gap, they built a coreference corpus of 266 ACL Anthology papers. Each paper's full text was obtained through OCR and annotated by hand in a customized MMAX2 tool. Annotation covered noun-phrase coreference with no restriction on entity type, and each referent received a single document-wide coreference set. Non-coreferent mentions were then added automatically to serve as negative training examples. An initial agreement check gave a low score, so the authors refined the guidelines and had one person re-check every paper. The resulting corpus is considerably larger than ACE. It is released publicly with the guidelines and is meant for training CR systems for scholarly text, supporting tasks such as semantic search, question answering and citation analysis.

Key points

  • The source papers come from ACL Anthology sections P08 (ACL-2008), D07 (EMNLP-CoNLL-2007) and C02 (COLING 2002). Text was extracted with commercial OCR rather than the PDFBox-based ACL-ARC files, which had extraction and reading-order problems.
  • The manual layer contains 1,326,147 tokens, 48,960 sentences and 65,293 mentions in non-singleton sets. ACE 2004, by comparison, has 189,620 tokens and 22,293 mentions. Definite NPs are the largest mention type (32,547), followed by proper names including citations (14,451).
  • Singleton mentions were labeled automatically using the Stanford Parser 1.6.3 (NPs and possessives), SProUT (named entities) and a custom regex for citations. Roughly 10% of sentences contain a citation.
  • On a doubly annotated 13% of the corpus, inter-annotator agreement was 49.5 MUC points. The authors then revised the guidelines, and a single corrector reviewed the whole corpus over about 9 months, part-time.
  • The release contains the data in MMAX2 format and roughly 20 pages of annotation guidelines, available via take.dfki.de and the ACL Anthology. The authors note that the corpus would still benefit from another correction pass.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Coreference resolution · Treebanks

Cited by

13 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 7 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{schafer2012fully,
  title = {{A Fully Coreference-annotated Corpus of Scholarly Papers from the ACL Anthology}},
  author = {Schäfer, Ulrich and Spurk, Christian and Steffen, Jörg},
  booktitle = {Proceedings of COLING 2012: Posters},
  year = {2012},
  url = {https://aclanthology.org/C12-2103/}
}

References

35 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bengtson, E. and Roth, D. (2008). Understanding the value of features for coreference resolution. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP-2008, pages 294–303, Morristown, NJ, USA. find our card
  2. Bird, S., Dale, R., Dorr, B., Gibson, B., Joseph, M., Kan, M.-Y., Lee, D., Powley, B., Radev, D., and Tan, Y. F. (2008). The ACL anthology reference corpus: A reference dataset for bibliographic research. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC-2… find
  3. Chen, B., Yang, X., Su, J., and Tan, C. L. (2008). Other-anaphora resolution in biomedical texts with automatically mined patterns. In Proceedings of the 22nd International Conference on Computational Linguistics (COLING-2008), pages 121–128, Manchester, UK. find
  4. Cohen, K. B., Lanfranchi, A., Corvey, W., Jr., W. A. B., Roeder, C., Ogren, P. V., Palmer, M., and Hunter, L. (2010). Annotation of all coreference in biomedical text: Guideline selection and adaptation. In Proceedings of the 2nd Workshop on Building and Evaluating Resources for Biomedical Text Mini… find
  5. Doddington, G., Mitchell, A., Przybocki, M., Ramshaw, L., Strassel, S., and Weischedel, R. (2004). The automatic content extraction (ACE) program – tasks, data, and evaluation. In Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC-2004), pages 837–840. find our card
  6. Drożdżyński, W., Krieger, H.-U., Piskorski, J., Schäfer, U., and Xu, F. (2004). Shallow processing with unification and typed feature structures – foundations and applications. Künstliche Intelligenz, 2004(1):17–23. find
  7. Gaizauskas, R. and Humphreys, K. (2000). Quantitative evaluation of coreference algorithms in an information extraction system. In Botley, S. and McEnery, T., editors, Corpus-based and Computational Approaches to Discourse Anaphora, pages 145–169. John Benjamins, Amsterdam. find
  8. Gasperin, C. V. (2009). Statistical anaphora resolution in biomedical texts. PhD thesis, University of Cambridge, Cambridge, UK. find our card
  9. Grishman, R. and Sundheim, B. (1996). Message understanding conference - 6: A brief history. In Proceedings of COLING-96, pages 466–471. find
  10. Haghighi, A. and Klein, D. (2009). Simple coreference resolution with rich syntactic and semantic features. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1152–1161, Singapore. find our card
  11. Haghighi, A. and Klein, D. (2010). Coreference resolution in a modular, entity-centered model. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 385–393, Los Angeles, California. find our card
  12. Hirschman, L., Robinson, P., Burger, J., and Vilain, M. (1997). Automating coreference: The role of annotated training data. In Proceedings of the AAAI Spring Symposium on Applying Machine Learning to Discourse Processing. find
  13. Hovy, E., Marcus, M., Palmer, M., Ramshaw, L., and Weischedel, R. (2006). OntoNotes: The 90% solution. In Proceedings of the HLT-NAACL 2006 Companion Volume. find
  14. Kaplan, D., Iida, R., and Tokunaga, T. (2009). Automatic extraction of citation contexts for research paper summarization: A coreference-chain based approach. In Workshop on text and citation analysis for scholarly digital libraries (NLPIR4DL), ACL-IJCNLP-09, pages 88–95, Singapore. find
  15. Kim, Y. and Webber, B. (2006). Implicit references to citations: A study of astronomy papers. In Proceedings of the 20th International CODATA Conference: Scientific Data and Knowledge within the Information Society. find
  16. Klein, D. and Manning, C. D. (2003). Accurate unlexicalized parsing. In Proceedings of the 41st Meeting of the Association for Computational Linguistics (ACL-2003), pages 423–430. Association for Computational Linguistics. find our card
  17. LDC (2004). Annotation guidelines for entity detection and tracking (EDT) – version 4.2.6. find
  18. Lee, H., Peirsman, Y., Chang, A., Chambers, N., Surdeanu, M., and Jurafsky, D. (2011). Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In 15th CoNLL: Shared Task, pages 28–34. find our card
  19. Luo, X. (2005). On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25–32, Vancouver, British Columbia, Canada. Association for Computational Linguistics. find our card
  20. Mitkov, R. (1999). Anaphora resolution: The state of the art. Technical report, School of Languages and European Studies, University of Wolverhampton. find
  21. Miwa, M., Thompson, P., and Ananiadou, S. (2012). Boosting automatic event extraction from the literature using domain adaptation and coreference resolution. Bioinformatics. find
  22. Müller, C. and Strube, M. (2006). Multi-level annotation of linguistic data with MMAX2. In Braun, S., Kohn, K., and Mukherjee, J., editors, Corpus Technology and Language Pedagogy: New Resources, New Tools, New Methods, pages 197–214. Peter Lang, Frankfurt a.M., Germany. find
  23. Ng, V. (2010). Supervised noun phrase coreference research: The first fifteen years. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1396–1411, Uppsala, Sweden. find our card
  24. Nguyen, N., Kim, J.-D., and Tsujii, J. (2011). Overview of BioNLP 2011 protein coreference shared task. In Proceedings of BioNLP Shared Task 2011 Workshop, pages 74–82, Portland, Oregon, USA. find our card
  25. Pradhan, S., Ramshaw, L., Marcus, M., Palmer, M., Weischedel, R., and Xue, N. (2011). CoNLL-2011 shared task: Modeling unrestricted coreference in ontonotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–27, Portland, Oregon, USA. find our card
  26. Pradhan, S. S., Ramshaw, L., Ralph, Weischedel, MacBride, J., and Micciulla, L. (2007). Unrestricted coreference: Identifying entities and events in OntoNotes. In Proceedings of the International Conference on Semantic Computing, pages 446–453. IEEE. find
  27. Raghunathan, K., Lee, H., Rangarajan, S., Chambers, N., Surdeanu, M., Jurafsky, D., and Manning, C. (2010). A multi-pass sieve for coreference resolution. In Proceedings of EMNLP-2010, pages 492–501, Cambridge, MA. find our card
  28. Rizzolo, N. and Roth, D. (2010). Learning based Java for rapid development of NLP systems. In Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC-2010), Valletta, Malta. find
  29. Schäfer, U., Kiefer, B., Spurk, C., Steffen, J., and Wang, R. (2011). The ACL Anthology Searchbench. In Proceedings of the ACL-HLT 2011 System Demonstrations, pages 7–13, Portland, Oregon. Association for Computational Linguistics. find
  30. Schäfer, U., Read, J., and Oepen, S. (2012). Towards an ACL Anthology Corpus with logical document structure. An overview of the ACL 2012 contributed task. In Proceedings of the ACL-2012 Special Workshop on Rediscovering 50 Years of Discoveries, pages 88–97, Jeju Island, Korea. Association for Compu… find
  31. Stoyanov, V., Babbar, U., Gupta, P., and Cardie, C. (2011). Reconciling OntoNotes: Unrestricted coreference resolution in OntoNotes with Reconcile. In Proceedings of 15th CoNLL: Shared Task, pages 122–126. find
  32. Stoyanov, V., Cardie, C., Gilbert, N., Riloff, E., Buttler, D., and Hysom, D. (2010). Coreference resolution with Reconcile. In Proceedings of the ACL 2010 Conference Short Papers, pages 156–161, Uppsala, Sweden. van Deemter, K. and Kibble, R. (2000). On coreferring: coreference in MUC and related a… find our card
  33. Vilain, M., Burger, J., Aberdeen, J., Connolly, D., and Hirschman, L. (1995). A model-theoretic coreference scoring scheme. In Proceedings of the Fourth Message Understanding Conference (MUC-4), San Mateo, CA. Morgan Kaufmann. find our card
  34. Watson, R., Preiss, J., and Briscoe, E. J. (2003). Contribution of domain-independent robust pronominal anaphora resolution to open-domain question answering. In Proceedings of the International Symposium on Reference Resolution. find
  35. Yang, X., Su, J., Zhou, G., and Tan, C. L. (2004). An NP-cluster based approach to coreference resolution. In Proceedings of COLING-2004, pages 226–232, Geneva, Switzerland. find our card

↑ back to top