Elamur Research⌕
☰
home / papers / 2008 / The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics

The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics

Summary

Studies on scholarly document processing, such as metadata extraction, reference parsing and citation matching, often used the ACL Anthology as test data. Each group picked its own subset at a different moment, so results could not be compared. General IR collections such as TREC and CLEF are mostly newswire and do not fit bibliographic work. The authors address this by releasing the ACL Anthology Reference Corpus (ACL ARC), a fixed, versioned snapshot of the Anthology as it stood in February 2007. The release contains:

  • the original PDFs of 10,921 papers,
  • plain text extracted automatically with PDFBox,
  • BibTeX-style metadata checked against the Anthology website, which serves as gold standard for evaluating header-metadata extraction.

The paper also outlines planned gold-standard annotations for linking citations inside papers and between papers. It places the corpus within community efforts such as the Linked Anthology proposal and the iOPENER survey-generation project. The main contribution is a shared, frozen testbed. It lets research on digital libraries and bibliometrics in NLP be benchmarked consistently, in contrast to live collections that change every day.

Key points

  • Release contents: ACL ARC (version 20080325) covers 10,921 articles from the February 2007 Anthology snapshot. At the time of writing, the live Anthology held about 14,000 articles.
  • Text extraction and filtering: Text was extracted directly from PDF with PDFBox 0.72, without OCR. The 476 papers (about 4%) that yielded no output or crashed the extractor were dropped. Garbled output came mainly from custom font encodings.
  • Reference statistics: The corpus contains 152,546 references. Of these, 38,767 (25.4%) point to papers inside ACL ARC and 113,779 (74.6%) point to work outside it.
  • Paper IDs: Each paper has an ID that encodes the venue letter, the year and a sequence number. For example, P00-1004 is the fourth paper of the ACL 2000 main conference.
  • History of the Anthology:
    • Steven Bird proposed it in 2001.
    • More than US$50,000 in donations funded the digitization, with pages scanned at 600dpi and downsampled to 300dpi.
    • It launched in 2002 with 3,100 papers.

Interesting ideas

  • I1. Objective inclusion rule: Documents were kept if extraction produced any output, rather than judged against a subjective quality threshold. This makes corpus membership reproducible, and the same rule could be used to filter noisy PDF or OCR collections.
  • I2. Freeze and version a living source: The corpus takes snapshots of a constantly updated archive and publishes only major revisions. This turns a moving target into a fair benchmark, which suits any dataset built from websites, registries or trade databases.
  • I3. Push corrections upstream: Metadata errors found while building the corpus were reported back to the Anthology editor. The benchmark effort thereby improves the source resource too, which is a cheap way to repay data providers.
  • I4. Self-describing stable identifiers: IDs that encode venue, year and position are readable by humans and easy to parse. This simplifies joins between metadata, text and citation links.
  • I5. Annotate citation scope, not just citation targets: The planned gold data covers which span of text a citation actually supports, which can extend across clause and sentence boundaries. This finer target could help citation-aware summarization or claim attribution in other domains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

126 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 123 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bird2008acl,
  title = {{The ACL Anthology Reference Corpus: A Reference Dataset for Bibliographic Research in Computational Linguistics}},
  author = {Bird, Steven and Dale, Robert and Dorr, Bonnie and Gibson, Bryan and Joseph, Mark and Kan, Min-Yen and Lee, Dongwon and Powley, Brett and Radev, Dragomir and Tan, Yee Fan},
  booktitle = {Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08)},
  year = {2008},
  doi = {10.63317/4pzr64y6ri9j},
  url = {https://doi.org/10.63317/4pzr64y6ri9j}
}

References

17 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aaron Elkiss, Siwei Shen, Anthony Fader, Güneş Erkan, David States, and Dragomir R. Radev. 2008. Blind men and elephants: what do citation summaries tell us about a research article? Journal of the American Society for Information Science and Technology, 59(1):51–62. find
  2. Mark T. Joseph and Dragomir R. Radev. 2007. Citation analysis, centrality, and the acl anthology. Technical Report CSE-TR-535-07, University of Michigan. Department of Electrical Engineering and Computer Science. find
  3. Min-Yen Kan and Yee Fan Tan. 2008. Record matching in digital library metadata. Communications of the ACM (CACM), 51(2):91–94. find
  4. Min-Yen Kan. 2007. SlideSeer: A digital library of aligned document and presentation pairs. In Proceedings of the Joint Conference on Digital Libraries (JCDL ’07), pages 81–90, Vancouver, Canada, June. find
  5. Steve Lawrence, C. Lee Giles, and Kurt Bollacker. 1999. Digital libraries and autonomous citation indexing. IEEE Computer, 32(6):67–71. find
  6. Dongwon Lee. 2007. Toward web-scale academic video search engine. Technical report, The Pennsylvania State University, December. find
  7. Michael Ley. 2002. The DBLP computer science bibliography: Evolution, research issues, perspectives. In International Symposium on String Processing and Information Retrieval (SPIRE), pages 1–10, September. find
  8. Thuy Dung Nguyen and Min-Yen Kan. 2007. Keyphrase extraction in scientific publications. In Proc. of International Conference on Asian Digital Libraries (ICADL ’07), pages 317–326, Hanoi, Vietnam. Springer. find
  9. Byung-Won On. 2007. Data Cleaning Techniques by means of Entity Resolution. Ph.D. thesis, The Pennsylvania State University, May. find
  10. Fuchun Peng and Andrew McCallum. 2004. Accurate information extraction from research papers using conditional random fields. In Proceedings of Human Language Technology Conference / North American Chapter of the Association for Computational Linguistics annual meeting, pages 329–336. find
  11. Brett Powley and Robert Dale. 2007. Evidence-based information extraction for high accuracy citation and author name identification. In Proceedings of RIAO 2007: the 8th Conference on Large-Scale Semantic Access to Content, Pittsburgh, Pa., USA, 30 May to 1 June. find
  12. Dragomir R. Radev, Timothy Allison, Sasha Blair-Goldensohn, John Blitzer, Arda Celebi, Stanko Dimitrov, Elliott Drabek, Ali Hakim, Wai Lam, Danyu Liu, Jahna Otterbacher, Hong Qi, Horacio Saggion, Simone Teufel, Michael Topper, Adam Winkel, and Zhu Zhang. 2004. MEAD: A platform for multidocument mult… find our card
  13. Dragomir R. Radev, Jahna Otterbacher, Adam Winkel, and Sasha Blair-Goldensohn. 2005. NewsInEssence: Summarizing online news topics. Communications of the ACM, 48(10):95–98. find
  14. Ariel Schwartz, Anna Divoli, and Marti Hearst. 2007. Multiple alignment of citation sentences with conditional random fields and posterior decoding. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNL… find
  15. Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006. Automatic classification of citation function. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 103–110, Sydney, Australia, July. Association for Computational Linguistics. find
  16. John White. 2001. ACM opens portal to computing literature. Communications of the ACM (CACM), 44(7):14– 16,28, July. find
  17. David M. Zajic, Bonnie J. Dorr, Jimmy Lin, and Richard Schwartz. 2007. Multi-candidate reduction: Sentence compression as a tool for document summarization tasks. Information Processing and Management Special Issue on Summarization, 43(6):1549–1570. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bird-2008-acl-anthology-reference-corpus-reference/