Elamur Research⌕
☰
home / papers / 2012 / Annotated Gigaword

Annotated Gigaword

Summary

Large-scale knowledge extraction and distributional semantics research has usually relied on privately processed corpora. Each group re-parsed its own text with its own tools, so results were hard to reproduce or compare closely. The authors address this by adding automatic linguistic annotation to the whole English Gigaword v.5 newswire collection: nearly 10 million documents from seven outlets, with over 4 billion words. The layers are sentence segmentation, tokenization, Penn-Treebank-style constituency parses, three variants of Stanford typed dependencies, named entities, and within-document coreference chains. All were produced with tools that were state of the art at the time.

The result is distributed through the LDC together with a Java API and command-line utilities for reading it efficiently. The authors present it as a shared, consistent reference corpus. It is roughly ten times larger than earlier parsed corpora available from the LDC, and it lets groups without large compute budgets work at this scale. Two early uses illustrate its value:

  • discovering DIRT-style relation paraphrases;
  • building rich phrase feature vectors for paraphrase scoring.

Key points

  • Pipeline and tools:
    • Splitta for sentence splitting.
    • The Stanford CoreNLP PTB tokenizer.
    • The Huang et al. (2010) self-trained ST-Prod grammar for parsing, which scores an average F of 91.4 on WSJ section 22.
    • A modified Stanford CoreNLP pipeline that takes these external parses and produces dependencies (basic, collapsed, and collapsed with propagated conjunctions) and NER.
    • The Lee et al. (2011) system for coreference.
  • Scale:
    • 183,119,464 sentences were parsed.
    • Sentences longer than 100 tokens were skipped; in one sampled file this removed 1,197 of 548,409 sentences, under 1%.
    • The annotated release is about 400 GB uncompressed, against roughly 26 GB for raw Gigaword.
  • Compute cost:
    • Parsing dominated the effort, needing about 10.5 days per Gigaword file with 8 cores and 16 GB of memory.
    • On six 48-core, 128 GB machines, throughput was around 700k lines per hour.
  • Access tooling:
    • A Java API built on VTD-XML supports random access and a streaming mode that keeps only one document in memory at a time.
    • Counting dependency edges over a 470 MB file of 33,108 sentences took just over 30 seconds, using one core and 64 MB of heap.
  • Downstream evidence: Ganitkevitch et al. (2012) built 12 million feature vectors from this corpus. Human judges rated the resulting paraphrase output significantly better than output from a far larger set of 200 million phrases drawn from a web-scale n-gram corpus.

Interesting ideas

  • I1. Shared reference annotation layers. Releasing one canonical, fully preprocessed version of a big corpus removes hidden variation between labs' preprocessing. A shared annotated Ukrainian news or web corpus could make results across projects directly comparable in the same way.
  • I2. Cheap heuristic filtering before costly parsing. The authors flagged list-like lines with a simple whitespace rule and skipped sentences over 100 tokens, which were mostly segmentation failures. They also recorded which sentences were dropped, so the corpus stays transparent while compute is not wasted on junk input.
  • I3. Swapping one component in an existing pipeline. They plugged a stronger external parser into CoreNLP and still kept its downstream modules. This lets you upgrade the weakest stage without rebuilding everything, which is useful when a better parser exists for a language but the surrounding toolkit is fixed.
  • I4. Memory-light access to huge XML. Memory-mapped, index-based XML reading plus per-document streaming lets very large annotated files be processed on modest hardware. Any project shipping a big annotated corpus could provide a similar lightweight reader alongside it.
  • I5. Richer features beating raw volume. Syntactically informed context features from a smaller parsed corpus outperformed a much larger n-gram collection. This suggests investing in annotation depth rather than raw size for distributional tasks, especially for languages with less data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

154 papers of our library cite it (of the 140884 whose bibliographies we read); 18 of them have a page here, 136 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{napoles2012annotated,
  title = {{Annotated Gigaword}},
  author = {Napoles, Courtney and Gormley, Matthew and Durme, Benjamin Van},
  booktitle = {Proceedings of the Joint Workshop on Automatic Knowledge Base Construction and Web-scale Knowledge Extraction (AKBC-WEKEX)},
  year = {2012},
  url = {https://aclanthology.org/W12-3018/}
}

References

22 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nathanael Chambers and Dan Jurafsky. 2008. Unsupervised learning of narrative event chains. In Proceedings of the Annual Meeting of the Association of Computational Linguistics: Human Language Technologies (ACL08: HLT), pages 789–797, Columbus, Ohio. Association for Computational Linguistics. find
  2. Nathanael Chambers and Dan Jurafsky. 2010. A database of narrative schemas. In Proceedings of the Seventh International Conference on Language Resources and Evaluation (LREC’10), Valletta, Malta. European Language Resources Association (ELRA). find
  3. Eugene Charniak, Don Blaheta, Niyu Ge, Keith Hall, John Hale, and Mark Johnson. 2000. BLLIP 1987-89 WSJ Corpus Release 1. Linguistic Data Consortium. find
  4. Kenneth Ward Church and Patrick Hanks. 1990. Word association norms, mutual information, and lexicography. Computational Linguistics, 16(1):22–29. find our card
  5. Marie-Catherine de Marneffe and Christopher D. Manning. 2008. Stanford typed dependencies manual. http://nlp.stanford.edu/software/dependencies manual. pdf. ↗
  6. Adriano Ferraresi, Eros Zanchetta, Marco Baroni, and Silvia Bernardini. 2008. Introducing and evaluating ukWaC, a very large web-derived corpus of English. In Proceedings of the 4th Web as Corpus Workshop (WAC-4) Can we beat Google, pages 47–54. find
  7. W. Nelson Francis and Henry Kuc̆era. 1964, 1971, 1979. A Standard Corpus of Present-Day Edited American English, for use with Digital Computers (Brown). find
  8. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2012. Monolingual distributional similarity for text-to-text generation. In Proceedings of the First Joint Conference on Lexical and Computational Semantics (*SEM). find
  9. Dan Gillick. 2009. Sentence boundary detection and the problem with the US. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, Companion Volume: Short Papers, pages 241–244, Boulder, Colorado. Asso… find
  10. Zhongqiang Huang, Mary Harper, and Slav Petrov. 2010. Self-training with products of latent variable grammars. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 12–22, Cambridge, MA. Association for Computational Linguistics. find
  11. Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Jurafsky. 2011. Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 28–… find our card
  12. Dekang Lin and Patrick Pantel. 2001. DIRT: Discovery of inference rules from text. In Proceedings of the seventh ACM SIGKDD international conference on Knowledge discovery and data mining, KDD ’01, pages 323–328, New York, NY, USA. ACM. find
  13. David McClosky, Eugene Charniak, and Mark Johnson. 2008. BLLIP North American News Text, Complete. Linguistic Data Consortium. find
  14. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive Science, 34(8):1388–1429. find
  15. Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev, Gülşen Eryiğit, Sandra Kübler, Stetoslav Marinov, and Erwin Marsi. 2007. MaltParser: A language-independent system for data-driven dependency parsing. Natural Language Engineering, 13(02):95–135. find
  16. Robert Parker, David Graff, Junbo Kong, Ke Chen, and Kazuaki Maeda. 2011. English Gigaword Fifth Edition. Linguistic Data Consortium. find
  17. Hoifung Poon and Pedro Domingos. 2009. Unsupervised semantic parsing. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 1–10, Singapore. Association for Computational Linguistics. find our card
  18. Deepak Ravichandran, Patrick Pantel, and Eduard Hovy. 2005. Randomized algorithms and NLP: Using locality sensitive hash functions for high speed noun clustering. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL’05), pages 622–629, Ann Arbor, Michigan.… find
  19. Peter D. Turney and Patrick Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research (JAIR), 37(1):141–188. find
  20. Benjamin Van Durme. 2012. Jerboa: A toolkit for randomized and streaming algorithms. Technical Report 7, Human Language Technology Center of Excellence, Johns Hopkins University. find
  21. Alexander Yates, Michele Banko, Matthew Broadhead, Michael Cafarella, Oren Etzioni, and Stephen Soderland. 2007. TextRunner: Open information extraction on the web. In Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational… find
  22. Jimmy Zhang. 2008. VTD-XML: XML processing for the future (Part I). http://www.codeproject.com/Articles/23516/VTD-XML-XML-Processing-for-the-Future-Part-I. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/napoles-2012-annotated-gigaword/