Elamur Research⌕
☰
home / papers / 2013 / A Dataset of Syntactic-Ngrams over Time from a Very Large Corpus of English Books

A Dataset of Syntactic-Ngrams over Time from a Very Large Corpus of English Books

Paper · 2013 · Yoav Goldberg and Jon Orwant

Summary

Large corpora are widely used to learn word properties from context, but syntactic counts at web scale were hard to obtain, and studies seldom shared a common corpus. The authors built a standard, reusable resource by parsing the English Google Books collection: about 3.5 million books, 345 billion tokens, published from 1520 to 2008. They tagged the text with a CRF tagger, parsed it into basic Stanford dependencies with a beam-search shift-reduce parser, and counted tree fragments called syntactic-ngrams. These fragments are defined by how many content words they contain, not by their position in the linear string. The released dataset has more than 10 billion distinct fragments. Each one comes with a total count and per-year counts, plus counts for subcorpora such as Fiction, British, American and a uniform 1M-book sample. The paper offers no new method. Its value is a free, large, temporally indexed set of counts that supports work on selectional preferences, subcategorization, syntactic language modeling and change in word meaning, without anyone having to rerun parsing at this scale.

Key points

  • Source corpus: English Google Books, 3,473,595 books, 17.6B sentences and 345.1B tokens.
    • Pages with more than 600 white-spaces and sentences longer than 60 tokens were dropped.
    • A fragment is kept only if it occurs at least 10 times in the whole corpus.
  • Processing pipeline:
    • Penn Treebank POS tags from a first-order CRF tagger. It was trained on WSJ, Brown and the Questions Treebank and uses word-cluster features derived from Books trigrams.
    • A reimplemented Zhang & Clark shift-reduce parser with beam size 8 and Zhang & Nivre features, trained with 4-way jack-knifing.
    • Output follows the basic variant of Stanford dependencies.
  • Fragment sets:
    • nodes (47M), arcs (919M), biarcs (1.78B) and triarcs (1.87B), plus larger quadarcs.
    • "extended-*" versions that also keep functional markers.
    • verbargs (130M) and nounargs (275M), each with an unlexicalized variant that keeps only the head and the top-1000 words.
  • Temporal example: comparing the syntactic contexts of "rock" year by year shows it moving toward "jazz" around 1968.
  • Release: the data is free under CC BY-NC-SA 3.0. The authors deliberately report no parsing accuracy figure, arguing that a single number would mislead on such a heterogeneous corpus.

Interesting ideas

  • I1. Fragment size is measured in content words only, and determiners, auxiliaries, negation and similar words are attached as optional "functional markers" in separate extended sets; this keeps the counts focused on meaning while still letting users recover polarity, modality and definiteness.
  • I2. Prepositions and coordinators stay as tree nodes rather than being folded into labels like prep_with; this preserves where they occur, still works when one side is missing (as in "hard to deal with"), and keeps the label inventory small.
  • I3. Verb-argument patterns are unlexicalized by replacing every word outside the top-1000 with a W placeholder, so that rare but structurally interesting frames survive the frequency cutoff.
  • I4. Counts are split by year, and similarity between a word's syntactic contexts is tracked over time to spot when a new sense appears.
  • I5. The parser is trained on jack-knifed predicted POS tags, and the authors refuse to publish one accuracy number for mixed-genre, mixed-OCR data.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): frames could be keyed on content-word arguments, with negation, modality and definiteness stored as attached features instead of as separate slots.
  • I2 (tree): when converting between UD dialects, keeping case/cc words as nodes avoids an explosion of labels and handles stranded prepositions consistently.
  • I3 (world): compiled subcategorization knowledge for verb classes could be built from such partly abstracted frames, so rare verbs still get usable argument patterns.
  • I4 (world): date-sliced context profiles could flag archaic or shifted senses when reading older fables and tales, helping pick the right WordNet sense.
  • I5 (learn): train the parser on tags from the en tagger rather than gold tags, and report silver-data quality per genre or slice instead of as a single overall score.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States353 works · 364,806 citations🎓Bar-Ilan UniversityRamat Gan · Israel118 works · 21,077 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Literature and Literary Theory · 340 works

🧭 Topics 1 English · 1,373 works

Cite

Cite as Yoav Goldberg and Jon Orwant. 2013. A Dataset of Syntactic-Ngrams over Time from a Very Large Corpus of English Books. Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{goldberg2013dataset,
  title = {{A Dataset of Syntactic-Ngrams over Time from a Very Large Corpus of English Books}},
  author = {Yoav Goldberg and Jon Orwant},
  booktitle = {Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity},
  year = {2013},
  url = {https://aclanthology.org/S13-1035/}
}

References

30 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohit Bansal and Dan Klein. 2011. Web-scale features for full-scale parsing. In ACL, pages 693–702. find
  2. Marco Baroni and Alessandro Lenci. 2010. Distributional memory: A general framework for corpus-based semantics. Computational Linguistics, 36(4):673– 721. ↗ our card
  3. Marco Baroni, Silvia Bernardini, Adriano Ferraresi, and Eros Zanchetta. 2009. The wacky wide web: a collection of very large linguistically processed web-crawled corpora. Language Resources and Evaluation, 43(3):209–226. ↗
  4. Eugene Charniak. 2000. Bllip 1987-89 wsj corpus release 1. In Linguistic Data Consortium, Philadelphia. find
  5. Wenliang Chen, Jun’ichi Kazama, Kiyotaka Uchimoto, and Kentaro Torisawa. 2009. Improving dependency parsing with subtrees from auto-parsed data. In EMNLP, pages 570–579. bers. We therefore chose not to provide a numeric estimation of syntactic-annotation quality, but note that we used a state-of-the-art parser, and believe most of its output to be correct, although we do expect a fair share of annotation errors as well. find
  6. Raphael Cohen, Yoav Goldberg, and Michael Elhadad. 2012. Domain adaptation of a dependency parser with a class-class selectional preference model. In Proceedings of ACL 2012 Student Research Workshop, pages 43–48, Jeju Island, Korea, July. Association for Computational Linguistics. find
  7. Marie-Catherine de Marneffe and Christopher D. Manning. 2008a. Stanford dependencies manual. Technical report, Stanford University. find
  8. Marie-Catherine de Marneffe and Christopher D. Manning. 2008b. The stanford typed dependencies representation. In Coling 2008: Proceedings of the workshop on Cross-Framework and Cross-Domain Parser Evaluation, CrossParser ’08, pages 1–8, Stroudsburg, PA, USA. Association for Computational Linguistics. our card
  9. Katrin Erk and Sebastian Padó. 2008. A structured vector space model for word meaning in context. In Proceedings of EMNLP, Honolulu, HI. To appear. our card
  10. Katrin Erk, Sebastian Padó, and Ulrike Padó. 2010. A flexible, corpus-driven model of regular and inverse selectional preferences. Computational Linguistics, 36(4):723–763. ↗
  11. Zellig Harris. 1954. Distributional structure. Word, 10(23):146–162. find
  12. John Judge, Aoife Cahill, and Josef van Genabith. 2006. Questionbank: Creating a corpus of parse-annotated questions. In Proc. of ACL, pages 497–504. Association for Computational Linguistics. find
  13. Terry Koo, Xavier Carreras, and Michael Collins. 2008. Simple semi-supervised dependency parsing. In Proc. of ACL, pages 595–603. our card
  14. Henry Kucera and W. Nelson Francis. 1967. Computational Analysis of Present-Day American English. Brown University Press. find
  15. Dekang Lin and Patrick Pantel. 2001. Dirt: discovery of inference rules from text. In KDD, pages 323–328. find
  16. Yuri Lin, Jean-Baptiste Michel, Erez Aiden Lieberman, Jon Orwant, Will Brockman, and Slav Petrov. 2012. Syntactic annotations for the google books ngram corpus. In ACL (System Demonstrations), pages 169– 174. find
  17. Dekang Lin. 1998. Automatic retrieval and clustering of similar words. In Proceedings of the 36th Annual Meeting of the Association for Computational Linguistics and 17th International Conference on Computational Linguistics - Volume 2, ACL ’98, pages 768– 774, Stroudsburg, PA, USA. Association for Computational Linguistics. our card
  18. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19:313–330. ↗ our card
  19. Jean-Baptiste Michel, Yuan Kui Shen, Aviva Presser Aiden, Adrian Veres, Matthew K. Gray, The Google Books Team, Joseph P. Pickett, Dale Hoiberg, Dan Clancy, Peter Norvig, Jon Orwant, Steven Pinker, Martin A. Nowak, and Erez Lieberman Aiden. 2011. Quantitative analysis of culture using millions of digitized books. Science, 331(6014):176–182. ↗
  20. Courtney Napoles, Matthew Gormley, and Benjamin Van Durme. 2012. Annotated gigaword. In AKBC-WEKEX Workshop at NAACL 2012, June. our card
  21. Sebastian Padó and Mirella Lapata. 2007. Dependency-based construction of semantic space models. Computational Linguistics, 33(2):161–199. ↗ our card
  22. Emily Pitler. 2012. Attacking parsing bottlenecks with unlabeled data and relevant factorizations. In ACL, pages 768–776. find
  23. Alan Ritter, Mausam, and Oren Etzioni. 2010. A latent dirichlet allocation method for selectional preferences. In ACL, pages 424–434. find
  24. Kenji Sagae and Andrew S. Gordon. 2009. Clustering words by syntactic similarity improves dependency parsing of predicate-argument structures. In IWPT, pages 192–201. find
  25. Diarmuid Ó Séaghdha. 2010. Latent variable models of selectional preference. In ACL, pages 435–444. find
  26. Libin Shen, Jinxi Xu, and Ralph M. Weischedel. 2008. A new string-to-dependency machine translation algorithm with a target dependency language model. In ACL, pages 577–585. our card
  27. P.D. Turney and P. Pantel. 2010. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 37(1):141–188. ↗ our card
  28. Fei Wu and Daniel S. Weld. 2010. Open information extraction using wikipedia. In ACL, pages 118–127. our card
  29. Yue Zhang and Stephen Clark. 2008. A tale of two parsers: Investigating and combining graph-based and transition-based dependency parsing. In Proc. of EMNLP, pages 562–571. our card
  30. Yue Zhang and Joakim Nivre. 2011. Transition-based dependency parsing with rich non-local features. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 188–193. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/goldberg-2013-dataset-syntactic-ngrams-over-time/