Elamur Research
☰
home / papers / 2013 / One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling

One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1312.3005

Summary

Language modeling research lacked a shared, free, large-scale test bed, so new techniques were hard to compare against strong n-gram systems at realistic data sizes. The authors built one from the English monolingual WMT11 news data. They normalized and tokenized the text and dropped repeated sentences, which leaves about 0.8 billion training words and a vocabulary of roughly 793K types; words seen fewer than three times are mapped to <UNK>. The sentences were shuffled and split into 100 partitions. One partition is held out and divided again for development and testing. On this data the authors compared Katz and Interpolated Kneser-Ney 5-grams (pruned and unpruned), Stupid Backoff, two MaxEnt variants, and RNN LMs paired with a MaxEnt component. The RNN models reach the lowest perplexity of any single model. Interpolating all the systems brings perplexity down to 43.8, compared with 67.6 for the KN baseline. The released scripts and per-word log-probabilities make results reproducible and make it cheaper for researchers to test new methods.

Key points

  • The data comes from WMT11. Removing duplicates cuts it from about 2.9B to about 0.8B words (829,250,940 tokens). The test set has 159,658 words and an out-of-vocabulary (OoV) rate of 0.28%.
  • On test perplexity, the unpruned Interpolated KN 5-gram (1.1B n-grams) scores 67.6, Katz scores 79.9 and Stupid Backoff scores 87.9. Aggressive pruning to 15M n-grams hurts KN much more (243.2) than Katz (127.5).
  • RNN + MaxEnt 9-gram models with 256, 512 and 1024 hidden units reach 58.3, 54.5 and 51.3. Each has about 20B parameters and was trained on a single CPU machine with no GPUs. The authors call the largest one the biggest RNN LM reported at that point.
  • Linearly interpolating the models gives 43.8, which is about 35% lower perplexity, or about 10% less cross-entropy. Stupid Backoff gets a surprisingly large weight in the mix (0.20), while KN gets only 0.06.
  • The benchmark was released on code.google.com with rebuild scripts and baseline log-probabilities for ten held-out sets. Because sentence order is shuffled, it can't be used for models that look at context across sentence boundaries.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

327 papers of our library cite it (of the 140673 whose bibliographies we read); 35 of them have a page here, 292 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1312.3005 · arXiv 1312.3005

@misc{chelba2013one,
  title = {{One Billion Word Benchmark for Measuring Progress in Statistical Language Modeling}},
  author = {Chelba, Ciprian and Mikolov, Tomas and Schuster, Mike and Ge, Qi and Brants, Thorsten and Koehn, Phillipp and Robinson, Tony},
  year = {2013},
  doi = {10.48550/arxiv.1312.3005},
  eprint = {1312.3005},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1312.3005}
}

References

32 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Bengio, R. Ducharme, and P. Vincent. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137-1155. find
  2. T. Brants, A. C. Popat, P. Xu, F. J. Och, and J. Dean. 2007. Large language models in machine translation. In Proceedings of EMNLP. find our card
  3. P. F. Brown, P. V. deSouza, R. L. Mercer, V. J. Della Pietra, and J. C. Lai. 1992. Class-Based n-gram Models of Natural Language. Computational Linguistics, 18, 467-479. find
  4. J. Elman. 1990. Finding Structure in Time. Cognitive Science, 14, 179-211. find
  5. A. Emami. 2006. A Neural Syntactic Language Model. Ph.D. thesis, Johns Hopkins University. find
  6. J. T. Goodman. 2001a. A bit of progress in language modeling, extended version. Technical report MSR-TR-2001-72. find
  7. J. T. Goodman. 2001b. Classes for fast maximum entropy training. In Proceedings of ICASSP. find
  8. F. Jelinek, B. Merialdo, S. Roukos, and M. Strauss. 1991. A Dynamic Language Model for Speech Recognition. In Proceedings of the DARPA Workshop on Speech and Natural Language. find
  9. C. Chelba and F. Jelinek. 2000. Structured language modeling. Computer Speech & Language. find
  10. C. Chelba, T. Brants, W. Neveitt, and P. Xu. 2010. Study on Interaction between Entropy Pruning and Kneser-Ney Smoothing. In Proceedings of Interspeech. find
  11. S. F. Chen and J. T. Goodman. 1996. An empirical study of smoothing techniques for language modeling. In Proceedings of ACL. find
  12. S. F. Chen. 2009. Shrinking exponential language models. In Proceedings of NAACL-HLT. find
  13. S. Katz. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. In IEEE Transactions on Acoustics, Speech and Signal Processing. find
  14. R. Kneser and H. Ney. 1995. Improved Backing-Off For M-Gram Language Modeling. In Proceedings of ICASSP. find
  15. T. Mikolov, M. Karafiát, L. Burget, J. Černocký, and S. Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of Interspeech. find
  16. T. Mikolov, S. Kombrink, L. Burget, J. Černocký, and S. Khudanpur. 2011. Extensions ofl Recurrent Neural Network Language Model. In Proceedings of ICASSP. find
  17. T. Mikolov, A. Deoras, S. Kombrink, L. Burget, and J. Černocký. 2011a. Empirical Evaluation and Combination of Advanced Language Modeling Techniques. In Proceedings of Interspeech. find
  18. T. Mikolov, A. Deoras, D. Povey, L. Burget, and J. Černocký. 2011b. Strategies for Training Large Scale Neural Network Language Models. In Proceedings of ASRU. find
  19. T. Mikolov. 2012. Statistical Language Models based on Neural Networks. Ph.D. thesis, Brno University of Technology. find
  20. A. Mnih and G. Hinton. 2007. Three new graphical models for statistical language modelling. In Proceedings of ICML. find
  21. F. Morin and Y. Bengio. 2005. Hierarchical Probabilistic Neural Network Language Model. In Proceedings of AISTATS. find
  22. B. Roark, M. Saralar, M. Collins, and M. Johnson. 2004. Discriminative language modeling with conditional random fields and the perceptron algorithm. In Proceedings of ACL. find
  23. R. Rosenfeld. 1994. Adaptive Statistical Language Modeling: A Maximum Entropy Approach. Ph.D. thesis, Carnegie Mellon University. find
  24. D. E. Rumelhart, G. E. Hinton, and R. J. Williams. 1986. Learning internal representations by back-propagating errors. Nature, 323:533-536. find
  25. H. Schwenk. 2007. Continuous space language models. Computer Speech and Language, vol. 21. find
  26. A. Stolcke. 1998. Entropy-based Pruning of Back-off Language Models. In Proceedings of News Transcription and Understanding Workshop. find
  27. M. Sundermeyer, R. Schluter, and H. Ney. 2012. LSTM Neural Networks for Language Modeling. In Proceedings of Interspeech. find
  28. Y. W. Teh. 2006. A hierarchical Bayesian language model based on PitmanYor processes. In Proceedings of Coling/ACL. find
  29. Y. Wu, H. Yamamoto, X. Lu, S. Matsuda, C. Hori, and H. Kashioka. 2012. Factored Recurrent Neural Network Language Model in TED Lecture Transcription. In Proceedings of IWSLT. find
  30. Peng Xu. 2005. Random forests and the data sparseness problem in language modeling. Ph.D. thesis, Johns Hopkins University. find
  31. Puyang Xu, A. Gunawardana, and S. Khudanpur. 2011. Efficient Subsampling for Training Complex Language Models. In Proceedings of EMNLP. find
  32. G. Zweig and K. Makarychev. 2013. Speed Regularization and Optimality in Word Classing. In Proceedings of ICASSP. find