Elamur Research
☰
home / papers / 2016 / Language Modeling with Gated Convolutional Networks

Language Modeling with Gated Convolutional Networks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1612.08083

Summary

Recurrent networks, especially LSTMs, have dominated language modeling, partly because they can in principle use unlimited history. However, their step-by-step computation is hard to parallelize across the tokens of a sequence. Dauphin and colleagues at Facebook AI Research instead build a word-level model from stacked causal convolutions with a fixed context window, organized as residual blocks and topped by an adaptive softmax. Their main contribution is the Gated Linear Unit (GLU). It multiplies a linear convolution output by a sigmoid gate, which gives gradients an unscaled linear route through deep stacks.

They evaluate on the Google Billion Word and WikiText-103 benchmarks:

  • On WikiText-103, the model sets a new state of the art.
  • On Google Billion Word, it is competitive with strong LSTMs while using fewer resources.
  • It scores sentences with roughly ten times lower latency than a recurrent baseline.

This matters because it shows that non-recurrent, parallelizable architectures can match recurrent ones on large-scale language modeling. It also suggests that very long contexts bring limited benefit.

Key points

  • Data: Google Billion Word has about 1B tokens, a vocabulary of over 800K, and 30,301,028 shuffled sentences, with one sentence per sequence. WikiText-103 has over 100M tokens and about 200K vocabulary, with paragraph-level sequences. Models were implemented in Torch and trained on Tesla M40 GPUs.
  • Gating comparison: GLU beats the LSTM-style gated tanh unit (GTU) of Oord et al., as well as Tanh and ReLU, both in convergence speed and in perplexity. On Google Billion Word, GLU leads ReLU by about 5 perplexity points.
  • Non-linearity ablation on Google Billion Word:
    • A purely linear model reaches perplexity 115, worse than Kneser-Ney 5-gram (67.6).
    • Bilinear layers reach 61.
    • GLU improves roughly 20 points further.
  • Context size: gains diminish sharply beyond about 40 words. On WikiText-103, a context of only about 30 tokens already performs strongly, even though documents average close to 4000 tokens.
  • Training: weight normalization more than doubles convergence speed, partly by allowing a learning rate of 1 instead of 0.01. Gradient clipping also speeds training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

225 papers of our library cite it (of the 140673 whose bibliographies we read); 24 of them have a page here, 201 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1612.08083 · arXiv 1612.08083

@misc{dauphin2016language,
  title = {{Language Modeling with Gated Convolutional Networks}},
  author = {Dauphin, Yann N. and Fan, Angela and Auli, Michael and Grangier, David},
  year = {2016},
  doi = {10.48550/arxiv.1612.08083},
  eprint = {1612.08083},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1612.08083}
}

References

34 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bengio, Yoshua, Ducharme, Réjean, Vincent, Pascal, and Jauvin, Christian. A neural probabilistic language model. journal of machine learning research, 3(Feb):1137–1155, 2003. find
  2. Chelba, Ciprian, Mikolov, Tomas, Schuster, Mike, Ge, Qi, Brants, Thorsten, Koehn, Phillipp, and Robinson, Tony. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005, 2013. find our card
  3. Chen, Stanley F and Goodman, Joshua. An empirical study of smoothing techniques for language modeling. In Proceedings of the 34th annual meeting on Association for Computational Linguistics, pp. 310–318. Association for Computational Linguistics, 1996. find
  4. Chen, Wenlin, Grangier, David, and Auli, Michael. Strategies for training large vocabulary neural language models. CoRR, abs/1512.04906, 2016. find
  5. Collobert, Ronan, Kavukcuoglu, Koray, and Farabet, Clement. Torch7: A Matlab-like Environment for Machine Learning. In BigLearn, NIPS Workshop, 2011. URL http://torch.ch. ↗
  6. Dauphin, Yann N and Grangier, David. Predicting distributions with linearizing belief networks. arXiv preprint arXiv:1511.05622, 2015. — [arXiv:1511.05622](https://arxiv.org/abs/1511.05622) find
  7. Glorot, Xavier and Bengio, Yoshua. Understanding the difficulty of training deep feedforward neural networks. The handbook of brain theory and neural networks, 2010. find
  8. Grave, E., Joulin, A., Cissé, M., Grangier, D., and Jégou, H. Efficient softmax approximation for GPUs. ArXiv e-prints, September 2016a. find
  9. Grave, E., Joulin, A., and Usunier, N. Improving Neural Language Models with a Continuous Cache. ArXiv e-prints, December 2016b. find
  10. Gutmann, Michael and Hyvärinen, Aapo. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385, 2015a. find
  11. He, Kaiming, Zhang, Xiangyu, Ren, Shaoqing, and Sun, Jian. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034, 2015b. find
  12. Hochreiter, Sepp and Schmidhuber, Jürgen. Long short-term memory. Neural computation, 9(8):1735–1780, 1997. find
  13. Ji, Shihao, Vishwanathan, SVN, Satish, Nadathur, Anderson, Michael J, and Dubey, Pradeep. Blackout: Speeding up recurrent neural network language models with very large vocabularies. arXiv preprint arXiv:1511.06909, 2015. find
  14. Jozefowicz, Rafal, Vinyals, Oriol, Schuster, Mike, Shazeer, Noam, and Wu, Yonghui. Exploring the limits of language modeling. arXiv preprint arXiv:1602.02410, 2016. find
  15. Kalchbrenner, Nal, Espeholt, Lasse, Simonyan, Karen, van den Oord, Aaron, Graves, Alex, and Kavukcuoglu, Koray. Neural Machine Translation in Linear Time. arXiv, 2016. find
  16. Kneser, Reinhard and Ney, Hermann. Improved backing-off for m-gram language modeling. In Acoustics, Speech, and Signal Processing, 1995. ICASSP-95., 1995 International Conference on, volume 1, pp. 181–184. find
  17. Koehn, Philipp. Statistical Machine Translation. Cambridge University Press, New York, NY, USA, 1st edition, 2010. ISBN 0521874157, 9780521874151. find
  18. Kuchaiev, Oleksii and Ginsburg, Boris. Factorization tricks for LSTM networks. CoRR, abs/1703.10722, 2017. URL http: //arxiv.org/abs/1703.10722. — [arXiv:1703.10722](https://arxiv.org/abs/1703.10722) find
  19. LeCun, Yann and Bengio, Yoshua. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995. find
  20. Manning, Christopher D and Schütze, Hinrich. Foundations of statistical natural language processing, 1999. find
  21. Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer Sentinel Mixture Models. ArXiv e-prints, September 2016. find our card
  22. Mikolov, Tomáš, Martin, Karafiát, Burget, Lukáš, Cernocký, Jan, and Khudanpur, Sanjeev. Recurrent Neural Network based Language Model. In Proc. of INTERSPEECH, pp. 1045–1048, 2010. find
  23. Mnih, Andriy and Hinton, Geoffrey. Three new graphical models for statistical language modelling. In Proceedings of the 24th international conference on Machine learning, pp. 641–648. ACM, 2007. find
  24. Morin, Frederic and Bengio, Yoshua. Hierarchical probabilistic neural network language model. In Aistats, volume 5, pp. 246– 252. Citeseer, 2005. find
  25. Oord, Aaron van den, Kalchbrenner, Nal, and Kavukcuoglu, Koray. Pixel recurrent neural networks. arXiv preprint arXiv:1601.06759, 2016a. — [arXiv:1601.06759](https://arxiv.org/abs/1601.06759) find
  26. Oord, Aaron van den, Kalchbrenner, Nal, Vinyals, Oriol, Espeholt, Lasse, Graves, Alex, and Kavukcuoglu, Koray. Conditional image generation with pixelcnn decoders. arXiv preprint arXiv:1606.05328, 2016b. — [arXiv:1606.05328](https://arxiv.org/abs/1606.05328) find
  27. Pascanu, Razvan, Mikolov, Tomas, and Bengio, Yoshua. On the difficulty of training recurrent neural networks. In Proceedings of The 30th International Conference on Machine Learning, pp. 1310–1318, 2013. find
  28. Salimans, Tim and Kingma, Diederik P. Weight normalization: A simple reparameterization to accelerate training of deep neural networks. arXiv preprint arXiv:1602.07868, 2016. — [arXiv:1602.07868](https://arxiv.org/abs/1602.07868) find
  29. Shazeer, Noam, Pelemans, Joris, and Chelba, Ciprian. Skip-gram language modeling using sparse non-negative matrix probability estimation. arXiv preprint arXiv:1412.1454, 2014. — [arXiv:1412.1454](https://arxiv.org/abs/1412.1454) find
  30. Shazeer, Noam, Mirhoseini, Azalia, Maziarz, Krzysztof, Davis, Andy, Le, Quoc V., Hinton, Geoffrey E., and Dean, Jeff. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. CoRR, abs/1701.06538, 2017. URL http: //arxiv.org/abs/1701.06538. find
  31. Steedman, Mark. The syntactic process. 2002. find
  32. Sutskever, Ilya, Martens, James, Dahl, George E, and Hinton, Geoffrey E. On the importance of initialization and momentum in deep learning. 2013. find
  33. Wang, Mingxuan, Lu, Zhengdong, Li, Hang, Jiang, Wenbin, and Liu, Qun. gencnn: A convolutional architecture for word sequence prediction. CoRR, abs/1503.05034, 2015. URL http://arxiv.org/abs/1503.05034. ↗
  34. Yu, Dong and Deng, Li. Automatic Speech Recognition: A Deep Learning Approach. Springer Publishing Company, Incorporated, 2014. ISBN 1447157788, 9781447157786. find