Elamur Research⌕
☰
home / papers / 2018 / Character-Level Language Modeling with Deeper Self-Attention

Character-Level Language Modeling with Deeper Self-Attention

Paper · 2018 · Rami Al-Rfou, Dokook Choe, Noah Constant et al.

Summary

Character-level language models have long been built on RNNs such as LSTMs. These are trained with truncated backpropagation through time, which carries hidden state from one batch to the next and is usually credited with capturing long contexts. The authors test whether a purely attention-based model with a fixed window can outperform them. They train a 64-layer transformer that uses causal (left-only) attention on 512-character windows drawn from random corpus positions, with no state shared between batches. Networks this deep trained poorly, so the authors add three auxiliary losses: predicting at every sequence position, predicting from each intermediate layer, and predicting characters further ahead. They also swap the sinusoidal timing signal for learned positional embeddings injected before every layer. On text8 and enwik8 the model sets new state-of-the-art scores and exploits dependencies spanning over 400 characters. The result suggests transformers can replace recurrent networks for modeling long character sequences.

Key points

  • Results: the model scored 1.13 bpc on text8 and 1.06 bpc on enwik8, outperforming RNN variants by a wide margin. Even a 12-layer transformer reached state of the art.
  • Architecture (T64): 64 layers with 2 attention heads each, hidden size 512, filter size 2048, a 256-way byte output, and roughly 235M parameters. Some of these parameters are used only during training.
  • Auxiliary loss schedule: losses from intermediate layers are removed one by one, and all of them are gone halfway through training. Two-step-ahead targets are weighted at 0.5. At inference, only the final position of the final layer is used.
  • Training: 4M steps with batches of 16 sequences, dropout 0.55, a momentum optimizer (0.99), and a fixed learning rate of 0.003.
  • Data: both text8 (27-character vocabulary) and enwik8 (raw Wikipedia bytes with markup) are split into 90M/5M/5M characters for train/dev/test.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,462 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo and Llion Jones. 2018. Character-Level Language Modeling with Deeper Self-Attention. arXiv:1808.04444.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1808.04444 · arXiv 1808.04444

@misc{alrfou2018character,
  title = {{Character-Level Language Modeling with Deeper Self-Attention}},
  author = {Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo and Llion Jones},
  year = {2018},
  doi = {10.48550/arxiv.1808.04444},
  eprint = {1808.04444},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1808.04444}
}

References

48 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arjovsky, M.; Shah, A.; and Bengio, Y. 2015. Unitary evolution recurrent neural networks. CoRR abs/1511.06464. — [arXiv:1511.06464](https://arxiv.org/abs/1511.06464) find
  2. Bengio, Y.; Ducharme, R.; Vincent, P.; and Janvin, C. 2003. A neural probabilistic language model. J. Mach. Learn. Res. 3:1137–1155.
  3. Chelba, C.; Mikolov, T.; Schuster, M.; Ge, Q.; Brants, T.; Koehn, P.; and Robinson, T. 2013. One billion word benchmark for measuring progress in statistical language modeling. arXiv preprint arXiv:1312.3005.
  4. Cho, K.; van Merrienboer, B.; Gülçehre, Ç.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. CoRR abs/1406.1078.
  5. Chung, J.; Ahn, S.; and Bengio, Y. 2016. Hierarchical multiscale recurrent neural networks. arXiv preprint arXiv:1609.01704.
  6. Chung, J.; Gulcehre, C.; Cho, K.; and Bengio, Y. 2015. Gated feedback recurrent neural networks. In International Conference on Machine Learning, 2067–2075. find
  7. Conneau, A.; Schwenk, H.; Barrault, L.; and LeCun, Y. 2016. Very deep convolutional networks for natural language processing. CoRR abs/1606.01781. find
  8. Cooijmans, T.; Ballas, N.; Laurent, C.; and Courville, A. C. 2016. Recurrent batch normalization. CoRR abs/1603.09025. — [arXiv:1603.09025](https://arxiv.org/abs/1603.09025) find
  9. Daniluk, M.; Rocktäschel, T.; Welbl, J.; and Riedel, S. 2017. find
  10. Frustratingly short attention spans in neural language modeling. CoRR abs/1702.04521. — [arXiv:1702.04521](https://arxiv.org/abs/1702.04521) find
  11. Dauphin, Y. N.; Fan, A.; Auli, M.; and Grangier, D. 2016. find
  12. Language modeling with gated convolutional networks. CoRR abs/1612.08083.
  13. Gal, Y., and Ghahramani, Z. 2015. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. ArXiv e-prints.
  14. Grave, E.; Joulin, A.; and Usunier, N. 2016. Improving neural language models with a continuous cache. CoRR abs/1612.04426.
  15. Hochreiter, S., and Schmidhuber, J. 1997. Long short-term memory. 9:1735–80. find
  16. Hochreiter, S.; Bengio, Y.; Frasconi, P.; Schmidhuber, J.; et al. 2001. Gradient flow in recurrent nets: the difficulty of learning long-term dependencies. find
  17. Ioffe, S., and Szegedy, C. 2015. Batch normalization: Accelerating deep network training by reducing internal covariate shift. find
  18. CoRR abs/1502.03167. Józefowicz, R.; Vinyals, O.; Schuster, M.; Shazeer, N.; and Wu, Y. 2016. Exploring the limits of language modeling. CoRR abs/1602.02410. find
  19. Kalchbrenner, N.; Espeholt, L.; Simonyan, K.; Oord, A. v. d.; Graves, A.; and Kavukcuoglu, K. 2016. Neural machine translation in linear time. arXiv preprint arXiv:1610.10099. find
  20. Ke, N. R.; Goyal, A.; Bilaniuk, O.; Binas, J.; Charlin, L.; Pal, C.; and Bengio, Y. 2017. Sparse attentive backtracking: Long-range credit assignment in recurrent networks. arXiv preprint arXiv:1711.02326. — [arXiv:1711.02326](https://arxiv.org/abs/1711.02326) find
  21. Kenter, T.; Jones, L.; and Hewlett, D. 2018. Byte-level machine reading across morphologically varied languages. find
  22. Khandelwal, U.; He, H.; Qi, P.; and Jurafsky, D. 2018. Sharp nearby, fuzzy far away: How neural language models use context. In Association for Computational Linguistics (ACL).
  23. Knol, B. 2017. cmix - http://www.byronknoll.com/cmix.html. ↗
  24. Krause, B.; Lu, L.; Murray, I.; and Renals, S. 2016. Multiplicative lstm for sequence modelling. arXiv preprint arXiv:1609.07959. — [arXiv:1609.07959](https://arxiv.org/abs/1609.07959) find
  25. Krause, B.; Kahembwe, E.; Murray, I.; and Renals, S. 2017. find
  26. Dynamic evaluation of neural sequence models. arXiv preprint arXiv:1709.07432. Krueger, D.; Maharaj, T.; Kramár, J.; Pezeshki, M.; Ballas, N.; Ke, N. R.; Goyal, A.; Bengio, Y.; Courville, A.; and Pal, C. 2016. Zoneout: Regularizing rnns by randomly preserving hidden activations. arXiv preprint arXi… find
  27. Lei Ba, J.; Kiros, J. R.; and Hinton, G. E. 2016. Layer Normalization. ArXiv e-prints.
  28. Li, S.; Li, W.; Cook, C.; Zhu, C.; and Gao, Y. 2018. Independently recurrent neural network (indrnn): Building A longer and deeper RNN. CoRR abs/1803.04831. — [arXiv:1803.04831](https://arxiv.org/abs/1803.04831) find
  29. Mahoney, M. 2009. Large text compression benchmark. http://www.mattmahoney.net/text/text.html. ↗
  30. Merity, S.; Keskar, N. S.; and Socher, R. 2017. Regularizing and optimizing LSTM language models. CoRR abs/1708.02182.
  31. Mikolov, T.; Karafit, M.; Burget, L.; Cernock, J.; and Khudanpur, S. 2010. Recurrent neural network based language model. find
  32. In Kobayashi, T.; Hirose, K.; and Nakamura, S., eds., INTER-SPEECH, 1045–1048. ISCA. Mikolov, T.; Kombrink, S.; Burget, L.; ernock, J.; and Khudanpur, S. 2011. Extensions of recurrent neural network language model. In 2011 IEEE International Conference on Acoustics, Speech and Signal Processing (ICA… find
  33. Mikolov, T.; Sutskever, I.; Deoras, A.; Le, H.-S.; Kombrink, S.; and Cernocky, J. 2012. Subword language modeling with neural networks. preprint (http://www. fit. vutbr. cz/imikolov/rnnlm/char. pdf) 8. find
  34. Mujika, A.; Meier, F.; and Steger, A. 2017. Fast-slow recurrent neural networks. In Advances in Neural Information Processing Systems, 5915–5924. find
  35. Radford, A.; Józefowicz, R.; and Sutskever, I. 2017. Learning to generate reviews and discovering sentiment. CoRR abs/1704.01444.
  36. Rocki, K. M. 2016. Surprisal-driven feedback in recurrent networks. arXiv preprint arXiv:1608.06027. — [arXiv:1608.06027](https://arxiv.org/abs/1608.06027) find
  37. Salimans, T.; Zhang, H.; Radford, A.; and Metaxas, D. N. 2018. Improving gans using optimal transport. CoRR abs/1803.05573. find
  38. Shazeer, N.; Mirhoseini, A.; Maziarz, K.; Davis, A.; Le, Q. V.; Hinton, G. E.; and Dean, J. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. CoRR abs/1701.06538. find
  39. Sukhbaatar, S.; Weston, J.; Fergus, R.; et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, 2440–2448. find
  40. Sundermeyer, M.; Schlüter, R.; and Ney, H. 2012. Lstm neural networks for language modeling. In Thirteenth annual conference of the international speech communication association. find
  41. Tallec, C., and Ollivier, Y. 2017. Unbiasing truncated backpropagation through time. arXiv preprint arXiv:1705.08209. — [arXiv:1705.08209](https://arxiv.org/abs/1705.08209) find
  42. Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30. Curran Associates, Inc. 5998–6008.
  43. Werbos, P. J. 1990. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE 78(10):1550–1560. find
  44. Weston, J.; Chopra, S.; and Bordes, A. 2014. Memory networks. CoRR abs/1410.3916. find
  45. Zaremba, W.; Sutskever, I.; and Vinyals, O. 2014. Recurrent neural network regularization. CoRR abs/1409.2329.
  46. Zhang, S.; Wu, Y.; Che, T.; Lin, Z.; Memisevic, R.; Salakhutdinov, R. R.; and Bengio, Y. 2016. Architectural complexity measures of recurrent neural networks. In Advances in Neural Information Processing Systems, 1822–1830. find
  47. Zhang, X.; Zhao, J. J.; and LeCun, Y. 2015. Character-level convolutional networks for text classification. CoRR abs/1509.01626.
  48. Zilly, J. G.; Srivastava, R. K.; Koutnı́k, J.; and Schmidhuber, J. 2016. Recurrent highway networks. CoRR abs/1607.03474.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/al-rfou-2018-character-level-language-modeling-deeper/