Elamur Research
☰
home / papers / 2015 / Character-Aware Neural Language Models

Character-Aware Neural Language Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1508.06615

Summary

Standard neural language models give each word its own input vector. Because of that, they cannot see that word forms built from the same parts are related, and they learn weak vectors for rare words. This is worst in morphologically rich languages. Kim, Jernite, Sontag and Rush propose a language model that drops input word embeddings completely. Each word is built from its characters: a convolutional network (CharCNN) with max-over-time pooling reads the characters, a highway network refines the result, and a multi-layer LSTM uses it to predict the next word over a word vocabulary. No morphological tagging is needed. On the English Penn Treebank, the model roughly matches the best results of the time with about 60% fewer parameters. On Arabic, Czech, French, German, Spanish and Russian, it beats Kneser-Ney, word-level LSTM and morpheme-level LSTM baselines while using fewer parameters. The learned word vectors capture both meaning and spelling similarity. The paper suggests that character-level input can be enough for language modeling, which is relevant to inflection-heavy languages such as Ukrainian.

Key points

  • Architecture: character embeddings feed a narrow 1-D convolution with filters of several widths, then max-over-time pooling, then a highway layer, then an LSTM, and finally a softmax over words. Start-of-word and end-of-word markers are added so the model can pick up prefixes and suffixes.
  • On PTB, the large model reaches a perplexity close to the previous best LSTM (about 79 vs 78) with roughly 19M parameters instead of about 52M. These figures are from my knowledge of the full paper, because the provided extract left out the results tables.
  • Feeding the LSTM both word embeddings and CharCNN output made results slightly worse (81 PPL on PTB). Using a plain MLP instead of the highway layer also hurt.
  • Training is about half as fast as a word-level model (1500 vs 3000 tokens/sec on GPU). For scoring, the CharCNN outputs can be computed in advance for every word, which removes that cost.
  • The non-English datasets were preprocessed by Jan Botha. The code is public at github.com/yoonkim/lstm-char-cnn.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

360 papers of our library cite it (of the 140673 whose bibliographies we read); 25 of them have a page here, 335 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1508.06615 · arXiv 1508.06615

@misc{kim2015character,
  title = {{Character-Aware Neural Language Models}},
  author = {Kim, Yoon and Jernite, Yacine and Sontag, David and Rush, Alexander M.},
  year = {2015},
  doi = {10.48550/arxiv.1508.06615},
  eprint = {1508.06615},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1508.06615}
}

References

43 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexandrescu, A., and Kirchhoff, K. 2006. Factored Neural Language Models. In Proceedings of NAACL. find
  2. Ballesteros, M.; Dyer, C.; and Smith, N. A. 2015. Improved Transition-Based Parsing by Modeling Characters instead of Words with LSTMs. In Proceedings of EMNLP. find
  3. Bengio, Y.; Ducharme, R.; and Vincent, P. 2003. A Neural Probabilistic Language Model. Journal of Machine Learning Research 3:1137–1155. find
  4. Bengio, Y.; Simard, P.; and Frasconi, P. 1994. Learning Long-term Dependencies with Gradient Descent is Difficult. IEEE Transactions on Neural Networks 5:157–166. find
  5. Bilmes, J., and Kirchhoff, K. 2003. Factored Language Models and Generalized Parallel Backoff. In Proceedings of NAACL. find
  6. Botha, J., and Blunsom, P. 2014. Compositional Morphology for Word Representations and Language Modelling. In Proceedings of ICML. find
  7. Botha, J. 2014. Probabilistic Modelling of Morphologically Rich Languages. DPhil Dissertation, Oxford University. find our card
  8. Chen, S., and Goodman, J. 1998. An Empirical Study of Smoothing Techniques for Language Modeling. Technical Report, Harvard University. find
  9. Cheng, W. C.; Kok, S.; Pham, H. V.; Chieu, H. L.; and Chai, K. M. 2014. Language Modeling with Sum-Product Networks. In Proceedings of INTERSPEECH. find
  10. Cho, K.; van Merrienboer, B.; Gulcehre, C.; Bahdanau, D.; Bougares, F.; Schwenk, H.; and Bengio, Y. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. In Proceedings of EMNLP. find
  11. Collobert, R.; Weston, J.; Bottou, L.; Karlen, M.; Kavukcuoglu, K.; and Kuksa, P. 2011. Natural Language Processing (almost) from Scratch. Journal of Machine Learning Research 12:2493–2537. find our card
  12. Creutz, M., and Lagus, K. 2007. Unsupervised Models for Morpheme Segmentation and Morphology Learning. In Proceedings of the ACM Transations on Speech and Language Processing. find
  13. Deerwester, S.; Dumais, S.; and Harshman, R. 1990. Indexing by Latent Semantic Analysis. Journal of American Society of Information Science 41:391–407. dos Santos, C. N., and Guimaraes, V. 2015. Boosting Named Entity Recognition with Neural Character Embeddings. In Proceedings of ACL Named Entities… find
  14. Graves, A. 2013. Generating Sequences with Recurrent Neural Networks. arXiv:1308.0850. find
  15. Hinton, G.; Srivastava, N.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. 2012. Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors. arxiv:1207.0580. find
  16. Hochreiter, S., and Schmidhuber, J. 1997. Long Short-Term Memory. Neural Computation 9:1735–1780. find
  17. Kalchbrenner, N.; Grefenstette, E.; and Blunsom, P. 2014. A Convolutional Neural Network for Modelling Sentences. In Proceedings of ACL. find our card
  18. Kim, Y. 2014. Convolutional Neural Networks for Sentence Classification. In Proceedings of EMNLP. find our card
  19. Krizhevsky, A.; Sutskever, I.; and Hinton, G. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Proceedings of NIPS. find
  20. LeCun, Y.; Boser, B.; Denker, J. S.; Henderson, D.; Howard, R. E.; Hubbard, W.; and Jackel, L. D. 1989. Handwritten Digit Recognition with a Backpropagation Network. In Proceedings of NIPS. find
  21. Lei, T.; Barzilay, R.; and Jaakola, T. 2015. Molding CNNs for Text: Non-linear, Non-consecutive Convolutions. In Proceedings of EMNLP. find
  22. Ling, W.; Lui, T.; Marujo, L.; Astudillo, R. F.; Amir, S.; Dyer, C.; Black, A. W.; and Trancoso, I. 2015. Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representation. In Proceedings of EMNLP. find our card
  23. Luong, M.-T.; Socher, R.; and Manning, C. 2013. Better Word Representations with Recursive Neural Networks for Morphology. In Proceedings of CoNLL. find our card
  24. Marcus, M.; Santorini, B.; and Marcinkiewicz, M. 1993. Building a Large Annotated Corpus of English: the Penn Treebank. Computational Linguistics 19:331–330. find our card
  25. Mikolov, T., and Zweig, G. 2012. Context Dependent Recurrent Neural Network Language Model. In Proceedings of SLT. find
  26. Mikolov, T.; Karafiat, M.; Burget, L.; Cernocky, J.; and Khudanpur, S. 2010. Recurrent Neural Network Based Language Model. In Proceedings of INTERSPEECH. find
  27. Mikolov, T.; Deoras, A.; Kombrink, S.; Burget, L.; and Cernocky, J. 2011. Empirical Evaluation and Combination of Advanced Language Modeling Techniques. In Proceedings of INTERSPEECH. find
  28. Mikolov, T.; Sutskever, I.; Deoras, A.; Le, H.-S.; Kombrink, S.; and Cernocky, J. 2012. Subword Language Modeling with Neural Networks. preprint: www.fit.vutbr.cz/ĩmikolov/rnnlm/char.pdf. find
  29. Mikolov, T.; Chen, K.; Corrado, G.; and Dean, J. 2013. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781. find
  30. Mnih, A., and Hinton, G. 2007. Three New Graphical Models for Statistical Language Modelling. In Proceedings of ICML. find
  31. Morin, F., and Bengio, Y. 2005. Hierarchical Probabilistic Neural Network Language Model. In Proceedings of AISTATS. find
  32. Pascanu, R.; Culcehre, C.; Cho, K.; and Bengio, Y. 2013. How to Construct Deep Neural Networks. arXiv:1312.6026. find
  33. Qui, S.; Cui, Q.; Bian, J.; and Gao, B. 2014. Co-learning of Word Representations and Morpheme Representations. In Proceedings of COLING. find
  34. Shen, Y.; He, X.; Gao, J.; Deng, L.; and Mesnil, G. 2014. A Latent Semantic Model with Convolutional-pooling Structure for Information Retrieval. In Proceedings of CIKM. find
  35. Srivastava, R. K.; Greff, K.; and Schmidhuber, J. 2015. Training Very Deep Networks. arXiv:1507.06228. — [arXiv:1507.06228](https://arxiv.org/abs/1507.06228) find
  36. Sundermeyer, M.; Schluter, R.; and Ney, H. 2012. LSTM Neural Networks for Language Modeling. find
  37. Sutskever, I.; Martens, J.; and Hinton, G. 2011. Generating Text with Recurrent Neural Networks. find
  38. Sutskever, I.; Vinyals, O.; and Le, Q. 2014. Sequence to Sequence Learning with Neural Networks. find
  39. Wang, M.; Lu, Z.; Li, H.; Jiang, W.; and Liu, Q. 2015. genCNN: A Convolutional Architecture for Word Sequence Prediction. In Proceedings of ACL. find
  40. Werbos, P. 1990. Back-propagation Through Time: what it does and how to do it. In Proceedings of IEEE. find
  41. Zaremba, W.; Sutskever, I.; and Vinyals, O. 2014. Recurrent Neural Network Regularization. arXiv:1409.2329. find our card
  42. Zhang, S.; Jiang, H.; Xu, M.; Hou, J.; and Dai, L. 2015. The Fixed-Size Ordinally-Forgetting Encoding Method for Neural Network Language Models. In Proceedings of ACL. find
  43. Zhang, X.; Zhao, J.; and LeCun, Y. 2015. Character-level Convolutional Networks for Text Classification. In Proceedings of NIPS. find our card