Elamur Research⌕
☰
home / papers / 2018 / Neural Lattice Language Models

Neural Lattice Language Models

Paper · 2018 · Jacob Buckman and Graham Neubig

Summary

Conventional LSTM language models require each sentence to have a single fixed tokenization, at either the character or the word level. As a result, they cannot explicitly represent linguistic facts such as multiword lexical units or words with several senses. Buckman and Neubig propose neural lattice language models. In this approach, every admissible split of a sentence into chunks becomes a path through a lattice, and the sentence probability is the sum over all paths. A recursive computation over each node's immediate predecessors keeps this tractable. Because a recurrent network depends on its entire history, the authors also develop approximations for the hidden state at nodes reached by several paths. Examples include summing predecessor states in the TreeLSTM style or sampling a single predecessor. The lattice can encode multi-token chunks or several embeddings per word. Experiments on the Billion Word Corpus (English) and Chinese GigaWord show clear perplexity gains over LSTM baselines. The work offers a general, end-to-end way to give neural language models linguistically motivated inductive biases.

Key points

  • English models with polysemous (multi-embedding) words reduce perplexity by 9.95% relative to a word-level LSTM baseline.
  • Compared with a baseline operating on individual characters, a Chinese model using multi-character tokens achieves 20.94% lower perplexity.
  • Marginalizing over the lattice costs O(|X|²) time, or O(D|X|) when each node has at most D incoming edges. A standard token-level model is the special case D = 1.
  • The base architecture is an LSTM with coupled input and forget gates (Greff et al., 2016). Hidden-state approximations include TreeLSTM-style summation and Monte-Carlo sampling of predecessors.
  • Qualitative analysis shows that the learned word embeddings separate usages mainly by context rather than by clean semantic senses. Some words, such as "rodham", always prefer one embedding, which suggests polysemy capacity could be allocated per word.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Jacob Buckman and Graham Neubig. 2018. Neural Lattice Language Models. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00036

@article{buckman2018neural,
  title = {{Neural Lattice Language Models}},
  author = {Jacob Buckman and Graham Neubig},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.1162/tacl\_a\_00036},
  url = {https://doi.org/10.1162/tacl_a_00036}
}

References

35 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Oliver Adams, Adam Makarucha, Graham Neubig, Steven Bird, and Trevor Cohn. 2017. Cross-lingual word embeddings for low-resource language modeling. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, volume 1, pages 93… find
  2. Ben Athiwaratkun and Andrew Wilson. 2017. Multimodal word distributions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1645–1656. find
  3. Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Philemon Brakel, and Yoshua Bengio. 2016. End-to-end attention-based large-vocabulary speech recognition. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 4945–4949. find
  4. IEEE. Colin Bannard and Danielle Matthews. 2008. Stored word sequences in language learning: The effect of familiarity on children’s repetition of four-word combinations. Psychological Science, 19(3):241–248. find
  5. William Chan, Yu Zhang, Quoc Le, and Navdeep Jaitly. 2017. Latent sequence decompositions. 5th International Conference on Learning Representations. find
  6. Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillipp Koehn, and Tony Robinson. 2014. One billion word benchmark for measuring progress in statistical language modeling.
  7. Interspeech. Xinchi Chen, Xipeng Qiu, Jingxiang Jiang, and Xuanjing Huang. 2015. Gaussian mixture embeddings for multiple word prototypes. CoRR, abs/1511.06246. — [arXiv:1511.06246](https://arxiv.org/abs/1511.06246) find
  8. Sumit Chopra, Michael Auli, Alexander M Rush, and SEAS Harvard. 2016. Abstractive sentence summarization with attentive recurrent neural networks. North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 93–98.
  9. Pierre Dupont and Ronald Rosenfeld. 1997. Lattice based language models. Technical report, DTIC Document. find
  10. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A Smith. 2016. Recurrent neural network grammars. North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199–209. find
  11. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. find
  12. Yarin Gal and Zoubin Ghahramani. 2016. A theoretically grounded application of dropout in recurrent neural networks. In Advances in Neural Information Processing Systems, pages 1019–1027.
  13. Sharon Goldwater, Thomas L. Griffiths, Mark Johnson, et al. 2007. Distributional cues to word boundaries: Context is important. In H. Caunt-Nulton, S. Kilati-late, and I. Woo, editors, BUCLD 31: Proceedings of the 31st Annual Boston University Conference on Language Development, pages 239–250. find
  14. Somerville, Massachusetts: Cascadilla Press. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 6645– 6649. IEEE. Klaus Greff, Rupesh K. Srivastava, J… find
  15. Kyuyeon Hwang and Wonyong Sung. 2017. Character-level language modeling with hierarchical recurrent neural networks. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages 5720– 5724. find
  16. IEEE. Hakan Inan, Khashayar Khosravi, and Richard Socher. 2017. Tying word vectors and word classifiers: A loss framework for language modeling. 5th International Conference on Learning Representations. find
  17. Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with Gumbel-Softmax. 5th International Conference on Learning Representations.
  18. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv:1602.02410. find
  19. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation, pages 28–39. find
  20. Jan Koutnik, Klaus Greff, Faustino Gomez, and Juergen Schmidhuber. 2014. A clockwork RNN. Proceedings of Machine Learning Research. find
  21. Wang Ling, Isabel Trancoso, Chris Dyer, and Alan W. Black. 2015. Character-based neural machine translation. CoRR, abs/1511.04586. Wang Ling, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, Andrew Senior, Fumin Wang, and Phil Blunsom. 2016. Latent predictor networks for code generation.…
  22. Chris J Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The concrete distribution: A continuous relaxation of discrete random variables. 5th International Conference on Learning Representations.
  23. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. 4th International Conference on Learning Representations.
  24. Arvind Neelakantan, Jeevan Shankar, Re Passos, and Andrew Mccallum. 2014. Efficient nonparametric estimation of multiple embeddings per word in vector space. In Proceedings of EMNLP. find
  25. Citeseer. Graham Neubig, Masato Mimura, Shinsuke Mori, and Tatsuya Kawahara. 2010. Learning a language model from continuous speech. In INTERSPEECH, pages 1053–1056. find
  26. Graham Neubig, Chris Dyer, Yoav Goldberg, Austin Matthews, Waleed Ammar, Antonios Anastasopoulos, Miguel Ballesteros, David Chiang, Daniel Clothiaux, Trevor Cohn, et al. 2017. DyNet: The dynamic neural network toolkit. arXiv preprint arXiv:1701.03980.
  27. Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. 5th International Conference on Learning Representations.
  28. Yael Ravin and Claudia Leacock. 2000. find
  29. Polysemy: Theoretical and Computational Approaches. OUP Oxford. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. Association for Computational Linguistics.
  30. Anna Siyanova-Chanturia, Kathy Conklin, and Norbert Schmitt. 2011. Adding more fuel to the fire: An eye-tracking study of idiom processing by native and non-native speakers. Second Language Research, 27(2):251–272. find
  31. Matthias Sperber, Graham Neubig, Jan Niehues, and Alex Waibel. 2017. Neural lattice-to-sequence models for uncertain inputs. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1380–1389. find
  32. Jinsong Su, Zhixing Tan, Deyi Xiong, and Yang Liu. 2016. Lattice-based recurrent neural network encoders for neural machine translation. CoRR, abs/1609.07730, ver. 2. — [arXiv:1609.07730](https://arxiv.org/abs/1609.07730) find
  33. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. Association for Computational Linguistics. Lukáš Burget Jan Honza Cernock Sanjeev Khudanpur Tomáš Mikolov, Martin Karafiát. 2010. Recurrent ne… find
  34. Ladislav Zgusta. 1967. Multiword lexical units. Word, 23(1-3):578–587. find
  35. Hao Zhou, Zhaopeng Tu, Shujian Huang, Xiaohua Liu, Hang Li, and Jiajun Chen. 2017. Chunk-based bi-scale decoder for neural machine translation. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/buckman-2018-neural-lattice-language-models/