Elamur Research⌕
☰
home / papers / 2016 / A Latent Variable Recurrent Neural Network for Discourse-Driven Language Models

A Latent Variable Recurrent Neural Network for Discourse-Driven Language Models

Paper · 2016 · Yangfeng Ji, Gholamreza Haffari and Jacob Eisenstein

Summary

Neural language models learn strong distributed representations. Unlike probabilistic graphical models, however, they cannot easily treat linguistic annotations as variables to infer or sum out. Ji, Haffari and Eisenstein address this by coupling an LSTM-based recurrent language model with a discrete latent variable. This variable stands for the shallow discourse relation linking each pair of adjacent sentences or utterances. The relation is predicted from a summary vector of the previous sentence. It then selects relation-specific output matrices and biases that shape word generation in the next sentence. Given the text, the relations are discrete and independent of each other, so exact inference needs only a small sum. When trained conditionally, the model works as a classifier. It is reported to beat earlier state-of-the-art systems at identifying implicit relations on the Penn Discourse Treebank, as well as at tagging dialog acts on Switchboard. When trained jointly, it sums out relations at test time and achieves lower perplexity than RNNLM and DCLM baselines. The work offers a simple way to add interpretable structure to neural text models.

Key points

  • The model, called the Discourse Relation Language Model (DRLM), extends the Document Context Language Model (DCLM). It makes the intra-sentence and inter-sentence output weights depend on the latent relation. To limit parameters, each weight matrix is factored into a large shared V×H part and a small H×H relation-specific part.
  • There are two training objectives. Conditional likelihood of relations given text is used for classification. Joint likelihood of text and relations is used for language modeling, where test documents need no discourse annotations.
  • Perplexity on PDTB: RNNLM 117.8, DCLM 112.2, DRLM 108.3. Perplexity on SWDA (Switchboard): 56.0, 45.3 and 39.6, respectively.
  • Prior neural PDTB classifiers learned only from the roughly 16,000 annotated training relations. DRLM also benefits from the word-prediction signal over the whole text.
  • Limitation: training requires discourse annotations, so the approach does not yet scale to large language-modeling corpora. Learning from partially labeled data is proposed as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgia Institute of TechnologyAtlanta · United States133 works · 24,497 citations🎓Monash UniversityMelbourne · Australia60 works · 54,866 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksDiscourse analysis · 466 works

Cite

Cite as Yangfeng Ji, Gholamreza Haffari and Jacob Eisenstein. 2016. A Latent Variable Recurrent Neural Network for Discourse-Driven Language Models. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1037

@inproceedings{ji2016latent,
  title = {{A Latent Variable Recurrent Neural Network for Discourse-Driven Language Models}},
  author = {Yangfeng Ji, Gholamreza Haffari and Jacob Eisenstein},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1037},
  url = {https://doi.org/10.18653/v1/n16-1037}
}

References

42 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio. 2012. Practical recommendations for gradient-based training of deep architectures. In Neural Networks: Tricks of the Trade, pages 437–478. find
  2. Springer. Or Biran and Kathleen McKeown. 2013. In Proceedings of the Association for Computational Linguistics (ACL), pages 69–73, Sophia, Bulgaria. Chloé Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Proceedings of Empirical Methods…
  3. Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. 2016. Importance weighted autoencoders. In Proceedings of the International Conference on Learning Representations (ICLR). find
  4. Rich Caruana. 1997. Multitask learning. Machine learning, 28(1):41–75. find
  5. Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron Courville, and Yoshua Bengio. 2015. A recurrent latent variable model for sequential data. In Neural Information Processing Systems (NIPS), Montréal. find
  6. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537. João FG de Freitas, Mahesan Niranjan, Andrew H. Gee, and Arnaud Doucet. Sequential monte carlo methods to train n…
  7. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159. find
  8. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the Association for Computational Linguistics (ACL), pages 334–343, Beijing, China.
  9. Jenny Rose Finkel, Christopher D Manning, and Andrew Y Ng. 2006. Solving the problem of cascading errors: Approximate bayesian inference for linguistic annotation pipelines. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 618–626. find
  10. John J Godfrey, Edward C Holliman, and Jane McDaniel. 1992. Switchboard: Telephone speech corpus for research and development. In ICASSP, volume 1, pages 517–520. find
  11. IEEE. Shixiang Gu, Zoubin Ghahramani, and Richard E Turner. 2015. Neural adaptive sequential monte carlo. In Neural Information Processing Systems (NIPS), Montréal. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735– 1780. find
  12. Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In Proceedings of the Association for Computational Linguistics (ACL), Baltimore, MD.
  13. Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributional semantics for discourse relations. Transactions of the Association for Computational Linguistics (TACL), June. find
  14. Yangfeng Ji, Trevor Cohn, Lingpeng Kong, Chris Dyer, and Jacob Eisenstein. 2015. Document context language models. In International Conference on Learning Representations, Poster Paper, volume abs/1511.03962. — [arXiv:1511.03962](https://arxiv.org/abs/1511.03962) find
  15. Shafiq Joty, Giuseppe Carenini, and Raymond Ng. 2012. A novel discriminative framework for sentence-level discourse analysis. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP). find
  16. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent convolutional neural networks for discourse compositionality. In Proceedings of the Workshop on Continuous Vector Space Models and their Compositionality, pages 119–126, Sofia, Bulgaria, August. Association for Computational Linguistics.
  17. Diederik P Kingma and Max Welling. 2014. Autoencoding variational bayes. In Proceedings of the International Conference on Learning Representations (ICLR). find
  18. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of the International Conference on Machine Learning (ICML).
  19. Jiwei Li, Rumeng Li, and Eduard Hovy. 2014. Recursive deep models for discourse parsing. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP).
  20. Ziheng Lin, Min-Yen Kan, and Hwee Tou Ng. 2009. Recognizing implicit discourse relations in the penn discourse treebank. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 343–351, Singapore.
  21. Rui Lin, Shujie Liu, Muyun Yang, Mu Li, Ming Zhou, and Sheng Li. 2015. Hierarchical recurrent neural network for document modeling. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 899–907, Lisbon, September. find
  22. Christopher D. Manning. 2016. Computational linguistics and deep learning. Computational Linguistics, 41(4). find
  23. Tomas Mikolov and Geoffrey Zweig. 2012. Context dependent recurrent neural network language model. In Proceedings of Spoken Language Technology (SLT), pages 234–239. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language mod… find
  24. Dmitrijs Milajevs and Matthew Purver. 2014. Investigating the contribution of distributional semantic information for dialogue act classification. In Proceedings of the 2nd Workshop on Continuous Vector Space Models and their Compositionality (CVSC), pages 40– 47. Joonsuk Park and Claire Cardie. Imp…
  25. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063. Vu Pham, Théodore Bluche, Christopher Kermorvant, and Jérôme Louradour. 2014. Dropout improves recurrent neural networks for handwriting recognition. In… find
  26. IEEE. Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind Joshi. 2008. Easily identifiable discourse relations. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 87–90, Manchester, UK.
  27. Emily Pitler, Annie Louis, and Ani Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the Association for Computational Linguistics (ACL), Suntec, Singapore.
  28. Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse Treebank 2.0. In Proceedings of LREC.
  29. Attapol T Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proceedings of the European Chapter of the Association for Computational Linguistics (EACL).
  30. Attapol Rutherford and Nianwen Xue. 2015. Improving the inference of implicit discourse relations via classifying explicit discourse connectives. pages 799–808, May–June.
  31. Kenji Sagae. 2009. Analysis of discourse structure with syntactic dependencies and data-driven shift-reduce parsing. In Proceedings of the 11th International Conference on Parsing Technologies (IWPT’09), pages 81–84, Paris, France, October. Association for Computational Linguistics. find
  32. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank.
  33. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), Seattle, WA. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Meg Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of convers…
  34. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  35. Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics, 26(3):339–373.
  36. Joseph Turian, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. pages 384–394.
  37. Tian Wang and Kyunghyun Cho. 2015. Larger-context language modelling. arXiv preprint arXiv:1511.03729. find
  38. Nick Webb, Mark Hepple, and Yorick Wilks. 2005. Dialogue act classification based on intra-utterance features. In Proceedings of the AAAI Workshop on Spoken Language Understanding. find
  39. Bonnie Webber, Markus Egg, and Valia Kordoni. 2012. Discourse structure and language technology. Journal of Natural Language Engineering, 1. find
  40. Jason D Williams and Steve Young. 2007. Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language, 21(2):393– 422. find
  41. Nianwen Xue, Hwee Tou Ng, Sameer Pradhan, Rashmi Prasad, Christopher Bryant, and Attapol T Rutherford. 2015. The CoNLL-2015 shared task on shallow discourse parsing. In Proceedings of the Conference on Natural Language Learning (CoNLL).
  42. Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of Empirical Methods for Natural Language Processing (EMNLP), pages 2230– 2235, Lisbon, September.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ji-2016-latent-variable-recurrent-neural-network/