Elamur Research⌕
☰
home / papers / 2018 / Do latent tree learning models identify meaningful structure in sentences?

Do latent tree learning models identify meaningful structure in sentences?

Paper · 2018 · Adina Williams, Andrew Drozdov and Samuel R. Bowman

Summary

Latent tree learning trains TreeRNN-style sentence encoders that build their own binary parses. They learn only from a downstream objective and never see gold treebank trees. Earlier studies reported that such systems beat encoders fed with output from conventional parsers, but no one had closely examined the structures they discover. Williams, Drozdov and Bowman reimplemented two of these systems, RL-SPINN (Yogatama et al., 2017) and ST-Gumbel (Choi et al., 2018), alongside baselines in a single codebase. They trained all models for natural language inference on SNLI and MultiNLI. They then examined the induced trees for stability across random restarts, overlap with Penn Treebank (PTB) parses, and recognizable linguistic patterns. Only ST-Gumbel outperforms both a TreeRNN baseline using parser trees and a plain LSTM. Yet its grammars vary widely between runs, match PTB roughly at chance level, are shallower than PTB trees, and fit no known syntactic or semantic formalism. The work shows that strong task accuracy does not imply that a model has learned meaningful constituency. This leaves open what these encoders actually rely on.

Key points

  • Models compared:
    • RL-SPINN builds on the shift-reduce SPINN architecture and trains its parsing decisions with REINFORCE.
    • ST-Gumbel uses a chart-like structure and the Straight-Through Gumbel-Softmax estimator to commit to a single hard tree.
  • Data and evaluation: All models were trained on SNLI and MultiNLI. The induced trees were compared with PTB-style parses, including Stanford Parser output, and checked for self-consistency across restarts.
  • RL-SPINN: It settles on a trivial, mostly left-branching strategy that is stable across runs but linguistically uninformative.
  • ST-Gumbel's regularities:
    • It merges word pairs at sentence edges more often than expected.
    • It fairly consistently attaches negation to the constituent that follows it.
    • It handles determiners, prepositions and modifiers inconsistently, so its PP and ADJP constituents rarely coincide with PTB.
  • Conclusion: The authors argue that progress requires better composition functions in TreeRNNs and better methods for searching over possible grammars. Published in TACL vol. 6 (2018), pp. 253–267.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New York UniversityNew York · United States136 works · 86,390 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Adina Williams, Andrew Drozdov and Samuel R. Bowman. 2018. Do latent tree learning models identify meaningful structure in sentences?. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00019

@article{williams2018do,
  title = {{Do latent tree learning models identify meaningful structure in sentences?}},
  author = {Adina Williams, Andrew Drozdov and Samuel R. Bowman},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2018},
  doi = {10.1162/tacl\_a\_00019},
  url = {https://doi.org/10.1162/tacl_a_00019}
}

References

45 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Adger. 2003. Core Syntax: A Minimalist Approach. Oxford University Press. find
  2. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, pages 861–872. our card
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 632–642. find
  4. Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, pa… find
  5. Stanley F. Chen. 1995. Bayesian grammar induction for language modeling. In Proceedings of the 33rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 228–235. find
  6. Jihun Choi, Kang Min Yoo, and Sang-goo Lee. 2018. Learning to compose task-specific tree structures. In Proceedings of the Thirty-Second Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence (AAAI-18), volume 2. find
  7. Noam Chomsky. 1965. Aspects of the Theory of Syntax. MIT press. find
  8. Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. 2017. Hierarchical multiscale recurrent neural networks. In Proceedings of the International Conference on Learning Representations (ICLR). find
  9. Shay B. Cohen, Dipanjan Das, and Noah A. Smith. 2011. Unsupervised structure prediction with non-parallel multilingual guidance. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 50–61. find
  10. Sreerupa Das, C. Lee Giles, and Guo-Zheng Sun. 1992. Learning context-free grammars: Capabilities and limitations of a recurrent neural network with an external stack memory. In Proceedings of The Fourteenth Annual Conference of Cognitive Science Society. Indiana University, page 14. find
  11. John DeNero and Jakob Uszkoreit. 2011. Inducing sentence structure from parallel corpora for reordering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 193–203. find
  12. Richard O. Duda, Peter E. Hart, and David G. Stork. 1973. Pattern Classification. Wiley, New York. find
  13. Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-sequence attentional neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, pages 823–833. find
  14. Gottlob Frege. 1892. Über Sinn und Bedeutung. Wittgenstein Studien 1(1). find
  15. King Sun Fu. 1977. Introduction to syntactic pattern recognition. In Syntactic Pattern Recognition, Applications, pages 1–31. find
  16. Matthew R. Gormley, Margaret Mitchell, Benjamin Van Durme, and Mark Dredze. 2014. Low-resource semantic role labeling. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, pages 1177–1187. find
  17. Edward Grefenstette, Karl Moritz Hermann, Mustafa Suleyman, and Phil Blunsom. 2015. Learning to transduce with unbounded memory. In Advances in Neural Information Processing Systems (NIPS), pages 1828–1836. find
  18. Irene Heim and Angelika Kratzer. 1998. Semantics in generative grammar. Blackwell. find
  19. Daniel J. Hsu, Sham M. Kakade, and Percy S. Liang. 2012. Identifiability and unmixing of latent parse trees. In Advances in Neural Information Processing Systems (NIPS), pages 1511–1519. find
  20. Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with Gumbel-Softmax. In Proceedings of the International Conference on Learning Representations (ICLR). our card
  21. Armand Joulin and Tomas Mikolov. 2015. Inferring algorithmic patterns with stack-augmented recurrent nets. In Advances in Neural Information Processing Systems (NIPS), pages 190–198. find
  22. Yoon Kim, Carl Denton, Luong Hoang, and Alexander M. Rush. 2017. Structured attention networks. In Proceedings of the International Conference on Learning Representations (ICLR). find
  23. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR). find
  24. Dan Klein and Christopher D. Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics (ACL), pages 423– 430. our card
  25. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics (TACL) 4:521–535. our card
  26. Yang Liu and Mirella Lapata. 2018. Learning structured text representations. Transactions of the Association for Computational Linguistics (TACL) 6:63–75. our card
  27. Tsendsuren Munkhdalai and Hong Yu. 2017b. Neural tree indexers for text understanding. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (ACL), Volume 1: Long Papers, pages 11–21. find
  28. Jason Naradowsky, Sebastian Riedel, and David A Smith. 2012. Improving NLP through marginalization of hidden syntactic structure. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP), pages 810–820. find
  29. Tahira Naseem and Regina Barzilay. 2011. Using semantic cues to learn syntax. In Twenty-Fifth Association for the Advancement of Artificial Intelligence Conference on Artificial Intelligence (AAAI-11), pages 902–907. find
  30. Graham Neubig, Taro Watanabe, and Shinsuke Mori. 2012. Inducing a discriminative parser to optimize machine translation reordering. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 843–853. find
  31. Jean Maillard, Stephen Clark, and Dani Yogatama. 2017. Jointly learning sentence embeddings and syntax with unsupervised Tree-LSTMs. arXiv preprint 1705.09189. — [arXiv:1705.09189](https://arxiv.org/abs/1705.09189) find
  32. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global Vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532– 1543. our card
  33. Mitchell P. Marcus, Beatrice Santorini, Mary Ann Marcinkiewicz, and Ann Taylor. 1999. Treebank-3. LDC99T42. Linguistic Data Consortium. find
  34. Ivan A. Sag. 1991. Linguistic theory and natural language processing. In Natural Language and Speech, pages 69–83. find
  35. Lili Mou, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. Natural language inference by tree-based convolution and heuristic matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL), Volume 2: Short Papers, pages 130–136. our card
  36. Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 151–161. our card
  37. Tsendsuren Munkhdalai and Hong Yu. 2017a. Neural semantic encoders. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL): Volume 1, Long Papers, pages 397–407. find
  38. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP), pag… find
  39. Dominique Sportiche, Hilda Koopman, and Edward Stabler. 2013. An Introduction to Syntactic Analysis and Theory. John Wiley & Sons. find
  40. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research (JMLR) 15(1):1929– 1958. find
  41. G. Z. Sun, C. L. Giles, H. H. Chen, and Y. C. Lee. 1993. The neural network pushdown automaton: Model, stack and learning simulations. Technical Report UMIACS-TR-93-77/CS-TR-3118, University of Maryland. find
  42. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural La… find
  43. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). find
  44. Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8(3-4):229–256. find
  45. Dani Yogatama, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Wang Ling. 2017. Learning to compose words into sentences with reinforcement learning. In Proceedings of the International Conference on Learning Representations (ICLR). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/williams-2018-do-latent-tree-learning-models/