Elamur Research⌕
☰
home / papers / 2015 / When Are Tree Structures Necessary for Deep Learning of Representations?

When Are Tree Structures Necessary for Deep Learning of Representations?

Paper · 2015 · Jiwei Li, Thang Luong, Dan Jurafsky et al.

Summary

Recursive (tree-structured) neural networks build sentence representations by composing words along a syntactic parse, but it was unclear which tasks actually benefit from this structure compared with simpler sequential models. Li, Luong, Jurafsky and Hovy run a controlled comparison of tree-based models against unidirectional and bidirectional recurrent models, in both plain and LSTM variants. They test five setups spanning four task areas: sentiment on the Pang et al. (2002) data and the Stanford Sentiment Treebank, question–phrase matching on UMD-QA, relation classification on SemEval-2010, and discourse relation identification on the RST data. All models share one training regime: a single composition layer, AdaGrad with minibatches, and identical initialization. Trees give a clear advantage mainly when distant headwords must be linked, as in relation extraction, and on long inputs. A recurrent model that splits sentences at punctuation into clause-like pieces and then combines them largely closes that gap. The study clarifies when parsing is worth its cost and points to concrete ways to improve sequence models.

Key points

  • Models compared: standard tree vs. sequence vs. bi-sequence models, plus Tree LSTM vs. LSTM vs. bidirectional LSTM. Significance is checked with a bootstrap test at the 0.05 level.
  • On the Stanford Sentiment Treebank, recurrent models are trained on 215,154 phrase examples. Evaluation covers 82,600 phrase-level and 2,210 root-level instances. Trees help slightly at the root level but hardly at the phrase level, and bidirectional recurrent models mostly erase the difference.
  • On the Pang et al. dataset, where labels appear only at the top of long sequences, tree and sequence models show no significant difference.
  • On RST relation identification, accuracies are close:
    • Standard models: tree 0.568, sequence 0.572, bi-sequence 0.578.
    • LSTM models: tree 0.564, sequence 0.563, bi-LSTM 0.575, the only significant gain (p = 0.040).
  • On UMD-QA, intermediate recurrent outputs carry evidence as useful as parse-tree nodes, even though they are not linguistically meaningful units.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States310 works · 284,480 citations🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Jiwei Li, Thang Luong, Dan Jurafsky and Eduard Hovy. 2015. When Are Tree Structures Necessary for Deep Learning of Representations?. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d15-1278

@inproceedings{li2015when,
  title = {{When Are Tree Structures Necessary for Deep Learning of Representations?}},
  author = {Jiwei Li, Thang Luong, Dan Jurafsky and Eduard Hovy},
  booktitle = {Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing},
  year = {2015},
  doi = {10.18653/v1/d15-1278},
  url = {https://doi.org/10.18653/v1/d15-1278}
}

References

42 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv:1409.0473. arXiv preprint Samuel R Bowman, Christopher Potts, and Christopher D Manning. 2014. Recursive neural networks for learning logical semantics. arXiv prepri… our card
  2. Samuel R Bowman, Christopher D Manning, and Christopher Potts. 2015. Tree-structured composition in neural networks without tree-structured architectures. arXiv preprint arXiv:1506.04834. find
  3. Samuel R Bowman. 2013. Can recursive neural tensor networks learn logical reasoning? arXiv preprint arXiv:1312.6192. — [arXiv:1312.6192](https://arxiv.org/abs/1312.6192) find
  4. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. 2003. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Current and New Directions in Discourse and Dialogue Text, Speech and Language Technology. volume 22. Springer. our card
  5. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537. our card
  6. Andrew Cotter, Ohad Shamir, Nati Srebro, and Karthik Sridharan. 2011. Better mini-batch algorithms via accelerated gradient methods. In Advances in Neural Information Processing Systems, pages 1647– 1655. find
  7. Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014. Adaptive recursive neural network for target-dependent twitter sentiment classification. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 49–54. our card
  8. Georg Dorffner. 1996. Neural networks for time series processing. In Neural Network World. find
  9. John Duchi, Elad Hazan, and Yoram Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 12:2121–2159. find
  10. Bradley Efron and Robert J Tibshirani. 1994. An introduction to the bootstrap. CRC press. find
  11. Jeffrey L Elman. 1990. Finding structure in time. Cognitive science, 14(2):179–211. find
  12. Christoph Goller and Andreas Kuchler. 1996. Learning task-dependent distributed representations by backpropagation through structure. In Neural Networks, 1996., IEEE International Conference on, volume 1, pages 347–352. IEEE. find
  13. Alex Graves and Juergen Schmidhuber. 2009. Offline handwriting recognition with multidimensional recurrent neural networks. In Advances in Neural Information Processing Systems, pages 545–552. find
  14. Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013. Speech recognition with deep recurrent neural networks. In Acoustics, Speech and Signal Processing (ICASSP), 2013 IEEE International Conference on, pages 6645–6649. IEEE. find
  15. Alex Graves. 2012. Supervised sequence labeling with recurrent neural networks, In Studies in Computational Intelligence. volume 385. Springer. find
  16. Kazuma Hashimoto, Makoto Miwa, Yoshimasa Tsuruoka, and Takashi Chikayama. 2013. Simple customization of recursive neural networks for semantic relation classification. In EMNLP, pages 1372– 1376. find
  17. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2009. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the Workshop on Seman… our card
  18. Hugo Hernault, Helmut Prendinger, Mitsuru Ishizuka. 2010. Hilda: a discourse parser using support vector machine classification. Dialogue & Discourse, 1(3). our card
  19. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  20. Ozan Irsoy and Claire Cardie. 2013. Bidirectional recursive neural networks for token-level labeling with structure. arXiv preprint arXiv:1312.0493. — [arXiv:1312.0493](https://arxiv.org/abs/1312.0493) find
  21. Ozan Irsoy and Claire Cardie. 2014. Deep recursive neural networks for compositionality in language. In Advances in Neural Information Processing Systems, pages 2096–2104. find
  22. Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daumé III. 2014. A neural network for factoid question answering over paragraphs. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 633–644. find
  23. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP, pages 1700–1709. our card
  24. Quoc V Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. arXiv preprint arXiv:1405.4053. our card
  25. Jiwei Li and Eduard Hovy. 2014. A model of coherence based on distributed sentence representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) find
  26. Jiwei Li, Rumeng Li, and Eduard Hovy. 2014. Recursive deep models for discourse parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2061–2069. our card
  27. Richard P Lippmann. 1989. Review of neural networks for speech recognition. Neural computation, 1(1):1–38. find
  28. Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2014. Addressing the rare word problem in neural machine translation. Proceedings of ACL. 2015. our card
  29. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In HLT-NAACL, pages 746– 751. our card
  30. Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up?: sentiment classification using machine learning techniques. In Proceedings of the ACL-02 conference on Empirical methods in natural language processing-Volume 10, pages 79–86. Association for Computational Linguistics. our card
  31. Romain Paulus, Richard Socher, and Christopher D Manning. 2014. Global belief recursive neural networks. In Advances in Neural Information Processing Systems, pages 2888–2896. find
  32. Barak A Pearlmutter. 1989. Learning state space trajectories in recurrent neural networks. Neural Computation, 1(2):263–269. find
  33. Tony Robinson, Mike Hochberg, and Steve Renals. 1996. The use of recurrent neural networks in continuous speech recognition. In Automatic speech and speaker recognition, pages 233–258. Springer. find
  34. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45(11):2673–2681. find
  35. Richard Socher, Eric H Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y Ng. 2011a. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in Neural Information Processing Systems, pages 801–809. find
  36. Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011b. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 151–161. Association for Computa… our card
  37. Richard Socher, Brody Huval, Christopher D Manning, and Andrew Y Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 1201–1211. A… our card
  38. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages… find
  39. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. find
  40. Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. ACL. 2015. find
  41. Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2014. Grammar as a foreign language. arXiv preprint arXiv:1412.7449. find
  42. Xiaodan Zhu, Parinaz Sobihani, and Hongyu Guo. 2015. Long short-term memory over recursive structures. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 1604–1612. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/li-2015-when-are-tree-structures-necessary/