Elamur Research⌕
☰
home / papers / 2018 / The Importance of Being Recurrent for Modeling Hierarchical Structure

The Importance of Being Recurrent for Modeling Hierarchical Structure

Paper · 2018 · Ke Tran, Arianna Bisazza and Christof Monz

Summary

The paper asks whether non-recurrent sequence models can pick up hierarchical (syntactic or logical) structure without explicit structural supervision as well as recurrent ones do. The authors compare an LSTM with a Transformer, which they call a Fully Attentional Network (FAN). Both models are tuned through the same hyperparameter grid search and tested on two controlled tasks. The first is English subject-verb number agreement on the Linzen et al. (2016) dataset, trained with either a language-model objective or direct verb-number prediction. The second is classifying logical relations between sentence pairs in the artificial language of Bowman et al. (2015b). The LSTM wins in nearly every setting. Its advantage grows as the subject and verb move further apart, when more distracting nouns intervene, and when test sentences are longer than those seen in training. The authors argue that squeezing the history into a single state may push the model to learn structure. Their practical conclusion is that architectures chosen for parallel efficiency can lose structural generalization, which matters for language understanding systems.

Key points

  • Agreement setup: the agreement data are split 10% train, 1% validation and the rest test, with a 10k-word vocabulary. Rarer words are replaced by their part-of-speech tags.
  • Agreement results: the LSTM beats the FAN under both training objectives, and its lead widens with distance and with the number of agreement attractors. Validation perplexity is 67.06 for the LSTM and 69.14 for the FAN.
  • Attention analysis: a few attention heads in the FAN's upper layers frequently focus on the subject. How often they do so drops roughly linearly as subject-verb distance increases.
  • Logical inference task: the dataset has 60,000 generated samples (6 word types, 3 operators, 7 relations, and 1–12 operators per sentence), split 0.8/0.1/0.1. Bag-of-words baselines stay below 59% here but reach 77% on SNLI, showing that this task needs real structure.
  • Logical inference results: with all the training data, the two models perform about the same. When trained only on sentences with at most 6 operators, the LSTM scores higher and extrapolates better to longer inputs. Code and data are available at github.com/ketranm/fan_vs_rnn.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands151 works · 174,506 citations🎓Leiden UniversityLeiden · Netherlands37 works · 1,408 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Ke Tran, Arianna Bisazza and Christof Monz. 2018. The Importance of Being Recurrent for Modeling Hierarchical Structure. arXiv:1803.03585.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1803.03585 · arXiv 1803.03585

@misc{tran2018importance,
  title = {{The Importance of Being Recurrent for Modeling Hierarchical Structure}},
  author = {Ke Tran, Arianna Bisazza and Christof Monz},
  year = {2018},
  doi = {10.48550/arxiv.1803.03585},
  eprint = {1803.03585},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1803.03585}
}

References

18 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Terra Blevins, Omer Levy, and Luke Zettlemoyer. 2018. Deep rnns encode soft hierarchical syntax. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 14–19, Melbourne, Australia. Association for Computational Linguistics.
  2. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015a. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computat…
  3. Samuel R. Bowman, Christopher D. Manning, and Christopher Potts. 2015b. Tree-structured composition in neural networks without tree-structured architectures. In Proceedings of the 2015th International Conference on Cognitive Computation: Integrating Neural and Symbolic Approaches - Volume 1583, COCO… find
  4. Morten H. Christiansen and Nick Chater. 2016. The now-or-never bottleneck: A fundamental constraint on language. Behavioral and Brain Sciences, 39. find
  5. Hannah Cornish, Rick Dale, Simon Kirby, and Morten H Christiansen. 2017. Sequence memory constraints give rise to language-like structure through iterated learning. PloS one, 12(1):e0168532. find
  6. Richard Evans, David Saxton, David Amos, Pushmeet Kohli, and Edward Grefenstette. 2018. Can neural networks understand logical entailment? In ICLR. find
  7. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1243–1252, International Convention…
  8. Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In The 56th Annual Meeting of the Association for Computational Linguistics (ACL), Melbourne, Australia.
  9. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume…
  10. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno…
  11. Hakan Inan, Khashayar Khosravi, and Richard Socher. 2017. Tying word vectors and word classifiers: A loss framework for language modeling. ICLR.
  12. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021–2031, Copenhagen, Denmark. Association for Computational Linguistics.
  13. Diederik Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of ICLR, San Diego, CA, USA.
  14. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535.
  15. Ofir Press and Lior Wolf. 2017. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 157–163. Association for Computational Linguistics.
  16. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural mt learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526– 1534, Austin, Texas. Association for Computational Linguistics.
  17. Ke Tran, Arianna Bisazza, and Christof Monz. 2016. Recurrent memory networks for language modeling. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 321–331, San Diego, California. Association for… find
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tran-2018-importance-being-recurrent-modeling-hierarchical/