Elamur Research⌕
☰
home / papers / 2022 / Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale

Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale

Paper · 2022 · Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro et al.

Summary

Large Transformer LMs perform strongly without any explicit model of hierarchical syntax. It was unclear whether recursive syntactic composition, which helped small LSTM-based RNNGs, still adds value for Transformers at larger scale. The authors introduce Transformer Grammars (TGs), built on Transformer-XL, which generate a sentence jointly with its phrase-structure tree as a top-down action sequence. Composition happens entirely through the attention mask. Each closing nonterminal is duplicated: one copy (COMPOSE) condenses the finished constituent's children into one vector, and the other (STACK) places that vector on a stack. Later tokens see only the summary, not the words inside, and tree-aware relative positions are added. Trained on PTB and BLLIP-LG, TGs beat RNNGs and a words-only Transformer-XL on sentence-level perplexity, syntactic generalization, and parse reranking. They also train faster than batched RNNGs. However, an unmasked Transformer-XL over linearized trees reaches slightly lower perplexity, and TGs fall behind at document level. The results suggest that syntactic bias aids generalization, while squeezing context into composed vectors harms long-text memory.

Key points

  • Training data: Penn Treebank (PTB, ~1M words) and BLLIP-LG (~40M words). Baselines are TXL (terminals), TXL (trees), and RNNGs, among them the batched RNNG implementation introduced by Noji and Oseki (2021).
  • Mechanism: closing nonterminals are split into CNT1 (COMPOSE) and CNT2 (STACK) tokens. Both share the same attention heads and parameters, and the mask is computed causally by a stack-based algorithm.
  • On the syntactic generalization benchmark of Hu et al. (2020), TGs outperform both TXL baselines. They also beat GPT-2-small, Gopher, and Chinchilla, which are roughly 250×–1,000× larger and trained on far more data.
  • TXL (trees) slightly edges out TGs on sentence perplexity. A regression analysis and probing tasks attribute this to the attention restriction weakening lexical copying, showing a partial dissociation between perplexity and syntactic competence.
  • In document-level modeling, representing each earlier sentence as a single composed vector makes TGs clearly worse than both TXL variants. This points to a memory mechanism that is partly independent of syntax.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom104 works · 157,183 citations🏢Google DeepMindLondon · United Kingdom56 works · 78,880 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Miloš Stanojević, Phil Blunsom and Chris Dyer. 2022. Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00526

@article{sartran2022transformer,
  title = {{Transformer Grammars: Augmenting Transformer Language Models with Syntactic Inductive Biases at Scale}},
  author = {Laurent Sartran, Samuel Barrett, Adhiguna Kuncoro, Miloš Stanojević, Phil Blunsom and Chris Dyer},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.1162/tacl\_a\_00526},
  url = {https://doi.org/10.1162/tacl_a_00526}
}

References

55 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ramón Fernandez Astudillo, Miguel Ballesteros, Tahira Naseem, Austin Blodgett, and Radu Florian. 2020. Transition-based parsing with stack-transformers. In Findings of EMNLP. find
  2. Jiangang Bai, Yujing Wang, Yiren Chen, Yaming Yang, Jing Bai, Jing Yu, and Yunhai Tong. 2021. Syntax-BERT: Improving pre-trained transformers with syntax trees. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3011–302… find
  3. Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Manning, and Christopher Potts. 2016. A fast unified model for parsing and sentence understanding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1… find
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris… — Language Models are Few-Shot Learners
  5. Eugene Charniak, Don Blaheta, Niyu Ge, Keith Hall, John Hale, and Mark Johnson. 2000. BLLIP 1987–89 WSJ Corpus Release 1, LDC2000T43. LDC2000T43. Linguistic Data Consortium, 36. find
  6. Ciprian Chelba and Frederick Jelinek. 2000. Structured language modeling. Computer Speech & Language, 14(4):283–332. https://doi.org/10.1006/csla.2000.0147 — [doi:10.1006/csla.2000.0147](https://doi.org/10.1006/csla.2000.0147) ↗
  7. Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. 2017. Hierarchical multiscale recurrent neural networks. In Proceedings of ICLR. find
  8. Alexis Conneau, German Kruszewski, Guillaume Lample, Loı̈c Barrault, and Marco Baroni. 2018. What you can cram into a single $&!\#* vector: Probing sentence embeddings for linguistic properties. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long…
  9. Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of ACL. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL.
  11. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 199–209, San Diego, California. Assoc… find
  12. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics. https://doi.org/10.1162/tacl_a_00298 ↗
  13. Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2331–2336, Austin, Texas. Association for Computational Linguistics. find
  14. Martin B. H. Everaert, Marinus A. C. Huybregts, Noam Chomsky, Robert C. Berwick, and Johan J. Bolhuis. 2015. Structures, not strings: Linguistics as part of the cognitive sciences. Trends in Cognitive Sciences, 19(12):729–743. https://doi.org/10.1016/j.tics.2015 .09.008, PubMed: 26564247 — [doi:10.1016/j.tics.2015](https://doi.org/10.1016/j.tics.2015) ↗
  15. Noam Chomsky. 1957. Syntactic Structures. Mouton, The Hague/Paris. https://doi.org/10.1515/9783112316009 — [doi:10.1515/9783112316009](https://doi.org/10.1515/9783112316009) ↗
  16. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingui…
  17. Michael Hahn. 2020. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156–171. https://doi.org/10.1162/tacl_a_00306 ↗
  18. Adi Haviv, Ori Ram, Ofir Press, Peter Izsak, and Omer Levy. 2022. Transformer language models without positional encodings still learn positional information. (arXiv:2203.16634). find
  19. James Henderson. 2004. Discriminative training of a neural network statistical parser. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pages 95–102, Barcelona, Spain. https://doi.org/10.3115/1218955 .1218968 ↗
  20. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  21. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational… find
  22. Daniel Jurafsky, Chuck Wooters, Jonathan Segal, Andreas Stolcke, Eric Fosler, Gary N. Tajchman, and Nelson Morgan. 1995. Using a stochastic context-free grammar as a language model for speech recognition. In Proceedings of ICASSP. find
  23. Yoon Kim, Alexander Rush, Lei Yu, Adhiguna Kuncoro, Chris Dyer, and Gábor Melis. 2019. find
  24. Unsupervised recurrent neural network grammars. In Proceedings of NAACL. find
  25. Taku Kudo. 2018. Subword regularization: Improving neural network translation models with multiple subword candidates. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 66–75, Melbourne, Australia. Association for Computational…
  26. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium. Associ…
  27. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What do recurrent neural network grammars learn about syntax? In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers… find
  28. Adhiguna Kuncoro, Chris Dyer, Laura Rimell, Stephen Clark, and Phil Blunsom. 2019. Scalable syntax-aware language models using knowledge distillation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3472–3484, Florence, Italy. Association for Computa… find
  29. Adhiguna Kuncoro, Lingpeng Kong, Daniel Fried, Dani Yogatama, Laura Rimell, Chris Dyer, and Phil Blunsom. 2020. Syntactic structure distillation pretraining for bidirectional encoders. Transactions of the Association for Computational Linguistics, 8:776–794. https://doi.org/10.1162/tacl_a_00345 ↗
  30. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech…
  31. Christopher D. Manning, Kevin Clark, John Hewitt, Urvashi Khandelwal, and Omer Levy. 2020. Emergent linguistic structure in artificial neural networks trained by self-supervision. Proceedings of the National Academy of Sciences, 117(48). find
  32. Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19(2):313–330. https://doi.org/10.21236/ADA273556 ↗
  33. Piotr Mirowski and Andreas Vlachos. 2015. Dependency Recurrent Neural Language Models for Sentence Completion. In Proceeding of ACL-IJCNLP. https://doi.org/10 .3115/v1/P15-2084 find
  34. Xuan-Phi Nguyen, Shafiq R. Joty, Steven C. H. Hoi, and Richard Socher. 2020. Tree-structured attention with hierarchical accumulation. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26–30, 2020. OpenReview.net. find
  35. Hiroshi Noji and Yohei Oseki. 2021. Effective batching for recurrent neural network grammars. In Findings of the ACL-IJCNLP. find
  36. Hao Peng, Roy Schwartz, and Noah A. Smith. 2019. PaLM: A hybrid parser and language model. In Proceedings of EMNLP-IJCNLP. https://doi.org/10.18653/v1/D19 -1376 ↗
  37. Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL.
  38. Slav Petrov and Dan Klein. 2007. Improved inference for unlexicalized parsing. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference, pages 404–411, Rochester, New York. Association for C…
  39. Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of ACL. https://doi.org/10 .18653/v1/…
  40. Peng Qian, Tahira Naseem, Roger Levy, and Ramón Fernandez Astudillo. 2021. Structural guidance for transformer language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volu… find
  41. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  42. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, H. Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-S… find
  43. Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Liwei Peng, and Luo Si. 2020. Struct-BERT: Incorporating language structures into pre-training for deep language understanding. In Proceedings of ICLR. find
  44. Brian Roark. 2001. Probabilistic Top-Down Parsing and Language Modeling. Computational Linguistics, 27(2):249–276. https://doi.org/10.1162/089120101750300526 ↗
  45. Yaushian Wang, Hung-Yi Lee, and Yun-Nung Chen. 2019. Tree transformer: Integrating tree structures into self-attention. In Proceedings of EMNLP-IJCNLP. https://doi.org/10 .18653/v1/D19-1098 find
  46. Devendra Sachan, Yuhao Zhang, Peng Qi, and William L. Hamilton. 2021. Do syntax trees help pre-trained transformers extract information? In Proceedings of EACL. https://doi.org/10.18653/v1/2021.eacl-main .228 ↗
  47. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. TACL. https://doi.org/10 .1162/tacl_x_00375
  48. Satoshi Sekine and Michael John Collins. 1997. Evalb - bracket scoring program. find
  49. Ethan Wilcox, Peng Qian, Richard Futrell, Miguel Ballesteros, and Roger Levy. 2019. structural supervision improves learning of non-local grammatical dependencies. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tec… find
  50. Yikang Shen, Shawn Tan, Alessandro Sordoni, and Aaron Courville. 2019. Ordered neurons: Integrating tree structures into recurrent neural networks. In International Conference on Learning Representations. find
  51. Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5027–5038, Brussels, Belgium. Association for Comput…
  52. Dhanasekar Sundararaman, Vivek Subramanian, Guoyin Wang, Shijing Si, Dinghan Shen, Dong Wang, and Lawrence Carin. 2019. Syntax-infused transformer and BERT models for machine translation and natural language understanding. arXiv preprint arXiv:1911 .06156v1. find
  53. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc. find
  54. Elena Voita and Ivan Titov. 2020. Information-theoretic probing with minimum description Dani Yogatama, Yishu Miao, Gabor Melis, Wang Ling, Adhiguna Kuncoro, Chris Dyer, and Phil Blunsom. 2018. Memory architectures in recurrent neural network language models. In Proceedings of ICLR. find
  55. Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, Hai Zhao, and Rui Wang. 2020. SG-Net: Syntax-guided machine reading comprehension. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 202… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sartran-2022-transformer-grammars-augmenting-transformer-language/