Elamur Research⌕
☰
home / papers / 2024 / Targeted Syntactic Evaluation on the Chomsky Hierarchy

Targeted Syntactic Evaluation on the Chomsky Hierarchy

Summary

The authors propose a new approach to Targeted Syntactic Evaluation. Rather than testing individual grammatical phenomena, they examine how language models handle constructions of varying complexity according to the Chomsky hierarchy. To this end, four artificial formal languages are constructed, each modeling a syntactic phenomenon of natural language. The first imitates the arbitrary repetition of adjectives before a noun. The second imitates nested clauses without agreement, as in Japanese, and the third, nested dependencies with agreement, as in English. The fourth reproduces cross-serial dependencies characteristic of Swiss German. Each nonterminal symbol can be realized by several terminals, so each language has its own vocabulary. Five variants with different vocabulary sizes were created for each language. LSTM, Transformer LM and Stack-RNN were trained as language models. They were then tested on minimal pairs: the model had to assign a higher probability to the grammatical string than to the ungrammatical one. All architectures learned the simplest, regular language, while none of them coped with either type of dependency. The authors conclude that the models do not acquire the true dependency structure but merely approximate it with rules at the level of regular grammars.

Key points

  • Four formal languages: (Adj)ⁿ NP (regular, recognized by a finite-state automaton), NPⁿ VPⁿ, Nested Dependency and Cross Serial Dependency. Each has five variants with different vocabulary sizes.
  • Three architectures are compared: LSTM, Transformer LM and Stack-RNN. The evaluation compares each grammatical sentence with an ungrammatical counterpart, and success means a higher probability for the correct sentence.
  • All models mastered (Adj)ⁿ NP. LSTM and Transformer LM failed on NPⁿ VPⁿ. No architecture correctly recognized Nested Dependency or Cross Serial Dependency.
  • This approach differs from works such as Delétang et al. (2023) in two respects. First, the models are trained specifically as language models. Second, it accounts for a diversity of terminals closer to the vocabulary of natural language.
  • The code is open: github.com/osekilab/TSE-Chomsky.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics Evaluation & benchmarks

Cite

Cite as Taiga Someya, Ryo Yoshida and Yohei Oseki. 2024. Targeted Syntactic Evaluation on the Chomsky Hierarchy. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{someya2024targeted,
  title = {{Targeted Syntactic Evaluation on the Chomsky Hierarchy}},
  author = {Taiga Someya, Ryo Yoshida and Yohei Oseki},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  url = {https://aclanthology.org/2024.lrec-main.1356/}
}

References

46 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alfred V Aho and Jeffrey D Ullman. 1972. The Theory of Parsing, Translation and Compiling. Parsing, vol. I. Prentice-Hall, Englewood Cliffs. find
  2. Aixiu An, Peng Qian, Ethan Wilcox, and Roger Levy. 2019. Representation of constituents in neural language models: Coordination phrase as a case study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lang… find
  3. Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. 2020. On the Ability and Limitations of Transformers to Recognize Formal Languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7096–7116, Online. Association for Computational Linguistics… find
  4. Mikael Boden and Janet Wiles. 2002. On learning context free and context sensitive languages. IEEE Transactions on Neural Networks, 13:491– 493. find
  5. Mikael Bodón and Janet Wiles. 2000. Context-free and context-sensitive dynamics in recurrent neural networks. Connection Science, 12(3-4):197– 210. find
  6. Rui P Chaves. 2020. What don’t RNN language models learn about Filler-Gap dependencies? Proceedings of the Society for Computation in Linguistics, 3(1):20–30. find
  7. Noam Chomsky. 1957. Syntactic structures. Mouton. find
  8. Shammur Absar Chowdhury and Roberto Zamparelli. 2018. RNN simulations of grammaticality judgments on long-distance dependencies. In Proceedings of the 27th International Conference on Computational Linguistics, pages 133– 144, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  9. Jillian Da Costa and Rui Chaves. 2020. Assessing the ability of transformer-based neural models to represent structurally unbounded dependencies. In Proceedings of the Society for Computation in Linguistics 2020, pages 12–21, New York, New York. Association for Computational Linguistics. find
  10. Grégoire Delétang, Anian Ruoss, Jordi Grau-Moya, Tim Genewein, Li Kevin Wenliang, Elliot Catt, Chris Cundy, Marcus Hutter, Shane Legg, Joel Veness, and Pedro A Ortega. 2023. Neural networks and the chomsky hierarchy. In The Eleventh International Conference on Learning Representations. our card
  11. Javid Ebrahimi, Dhruv Gelda, and Wei Zhang. 2020. How can self-attention networks recognize dyck-n languages? CoRR, abs/2010.04303. find
  12. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingui… our card
  13. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume… our card
  14. Michael Hahn. 2020. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156–171. find
  15. S Hochreiter and J Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735–1780. find
  16. Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the Association of Computational Linguistics. find
  17. Armand Joulin and Tomas Mikolov. 2015. Inferring algorithmic patterns with stack-augmented recurrent nets. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS’15, page 190â��198, Cambridge, MA, USA. MIT Press. find
  18. Katharina Kann, Alex Warstadt, Adina Williams, and Samuel R Bowman. 2019. Verb argument structure alternations in word and sentence embeddings. In Proceedings of the Society for Computation in Linguistics (SCiL) 2019, pages 287–297. find
  19. D. Terence Langendoen. 1981. The generative capacity of word-formation components. Linguistic Inquiry, 12(2):320–322. find
  20. Sabine Lehmann, Stephan Oepen, Sylvie Regnier-Prost, Klaus Netter, Veronika Lux, Judith Klein, Kirsten Falkedal, Frederik Fouvry, Dominique Estival, Eva Dauphin, Herve Compagnion, Judith Baur, Lorna Balkan, and Doug Arnold. 1996. TSNLP - test suites for natural language processing. In COLING 1996 Vo… find
  21. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn Syntax-Sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521–535. our card
  22. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. find
  23. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. find
  24. Rowan Hall Maudslay and Ryan Cotterell. 2021. Do syntactic probes probe syntax? experiments with jabberwocky probing. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 124–131, Online. Association… find
  25. Vladislav Mikhailov, Ekaterina Taktasheva, Elina Sigdel, and Ekaterina Artemova. 2021. RuSen-tEval: Linguistic source, encoder force! In Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing, pages 43–65, Kiyv, Ukraine. Association for Computational Linguistics. find
  26. Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina, and Tal Linzen. 2020. Cross-linguistic syntactic evaluation of word prediction models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Stroudsburg, PA, USA. Association for Computati… find
  27. Shauli Ravfogel, Yoav Goldberg, and Francis Tyers. 2018. Can LSTM learn to capture agreement? the case of basque. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 98–107, Brussels, Belgium. Association for Computational Linguistics. find
  28. Paul Rodriguez and Janet Wiles. 1997. Recurrent neural networks can learn to implement symbol-sensitive counting. In Advances in Neural Information Processing Systems, volume 10. MIT Press. find
  29. Stuart M. Shieber. 1985. Evidence against the context-freeness of natural language. Linguistics and Philosophy, 8(3):333–343. find
  30. Natalia Skachkova, Thomas Trost, and Dietrich Klakow. 2018. Closing brackets with recurrent neural networks. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 232–239, Brussels, Belgium. Association for Computational Linguistics. find
  31. Taiga Someya and Yohei Oseki. 2023. JBLiMP: Japanese benchmark of linguistic minimal pairs. find
  32. In Findings of the Association for Computational Linguistics: EACL 2023, pages 1581– 1594, Dubrovnik, Croatia. Association for Computational Linguistics. find
  33. Mirac Suzgun, Yonatan Belinkov, Stuart Shieber, and Sebastian Gehrmann. 2019. LSTM networks can perform dynamic counting. In Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges, pages 44–54, Florence. Association for Computational Linguistics. find
  34. Daniela Trotta, Raffaele Guarasci, Elisa Leonardelli, and Sara Tonelli. 2021. Monolingual and cross-lingual acceptability judgments with the Italian CoLA corpus. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2929–2940, Punta Cana, Dominican Republic. Association for… find
  35. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł Ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, pages 5998–6008. Curran Associates, Inc. our card
  36. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. our card
  37. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. Superglue: A stickier benchmark for general-purpose language understanding systems. CoRR, abs/1905.00537. our card
  38. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355,… our card
  39. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019c. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations. our card
  40. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for english. Transactions of the Association for Computational Linguistics, 8:377–392. find
  41. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. find
  42. Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite precision RNNs for language recognition. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 740–745, Melbourne, Australia. Associati… our card
  43. Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about Filler–Gap dependencies? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 211–221, Brussels, Belgium. Association for Computati… our card
  44. Ethan Wilcox, Roger P. Levy, and Richard Futrell. 2019. Hierarchical representation in neural language models: Suppression and recovery of expectations. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 181– 190, Florence, Italy. Associati… find
  45. Janet Wiles and Jeffrey Elman. 1995. Learning to count without a counter: A case study of dynamics and activation landscapes in recurrent networks. find
  46. Beilei Xiang, Changbing Yang, Yu Li, Alex Warstadt, and Katharina Kann. 2021. CLiMP: A benchmark for Chinese language model evaluation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2784–2790, Online. Association fo… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/someya-2024-targeted-syntactic-evaluation-chomsky-hierarchy/