Elamur Research⌕
☰
home / papers / 2020 / Cross-Linguistic Syntactic Evaluation of Word Prediction Models

Cross-Linguistic Syntactic Evaluation of Word Prediction Models

Paper · 2020 · Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou et al.

Summary

Most evidence that neural word predictors capture syntax comes from English. This paper asks how that ability varies across languages. The authors build CLAMS (Cross-Linguistic Assessment of Models on Syntax), which carries the subject-verb agreement portion of Marvin and Linzen's (2018) benchmark over to French, German, Hebrew and Russian alongside English. The items are minimal pairs produced by hand-written grammars in a new lightweight formalism, attribute-varying grammars (AVGs). AVGs use non-recursive templates and a "vary" statement that alters chosen attributes to create the ungrammatical versions. A model counts as correct when it prefers the grammatical verb form. The authors test monolingual and multilingual LSTM LMs, trained with equal amounts of same-domain data per language, as well as English BERT and multilingual BERT. LSTMs handle simple agreement well but struggle with object relative clauses. Multilingual models show interference between languages rather than helpful transfer. The work matters because it supplies a controlled, extensible benchmark for comparing syntactic competence across languages.

Key points

  • CLAMS covers five languages (English, French, German, Hebrew, Russian). The AVG grammar framework is released at github.com/aaronmueller/clams.
  • Monolingual LSTMs score highly on agreement without attractors but poorly across object relative clauses (center-embedding).
  • The multilingual LSTM performed worse than the monolingual ones. mBERT was strong on English, mixed on French and German, and weak on Hebrew and Russian. Its English results also fell somewhat below monolingual English BERT.
  • For LSTMs, richer morphology tended to go with higher accuracy, with German as an outlier. The Spearman correlation was 0.4 with German included and 1.0 with it excluded.
  • mBERT showed the reverse pattern (ρ = −0.9), which the authors suspect reflects its limited subword vocabulary, especially for non-Latin scripts and Hebrew's non-concatenative morphology.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States187 works · 92,194 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 3Research subjects and languages the work is about.Evaluation & benchmarks · 685 worksSemitic languages · 1,484 worksHebrew & Aramaic · 481 works

Cite

Cite as Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina and Tal Linzen. 2020. Cross-Linguistic Syntactic Evaluation of Word Prediction Models. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.490 · arXiv 2005.00187

@inproceedings{mueller2020cross,
  title = {{Cross-Linguistic Syntactic Evaluation of Word Prediction Models}},
  author = {Aaron Mueller, Garrett Nicolai, Panayiota Petrou-Zeniou, Natalia Talmina and Tal Linzen},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.490},
  eprint = {2005.00187},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.490}
}

References

46 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aixiu An, Peng Qian, Ethan Wilcox, and Roger Levy. 2019. Representation of constituents in neural language models: Coordination phrase as a case study. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lang… find
  2. Christian Bentz, Tatyana Ruzsics, Alexander Koplenig, and Tanja Samardžić. 2016. A comparison between morphological complexity measures: Typological data vs. language corpora. In Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC), pages 142–153, Osaka, Japan… find
  3. The COLING 2016 Organizing Committee. find
  4. Noam Chomsky. 1956. Three models for the description of language. IRE Transactions on Information Theory, 2(3):113–124. find
  5. Yoav Goldberg. 2019. Assessing BERT’s syntactic abilities. arXiv preprint 1901.05287.
  6. Kristina Gulordava, Piotr Bojanowski, Edouard Grave, Tal Linzen, and Marco Baroni. 2018. Colorless green recurrent networks dream hierarchically. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume…
  7. Noam Chomsky. 1957. Syntactic Structures. Mouton, The Hague. find
  8. Yiding Hao. 2020. Attribution analysis of grammatical dependencies in lstms. arXiv preprint 2005.00062. — [arXiv:2005.00062](https://arxiv.org/abs/2005.00062) find
  9. Shammur Absar Chowdhury and Roberto Zamparelli. 2018. RNN simulations of grammaticality judgments on long-distance dependencies. In Proceedings of the 27th International Conference on Computational Linguistics, pages 133–144. Association for Computational Linguistics. find
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  12. Prajit Dhar and Arianna Bisazza. 2018. Does syntactic knowledge in multilingual language models transfer across languages? In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 374–377, Brussels, Belgium. Association for Computational Lingui… find
  13. Prajit Dhar and Arianna Bisazza. 2020. Understanding cross-lingual syntactic transfer in multilingual recurrent neural networks. arXiv preprint 2003.14056. find
  14. Matthew S. Dryer and Martin Haspelmath, editors. 2013. WALS Online. Max Planck Institute for Evolutionary Anthropology, Leipzig. find
  15. Richard Futrell, Ethan Wilcox, Takashi Morita, Peng Qian, Miguel Ballesteros, and Roger Levy. 2019. Neural language models as psycholinguistic subjects: Representations of syntactic state. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Lingui…
  16. Anastasia Giannakidou. 2011. Negative and positive polarity items: Variation, licensing, and compositionality. In Semantics: An international handbook of natural language meaning, pages 1660– 1712. Berlin: Mouton de Gruyter. find
  17. Pierre Isabelle, Colin Cherry, and George Foster. 2017. A challenge set approach to evaluating machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2486–2496, Copenhagen, Denmark. Association for Computational Linguistics.
  18. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. arXiv preprint 1602.02410. find
  19. Jaap Jumelet and Dieuwke Hupkes. 2018. Do language models understand anything? On the ability of LSTMs to understand negative polarity items. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 222–231, Brussels, Belgium. Association for C… find
  20. Katharina Kann, Alex Warstadt, Adina Williams, and Samuel R. Bowman. 2019. Verb argument structure alternations in word and sentence embeddings. In Proceedings of the Society for Computation in Linguistics (SCiL) 2019, pages 287–297. find
  21. Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape…
  22. Adhiguna Kuncoro, Chris Dyer, Laura Rimell, Stephen Clark, and Phil Blunsom. 2019. Scalable syntax-aware language models using knowledge distillation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3472–3484, Florence, Italy. Association for Computa… find
  23. Jey Han Lau, Alexander Clark, and Shalom Lappin. 2017. Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge. Cognitive Science, (5):1202–1247. find
  24. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics), 4:521– 535.
  25. Heidi Lorimor, Kathryn Bock, Ekaterina Zalkind, Alina Sheyman, and Robert Beard. 2008. Agreement and attraction in Russian. Language and Cognitive Processes, 23(6):769–799. find
  26. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202, Brussels, Belgium. Association for Computational Linguistics. find
  27. R. Thomas McCoy, Robert Frank, and Tal Linzen. 2018. Revisiting the poverty of the stimulus: Hierarchical generalization without a hierarchical bias in recurrent neural networks. In Proceedings of the 40th Annual Conference of the Cognitive Science Society, pages 2093—2098, Austin, TX. find
  28. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of the 11th Annual Conference of the International Speech Communication Association (IN-TERSPEECH 2010), pages 1045–1048, Makuhari, Chiba, Japan. find
  29. Hiroshi Noji and Hiroya Takamura. 2020. An analysis of the utility of explicit negative examples to improve the syntactic abilities of neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Seattle, Washington, USA. Association for Computa… find
  30. Robert Östling and Jörg Tiedemann. 2017. Continuous multilinguality with language vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 644–649, Valencia, Spain. Association for Computational Linguisti… find
  31. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  32. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint 1910.10683.
  33. Shauli Ravfogel, Yoav Goldberg, and Tal Linzen. 2019. Studying the inductive biases of RNNs with synthetic variations of natural languages. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 find
  34. (Long and Short Papers), pages 3532–3542, Minneapolis, Minnesota. Association for Computational Linguistics. find
  35. Shauli Ravfogel, Yoav Goldberg, and Francis Tyers. 2018. Can LSTM learn to capture agreement? the case of Basque. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 98–107, Brussels, Belgium. Association for Computational Linguistics. find
  36. Marten van Schijndel, Aaron Mueller, and Tal Linzen. 2019. Quantity doesn’t buy quality syntax with neural language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL… find
  37. Carson T Schütze. 1996. The empirical base of linguistics: Grammaticality judgments and linguistic methodology. University of Chicago Press. find
  38. Rico Sennrich. 2017. How grammatical is character-level neural machine translation? Assessing MT quality with contrastive translation pairs. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 376–382, Valencia…
  39. Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. 2012. LSTM neural networks for language modeling. In Proceedings of the 13th Annual Conference of the International Speech Communication Association (INTERSPEECH), pages 194–197. find
  40. Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4731–4736, Brussels, Belgium. Association for Computational Linguistics. find
  41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  42. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2019a. BLiMP: A benchmark of linguistic minimal pairs for English. arXiv preprint 1912.00582. find
  43. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019b. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. find
  44. Paul Wexler. 1990. The schizoid nature of modern Hebrew: A Slavic language in search of a Semitic past. Wiesbaden: Otto Harrassowitz Verlag. c. Der Polizist Ethan Wilcox, Roger Levy, Takashi Morita, and Richard Futrell. 2018. What do RNN language models learn about filler–gap dependencies? In Procee…
  45. Amir Zeldes. 2013. Is Modern Hebrew Standard Average European? The view from European. Linguistic Typology, 17(3):439–470. find
  46. Ghil’ad Zuckermann. 2006. A new vision for Israel Hebrew: Theoretical and practical implications of analyzing Israel’s main language as a semi-engineered Semito-European hybrid language. Journal of Modern Jewish Studies, 5(1):57–71. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mueller-2020-cross-linguistic-syntactic-evaluation-word/