Elamur Research⌕
☰
home / papers / 2026 / Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies

Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies

Summary

Large language models handle many single-sentence phenomena well, but it is less clear whether they grasp paradigmatic regularities that only emerge across several related sentences. Verb alternations are one such case. Samo and Merlo address this gap with multilingual datasets in the Blackbird Language Matrices (BLM) format. A BLM is a language analogue of Raven-style or ARC puzzles: a model sees an incomplete sequence of sentences and must choose the right continuation from a set of minimally contrasting candidates.

The datasets cover:

  • change-of-state and object-drop verbs in English, German and Italian
  • binyanim in Hebrew

The authors define three template families of increasing complexity. They populate them through three linguistically informed augmentation routes: masked-model generation, LLM prompting, and sentences drawn from natural corpora. Each template has a low and a high lexical-variation version.

The result is several thousand BLM problems, accompanied by simple baselines. The baselines show the data can expose what models get wrong, but they are not meant as a full evaluation. The resource gives a controlled, cross-lingual way to test systematic knowledge of argument structure.

Key points

  • Template types:
    • Type A uses cues outside the alternation itself, such as a temporal PP like "two years ago".
    • Type B uses linguistic diagnostics, such as subject relative clauses.
    • Type C mixes both and also varies inflection, such as tense.
  • Morphological contrasts across languages:
    • English marks the alternation with nothing overt.
    • Italian uses the reflexive-like si.
    • German uses a nominative/accusative case shift.
    • Hebrew relies on root-and-template morphology.
  • Lexical variation: each matrix comes in a MinLex version, where lexical material is shared across sentences, and a MaxLex version, where sentences share structure but not words.
  • Answer sets: wrong candidates contain either sequence errors (wrong constituent order) or grammar errors (constituents of the wrong syntactic or semantic kind).
  • Error analysis: the appendix gives raw error counts by error type for COS and OD in English and Italian, using test sets of 300 items per condition.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4t48qjruy2ce

@inproceedings{samo2026datasets,
  title = {{Datasets for Verb Alternations across Languages: BLM Templates and Data Augmentation Strategies}},
  author = {Samo, Giuseppe and Merlo, Paola},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4t48qjruy2ce},
  url = {https://doi.org/10.63317/4t48qjruy2ce}
}

References

60 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Judit Acs, Endre Hamerlik, Roy Schwartz, Noah A. Smith, and Andras Kornai. 2024. Morphosyntactic probing of multilingual BERT models. Natural Language Engineering, 30(4):753–792. find
  2. Furkan Akkurt, Onur Gungor, Büşra Marşan, Tunga Gungor, Balkiz Ozturk Basaran, Arzucan Özgür, and Susan Uskudarli. 2024. Evaluating the Quality of a Corpus Annotation Scheme Using Pre-trained Language Models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Lan… find
  3. Aixiu An, Chunyang Jiang, Maria A. Rodriguez, Vivi Nastase, and Paola Merlo. 2023. BLM-AgrF: A new French benchmark to investigate generalization of agreement in neural networks. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 136… find
  4. Maya Arad. 2005. Roots and Patterns: Hebrew Morpho-Syntax. Springer, New York. find
  5. Khuyagbaatar Batsuren, Omer Goldman, Salam Khalifa, Nizar Habash, Witold Kieraś, Gábor Bella, Brian Leonard, Garrett Nicolai, Kyle Gorman, Yustinus Ghanggo Ate, Maria Ryskina, Sabrina Mielke, Elena Budianskaya, Charbel El-Khaissi, Tiago Pimentel, Michael Gasser, William Abbott Lane, Mohit Raj, Matt… find
  6. Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics, 48(1):207–219. find our card
  7. Jonathan David Bobaljik. 2015. Suppletion: Some theoretical implications. Annual Review of Linguistics, 1(1):1–18. find
  8. François Chollet. 2019. On the Measure of Intelligence. CoRR, abs/1911.01547. find
  9. Mario Civico. 2025. Linguistic statistical Universals: comparing Computer- and Human-generated texts. International Journal of Digital Humanities, 7:1–37. find
  10. Edna Amir Coffin and Shmuel Bolozky. 2005. A Reference Grammar of Modern Hebrew. Cambridge University Press. find
  11. Marie-Catherine De Marneffe, Christopher D Manning, Joakim Nivre, and Daniel Zeman. 2021. Universal dependencies. Computational linguistics, 47(2):255–308. find our card
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. find
  13. Nomi Erteschik-Shir, Lena Ibnbari, and Sharon Taube. 2013. Missing objects as topic drop. Lingua, 136:145–169. find
  14. Prakhar Gautam, Jitendra Singh Thakur, and Ashish Mishra. 2024. Subject–verb agreement error handling using rnn architectures. In International Conference on Innovations in Computational Intelligence and Computer Vision, pages 215–224. Springer. find
  15. Francesca Grasso, Vladimiro Lovera Rulfi, and Luigi Di Caro. 2024. Verbalignet: Unlocking multilingual exploration of verbal arguments. In Selected Challenges in Grammar-Based Text Generation from the Semantic Web, pages 3–17. Springer Nature Switzerland. find
  16. Thomas L Griffiths, Jian-Qiao Zhu, Erin Grant, and R Thomas McCoy. 2024. Bayes in the age of intelligent machines. Current Directions in Psychological Science, 33(5):283–291. find
  17. Bruno Guillaume. 2021. Graph matching and graph rewriting: GREW tools for corpus exploration, maintenance and conversion. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 168–175, Online. Association for Comp… find
  18. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint, arXiv:2501.12948. find our card
  19. Martin Haspelmath, Andreea Calude, Michael Spagnol, Heiko Narrog, and Elif Bamyaci. 2014. Coding causal–noncausal verb alternations: A form–frequency correspondence explanation. Journal of linguistics, 50(3):587–625. find
  20. Elisabetta Jezek. 2024. Semantics first! Rethinking the role of syntax in verb classification. Lingue e linguaggio, 23(1):9–30. find
  21. Chunyang Jiang and Paola Merlo. 2026. Analogical structure, minimal contextual cues and contrastive distractors: Input design for sample-efficient linguistic rule induction. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (EACL 2026). find
  22. Chunyang Jiang, Giuseppe Samo, Vivi Nastase, and Paola Merlo. 2024. BLM-It - Blackbird Language Matrices for Italian: A CALAMITA Challenge. In Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024), pages 1135–1143, Pisa, Italy. CEUR Workshop Proceedings. find
  23. Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, and Arianna Bisazza. 2025. MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs. arXiv preprint arXiv:2504.02768. find
  24. Katharina Kann, Alex Warstadt, Adina Williams, and Samuel R. Bowman. 2019. Verb argument structure alternations in word and sentence embeddings. In Proceedings of the Society for Computation in Linguistics (SCiL) 2019, pages 287– 297. find
  25. Itamar Kastner. 2019. Templatic morphology as an emergent property: Roots and functional heads in Hebrew. Natural Language & Linguistic Theory, 37:571–619. find
  26. Stav Klein and Reut Tsarfaty. 2020. Getting the \#\#life out of living: How adequate are wordpieces for modelling complex morphology? In Proceedings of the 17th SIGMORPHON Workshop on Computational Research in Phonetics, Phonology, and Morphology, pages 204–209, Online. Association for Computational L… find
  27. Beth Levin. 1993. English Verb Classes and Alternations A Preliminary Investigation. University of Chicago Press, Chicago and London. find
  28. Tal Linzen and Marco Baroni. 2021. Syntactic structure from deep learning. Annual Review of Linguistics, 7(1):195–212. find
  29. John J. McCarthy. 1979. Formal Problems in Semitic Phonology and Morphology. Ph.D. thesis, Massachusetts Institute of Technology. find
  30. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal Dependency Annotation for Multilingual Parsing. In Proceedings of the 51… find
  31. Maria Meo, Maxwell J. Roberts, and Francesco S. Marucci. 2007. Element salience as a predictor of item difficulty for raven’s progressive matrices. Intelligence, 35(4):359–368. find
  32. Paola Merlo. 2023a. Blackbird language matrices (BLM), a new task for rule-like generalization in neural networks: Can Large Language Models pass the test? In Findings of the 2023 Conference on Empirical Methods in Natural Language Processing. find
  33. Paola Merlo. 2023b. Blackbird language matrices (blm), a new task for rule-like generalization in neural networks: Motivations and formal specifications. arXiv, https://arxiv.org/abs/2306.11444. — [arXiv:2306.11444](https://arxiv.org/abs/2306.11444) ↗
  34. Paola Merlo and Suzanne Stevenson. 2001. Automatic verb classification based on statistical distributions of argument structure. Computational Linguistics, 27(3):373–408. find
  35. Roser Morante and Eduardo Blanco. 2021. Recent advances in processing negation. Natural Language Engineering, 27(2):121–130. find
  36. Mihai Nadas, Laura Diosan, and Andreea Tomescu. 2025. Synthetic Data Generation Using Large Language Models: Advances in Text and Code. arXiv preprint arXiv:2503.14023. — [arXiv:2503.14023](https://arxiv.org/abs/2503.14023) find
  37. Vivi Nastase and Paola Merlo. 2024. Tracking linguistic information in transformer-based sentence embeddings through targeted sparsification. In Proceedings of the 9th Workshop on Representation Learning for NLP (RepL4NLP-2024), pages 203–214, Bangkok, Thailand. Association for Computational Linguis… find
  38. Vivi Nastase, Giuseppe Samo, Chunyang Jiang, and Paola Merlo. 2024a. Exploring Italian Sentence Embeddings Properties through Multi-tasking. In Proceedings of the Tenth Italian Conference on Computational Linguistics (Clic-It 2024), pages 1–10. find
  39. Vivi Nastase, Giuseppe Samo, Chunyang Jiang, and Paola Merlo. 2024b. Exploring syntactic information in sentence embeddings through multilingual subject-verb agreement. In Proceedings of the Tenth Italian Conference on Computational Linguistics (Clic-It 2024), pages 1–13. find
  40. Garrett Nicolai, Eleanor Chodroff, Frederic Mailhot, and Çağrı Çöltekin, editors. 2024. Proceedings of the 21st SIGMORPHON workshop on Computational Research in Phonetics, Phonology, and Morphology. Association for Computational Linguistics, Mexico City, Mexico. find
  41. Mattia Proietti, Gianluca Lebani, and Alessandro Lenci. 2022. Does BERT recognize an agent? modeling Dowty’s proto-roles with contextual embeddings. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4101–4112, Gyeongju, Republic of Korea. International Committee… find
  42. Malka Rappaport Hovav and Beth Levin. 2024. Variable agentivity: Polysemy or underspecification. Glossa: a journal of general linguistics, 9(1). find
  43. John C. Raven. 1938. Standardization of progressive matrices. British Journal of Medical Psychology, 19:137–150. find
  44. Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. 2021. Probing the probing paradigm: Does probing accuracy entail task relevance? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3363– 3377, Online. Associatio… find
  45. Tanja Samardžić and Paola Merlo. 2018. The probability of external causation: An empirical account of crosslinguistic variation in lexical causatives. Linguistics, 56(5):895–938. find
  46. Giuseppe Samo and Paola Merlo. 2026. Modelling the Morphology of Verbal Paradigms: A Case Study in the Tokenization of Turkish and Hebrew. arXiv, https://arxiv.org/abs/2602.05648 . ↗ our card
  47. Giuseppe Samo, Vivi Nastase, Chunyang Jiang, and Paola Merlo. 2023. BLM-s/lE: A structured dataset of English spray-load verb alternations for testing generalization in LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12276–12287, Singapore. Association for Compu… find
  48. Tijs Schepens, Marten van der Klis, and Willem Zuidema. 2025. Large language models as corpus generators: Opportunities and risks. arXiv preprint arXiv:2503.11245. — [arXiv:2503.11245](https://arxiv.org/abs/2503.11245) find
  49. Luca Simonetti, Elisabetta Ježek, and Guido Vetere. 2024. Subcategorization of Italian Verbs with LLMs and T-PAS. In Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024). CEUR Workshop Proceedings. find
  50. Reut Tsarfaty. 2004. ‘binyanim ba’avir’: An investigation of Aspect Semantics in Modern Hebrew. Master’s thesis, Universiteit van Amsterdam, Institute for Logic Language and Computation. find
  51. Reut Tsarfaty. 2013. A unified morpho-syntactic scheme of Stanford dependencies. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 578–584, Sofia, Bulgaria. Association for Computational Linguistics. find
  52. Anne Vainikka and Yonata Levy. 1999. Empty subjects in Finnish and Hebrew. Natural Language & Linguistic Theory, 17(3):613–671. find
  53. Simonetta Vietri. 2019. I verbi ad alternanza causativa. classificazione lessico-grammaticale. find
  54. Alex Warstadt, Yu Cao, Ioana Grosu, Wei Peng, Hagen Blix, Yining Nie, Anna Alsop, Shikha Bordia, Haokun Liu, Alicia Parrish, Sheng-Fu Wang, Jason Phang, Anhad Mohananey, Phu Mon Htut, Paloma Jeretic, and Samuel R. Bowman. 2019. Investigating BERT’s knowledge of language: Five analysis methods with N… find
  55. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The Benchmark of Linguistic Minimal Pairs for English. In Transactions of the Association for Computational Linguistics, volume 8, pages 391–406. find
  56. Michael Wilson, Jackson Petty, and Robert Frank. 2023. How abstract is linguistic generalization in large language models? experiments with argument structure. Transactions of the Association for Computational Linguistics, 11:1377–1395. find
  57. David Yi, James Bruno, Jiayu Han, Peter Zukerman, and Shane Steinert-Threlkeld. 2022. Probing for understanding of English verb classes and alternations in large pre-trained language models. In Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages… find
  58. Amir Zeldes, Nick Howell, Noam Ordan, and Yifat Ben Moshe. 2022. A second wave of UD Hebrew treebanking and cross-domain parsing. In Proceedings of EMNLP 2022, pages 4331– 4344, Abu Dhabi, UAE. find our card
  59. Lingze Zhang and Ellie Pavlick. 2025. Does training on synthetic data make models less robust? arXiv, https://arxiv.org/abs/2502.07164. ↗
  60. ENGLISH COSMinLex COSMaxLex - - 21 49 1 1 4 0 2 1 0 0 0 2 0 0 Wrong Answer 1 Pat Akt by-NP 2 Ag Akt by-NP 3 Pat Pass by-Ag 4 Ag Pass by-Pat 5 Pat Akt Ag 6 Ag Akt Pat 7 Pat Akt by-Pat 8 Ag Akt by-Pat Error Type Grammar Grammar Sequence Sequence Sequence Sequence Grammar Grammar Italian COSMinLex COSM… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/samo-2026-datasets-verb-alternations-across-languages/