Elamur Research⌕
☰
home / papers / 2026 / CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models

CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models

Paper · 2026 · Miyu Oba and Saku Sugawara

Summary

Most linguistic evaluations of language models check whether a model can tell grammatical sentences from ungrammatical ones. They rarely test whether it grasps the meaning that a grammatical form carries. CxMP addresses this gap using Construction Grammar, which views constructions as pairings of form and meaning.

The benchmark covers nine English constructions, ranging from partially fixed patterns such as let-alone to fully schematic argument-structure patterns. Each item pairs a sentence containing the construction with a diagnostic sentence. The diagnostic sentence comes in two minimally different versions, one with a plausible reading and one with an implausible reading. Models are scored by comparing likelihoods, so the same test applies to small, developmentally plausible models and to 70B-scale LLMs.

The authors find that constructional understanding develops slowly and stays incomplete even in large models. Acceptability on BLiMP is mastered early in training. During learning, reliance on heuristic biases follows a U-shaped curve. Replacing content words with pseudowords sharply lowers performance.

The work shows that acceptability benchmarks miss form-encoded meaning and offers a controlled resource for studying how models acquire constructions.

Key points

  • Scope and design: Nine construction types are selected following Bonial and Tayyar Madabushi (2024). Each construction has two complementary variants (A and B). Entities are expressed as personal names, common nouns, or name+alphabet forms, and participant roles are swapped to expose lexical biases. The dataset is released at github.com/nii-cl/cxmp.
  • Model size versus instruction tuning: Small models make few correct judgments, and 70B-scale open models still fail on some constructions. Larger size helps most with formally complex constructions, while instruction tuning matters more for argument-structure constructions.
  • Learning trajectories: BLiMP accuracy rises early in training, whereas CxMP scores improve gradually. Learning passes through stages: near-random answers, then spurious heuristics, then stabilization. Bias magnitude goes low, then high, then low again.
  • Pseudoword test: Pseudowords were generated with Wuggy and tested on Llama3.1-70B. Scores fell markedly but stayed slightly above chance. CAUSED-MOTION and INTRANSITIVE-MOTION, which contain closed-class words, showed the smallest drops.
  • Limitations: The benchmark is English-only, uses binary plausibility judgments, and relies on diagnostic sentences written by the authors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of TokyoTokyo · Japan96 works · 9,600 citations🎓Nara Institute of Science and TechnologyIkoma · Japan48 works · 4,565 citations🔬National Institute of InformaticsTokyo · Japan38 works · 3,247 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksPsychology · 1,649 worksDevelopmental and Educational Psychology · 523 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEvaluation & benchmarks · 1,341 works

Cite

Cite as Miyu Oba and Saku Sugawara. 2026. CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2132

@inproceedings{oba2026cxmp,
  title = {{CxMP: A Linguistic Minimal-Pair Benchmark for Evaluating Constructional Understanding in Language Models}},
  author = {Miyu Oba and Saku Sugawara},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2132},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2132}
}

References

42 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, and 1 others. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Co… find
  2. Claire Bonial and Harish Tayyar Madabushi. 2024. A construction grammar corpus of varying schematicity: A dataset for the evaluation of abstractions in language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COL…
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal. Association for Computati…
  4. Lucas Charpentier, Leshem Choshen, Ryan Cotterell, Mustafa Omer Gul, Michael Y. Hu, Jing Liu, Jaap Jumelet, Tal Linzen, Aaron Mueller, Candace Ross, Raj Sanjay Shah, Alex Warstadt, Ethan Gotlieb Wilcox, and Adina Williams. 2025. Findings of the third BabyLM challenge: Accelerating language modeling… find
  5. Lucas Georges Gabriel Charpentier and David Samuel. 2024. GPT or BERT: why not both? In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pages 262– 283, Miami, FL, USA. Association for Computational Linguistics. find
  6. Ionut Constantinescu, Tiago Pimentel, Ryan Cotterell, and Alex Warstadt. 2025. Investigating critical period effects in language acquisition through neural language models. Transactions of the Association for Computational Linguistics, 13:96–120. find
  7. William Croft. 2001. Radical Construction Grammar: Syntactic Theory in Typological Perspective. Oxford University Press. find
  8. Charles J. Fillmore. 1986. Varieties of conditional sentences. In Proceedings of the Eastern States Conference on Linguistics (ESCOL), volume 3, pages 163–182. find
  9. Charles J. Fillmore, Paul Kay, and Mary Catherine O’Connor. 1988. Regularity and idiomaticity in grammatical constructions: The case of Let Alone. Language, 64(3):501–538. find
  10. Richard Futrell and Kyle Mahowald. 2025. How linguistics learned to stop worrying and love the language models. Behavioral and Brain Sciences, page 1–98. find
  11. Adele Goldberg. 2006. Constructions at Work: The Nature of Generalization in Language. Oxford University Press. find
  12. Adele E. Goldberg. 1995. Constructions: A Construction Grammar Approach to Argument Structure. Cognitive Theory of Language and Culture Series. University of Chicago Press. find
  13. Adele E Goldberg. 2003. Constructions: a new theoretical approach to language. Trends in Cognitive Sciences, 7(5):219–224. find
  14. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techno…
  15. Thomas Hoffmann. 2017. Construction grammar as cognitive structuralism: the interaction of constructional networks and processing in the diachronic evolution of english comparative correlatives. English Language and Linguistics, 21(2):349–373. find
  16. Jennifer Hu and Roger Levy. 2023. Prompting is not a substitute for probability measurements in large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5040–5060, Singapore. Association for Computational Linguistics.
  17. Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox. 2024. Findings of the second BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In The 2nd BabyLM Challen… find
  18. Philip A. Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. BabyBERTa: Learning more grammar with small-scale child-directed language. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 624–646, Online. Association for Computational Linguistics.
  19. Yusuke Ide, Yuto Nishida, Justin Vasselli, Miyu Oba, Yusuke Sakai, Hidetaka Kamigaito, and Taro Watanabe. 2025. How to make the most of LLMs’ grammatical knowledge for acceptability judgments. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computa… find
  20. Anna A. Ivanova, Aalok Sathe, Benjamin Lipkin, Un-nathi U. Kumar, Setayesh Radkani, Thomas H. Clark, Carina Kauf, Jennifer Hu, R. T. Pramod, Gabriel Grand, Vivian C. Paulun, Maria Ryskina, Ekin Akyürek, Ethan G. Wilcox, Nafisa Rashid, Leshem Choshen, Roger Levy, Evelina Fedorenko, Joshua Tenenbaum,… find
  21. Edward Kako. 2006. The semantics of syntactic frames. Language and Cognitive Processes, 21(5):562–575. find
  22. Julie Kallini, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. 2024. Mission: Impossible language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14691–14714, Bangkok, Thailand. Association…
  23. Paul Kay and Charles J. Fillmore. 1999. Grammatical constructions and linguistic generalizations: The what’s X doing Y? construction. Language, 75(1):1–33. find
  24. Emmanuel Keuleers and Marc Brysbaert. 2010. Wuggy: A multilingual pseudoword generator. Behavior Research Methods, 42(3):627–633. find
  25. Tom Mackintosh, Harish Tayyar Madabushi, and Claire Bonial. 2025. Evaluating CxG generalisation in LLMs via construction-based NLI fine tuning. In Proceedings of the Second International Workshop on Construction Grammars and NLP, pages 180–189, Düsseldorf, Germany. Association for Computational Ling…
  26. R. Thomas McCoy, Ellie Pavlick, and Tal Linzen. 2019. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3428–3448, Florence, Italy. Association for Computationa…
  27. Meta. 2024. The Llama 3 herd of models. Preprint, arXiv:2407.21783.
  28. Kanishka Misra and Kyle Mahowald. 2024. Language models learn rare phenomena from less rare phenomena: The case of the missing AANNs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 913–929, Miami, Florida, USA. Association for Computational Linguisti…
  29. Marianna Nezhurina, Jörg Franke, Taishi Nakamura, Timur Carstensen, Niccolò Ajroldi, Ville Komulainen, David Salinas, and Jenia Jitsev. 2025. Open-sci-ref-0.01: open and reproducible reference baselines for language model and dataset comparison. Preprint, arXiv:2509.09009. — [arXiv:2509.09009](https://arxiv.org/abs/2509.09009) find
  30. Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, and 21 others. 2024. 2 OL… find
  31. Christopher Potts. 2024. Characterizing english preposing in PP constructions. Journal of Linguistics, page 1–39. find
  32. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. WinoGrande: An adversarial Winograd Schema Challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8732–8740. find
  33. Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020. Masked language model scoring. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2699–2712, Online. Association for Computational Linguistics.
  34. Wesley Scivetti, Tatsuya Aoyama, Ethan Wilcox, and Nathan Schneider. 2025a. Unpacking Let Alone: Human-scale models generalize to a rare construction in form but not meaning. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 27503–27514, Suzhou, China.… find
  35. Wesley Scivetti, Melissa Torgbi, Mollie Shichman, Taylor Pellegrin, Austin Blodgett, Claire Bonial, and Harish Tayyar Madabushi. 2025b. Beyond memorization: Assessing semantic generalization in large language models using phrasal constructions. In Proceedings of the 14th International Joint Conferen… find
  36. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, and 1 others. 2025. OpenAI GPT-5 system card. Preprint, arXiv:2601.03267.
  37. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R. Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the Association for Computational Linguistics, 8:377– 392.
  38. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. find
  39. Leonie Weissweiler, Valentin Hofmann, Abdullatif Köksal, and Hinrich Schütze. 2022. The better your syntax, the better your semantics? probing pretrained language models for the English comparative correlative. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing…
  40. Leonie Weissweiler, Kyle Mahowald, and Adele E. Goldberg. 2025. Linguistic generalizations are not rules: Impacts on evaluation of LMs. In Proceedings of the Second International Workshop on Construction Grammars and NLP, pages 61–74, Düsseldorf, Germany. Association for Computational Linguistics.
  41. Xinru Yu, Bin Guo, Shiwei Luo, Jie Wang, Tao Ji, and Yuanbin Wu. 2024. AntLM: Bridging causal and masked language models. In The 2nd BabyLM Challenge at the 28th Conference on Computational Natural Language Learning, pages 324–331, Miami, FL, USA. Association for Computational Linguistics. find
  42. Qi Zhang, Tianqi Du, Haotian Huang, Yifei Wang, and Yisen Wang. 2024. Look ahead or look around? A theoretical comparison between autoregressive and masked pretraining. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pa… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/oba-2026-cxmp-linguistic-minimal-pair-benchmark/