Elamur Research⌕
☰
home / papers / 2025 / From Form to Function: A Constructional NLI Benchmark

From Form to Function: A Constructional NLI Benchmark

Paper · 2025 · Claire Bonial, Taylor Pellegrin, Melissa Torgbi et al.

Summary

Construction Grammar treats phrasal constructions (cxns) as form–meaning pairings whose meaning goes beyond the words they contain. Earlier work mostly checked whether language models can spot such constructions. This paper instead tests whether models can use constructional meaning in a downstream task.

The authors built CoGS-NLI, an inference benchmark. They took about 500 corpus sentences covering 10 construction types from the CoGS corpus and used each one as a premise. For every premise, one annotator wrote an entailed, a neutral and a contradicted hypothesis, giving 1,500 triples. Agreement checks were followed by revision of the hypotheses.

GPT-3.5-turbo and GPT-4o were evaluated zero-shot and with few-shot examples drawn either from CoGS-NLI or from SNLI. Few-shot examples featuring the target constructions helped most, lifting GPT-4o to .94 accuracy. Generic SNLI examples gave little or no gain, and for GPT-3.5 they made results worse.

The authors also flag a limitation. High scores do not prove that models grasp constructional semantics, because some hypotheses can be solved through lexical cues unrelated to the construction.

Key points

  • Dataset and agreement: CoGS-NLI has 1,500 NLI triples over ~500 premises, with roughly 50 per construction type and 10 types. The types range from fixed-word cxns (e.g., Much-less) to fully schematic ones (e.g., Resultative, Ditransitive).
    • Initial agreement on 441 instances was 71–80% (Cohen's κ .55–.70).
    • After rewording, agreement rose to 89% (κ .84), comparable to SNLI's reported annotator agreement.
  • Setup: 50 instances were held out as in-context examples, and the remaining 391 formed the test set. The models were GPT-4o-2024-05-13 and GPT-3.5-turbo-0125 at temperature 0. Six prompt variants were tried, with and without chain-of-thought.
  • GPT-4o results: 0.89 zero-shot, 0.94 with 3-shot CoGS-NLI examples, and 0.90 with 3-shot SNLI examples.
  • GPT-3.5 results: 0.74 zero-shot, 0.83 with 3-shot CoGS-NLI examples, and 0.69 with 3-shot SNLI examples.
  • By construction type: Accuracy showed no clear dependence on schematicity level. GPT-4o did best on the fully variable Resultative and worst on the partially variable Way-manner. The authors suggest construction frequency in pretraining data as a possible explanation.

Interesting ideas

  • I1. Writing all three relation types (entailed, neutral, contradicted) for the same premise keeps the premise fixed, so any difference in model behaviour must come from how the hypothesis relates to it.
  • I2. Comparing in-context examples that contain the target construction against generic NLI examples separates "knowing the task format" from "knowing the construction"; here the generic examples even hurt GPT-3.5.
  • I3. The authors noticed that some items are solvable through cues outside the construction, such as an adverb clashing with an adjective, which inflates apparent understanding.
  • I4. Constructions like the Resultative license a result state that the verb alone does not express (brushing hair smooth yields smooth hair).
  • I5. Template-generated hypotheses and free-form human-written ones are treated as complementary, since templates may leak regular patterns that make the task easier.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Build reader tests as premise-centred triples, so that failures reveal which inference type breaks rather than which sentence was hard.
  • I2 (learn): When prompting LLM teacher tiers for silver frames, supply construction-matched exemplars and use generic exemplars as a negative-control condition.
  • I3 (learn): Tag each evaluation item by whether its decisive cue lies in the construction or in other lexical content, and score the construction-only subset separately.
  • I4 (frames): Add constructional frames (Resultative, Way-manner, Caused-motion, Ditransitive) on top of VerbNet roles, so that the result state or motion path is emitted even when the verb's own class lacks it.
  • I5 (learn): Pair self-generated templated test items with a smaller hand-written set, to detect whether a gain reflects real competence or overfitting to template regularities.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Claire Bonial, Taylor Pellegrin, Melissa Torgbi and Harish Tayyar Madabushi. 2025. From Form to Function: A Constructional NLI Benchmark. Proceedings of the Second International Workshop on Construction Grammars and NLP.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bonial2025form,
  title = {{From Form to Function: A Constructional NLI Benchmark}},
  author = {Claire Bonial, Taylor Pellegrin, Melissa Torgbi and Harish Tayyar Madabushi},
  booktitle = {Proceedings of the Second International Workshop on Construction Grammars and NLP},
  year = {2025},
  url = {https://aclanthology.org/2025.cxgsnlp-1.18/}
}

References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2016. Neural machine translation by jointly learning to align and translate.
  2. Claire Bonial and Harish Tayyar Madabushi. 2024a. Constructing Understanding: on the Constructional Information Encoded in Large Language Models. Language Resources and Evaluation, pages 1–40. ↗
  3. Claire Bonial and Harish Tayyar Madabushi. 2024b. A construction grammar corpus of varying schematicity: A dataset for the evaluation of abstractions in language models. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 243–255, Torino, Italia. ELRA and ICCL.
  4. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. ↗
  5. Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018. Annotation artifacts in natural language inference data. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics. ↗
  6. Matt A Johnson and Adele E Goldberg. 2013. Evidence for automatic accessing of constructional meaning: Jabberwocky sentences prime associated verbs. Language and Cognitive Processes, 28(10):1439–1452. ↗
  7. Lung-Hao Lee, Chen-Ya Chiou, and Tzu-Mi Lin. 2024. Nycu-nlp at semeval-2024 task 2: Aggregating large language models in biomedical natural language inference for clinical trials. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 1455–1462. find
  8. Hao Li, Wei Lu, Pengjun Xie, and Linlin Li. 2019. Neural Chinese address parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3421–3431, Minneapolis, Minnesota. Association for Computational Linguistics. find
  9. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  10. Marco Marelli, Luisa Bentivogli, Marco Baroni, Raffaella Bernardi, Stefano Menini, and Roberto Zamparelli. 2014. Semeval-2014 task 1: Evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment.
  11. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, et al. 2024. Gpt-4o system card. ↗
  12. Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, Jie Zhou, Siming Chen, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. ↗
  13. Laura Pérez-Mayos, Miguel Ballesteros, and Leo Wanner. 2021. How much pretraining data do language models need to learn syntax? In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1571–1582. ↗
  14. Shijia Zhou, Leonie Weissweiler, Taiqi He, Hinrich Schütze, David R. Mortensen, and Lori Levin. 2024. Constructions Are So Difficult That Even Large Language Models Get Them Right for the Wrong Reasons. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), page 3804–3811, Torino, Italia. ELRA and ICCL. find
  15. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(1). ↗
  16. Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. 2020. Super-Glue: Learning Feature Matching With Graph Neural Networks . In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4937–4946, Los Alamitos, CA, USA. IEEE Computer Society. find
  17. Wesley Scivetti, Tatsuya Aoyama, Ethan Wilcox, and Nathan Schneider. 2025a. Unpacking let alone: Human-scale models generalize to a rare construction in form but not meaning. arXiv preprint arXiv:2506.04408. ↗
  18. Wesley Scivetti, Melissa Torgbi, Austin Blodgett, Mollie Shichman, Taylor Hudson, Claire Bonial, and Harish Tayyar Madabushi. 2025b. Assessing language comprehension in large language models using construction grammar. find
  19. Harish Tayyar Madabushi, Laurence Romain, Dagmar Divjak, and Petar Milin. 2020. CxGBERT: BERT meets construction grammar. In Proceedings of the 28th International Conference on Computational Linguistics, pages 4020–4032, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  20. Harish Tayyar Madabushi, Melissa Torgbi, and Claire Bonial. 2025. Neither stochastic parroting nor agi: Llms solve tasks through context-directed extrapolation from training data priors. ↗
  21. Michael Tomasello. 2009. The usage-based theory of language acquisition. In The Cambridge handbook of child language, pages 69–87. Cambridge Univ. Press. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bonial-2025-form-function-constructional-nli-benchmark/