Elamur Research⌕
☰
home / papers / 2026 / Can a Remedy Find a Researcher? Exploring the Development of Semantic Knowledge in Italian BabyLMs

Can a Remedy Find a Researcher? Exploring the Development of Semantic Knowledge in Italian BabyLMs

Summary

The paper asks whether small Italian language models trained on child-scale input learn knowledge about events and their typical participants, and how they compare with larger Italian LMs. The authors port the English minimal-pair materials of Kauf et al. (2023) to Italian. ChatGPT drafted the translations, the authors corrected them by hand and added passive variants, and Italian speakers rated sentence plausibility to validate the set. Each pair contrasts a normal transitive sentence with a role-swapped version. The swapped version is either impossible because it breaks an animacy restriction (AI) or merely unlikely (AA), and reversible pairs serve as a control. The tested models are the BAMBI family, trained on growing, developmentally staged corpora with or without curriculum learning, plus Minerva-3B, Cerbero-7B and Velvet_2B. Scoring uses accuracy, mean log-likelihood (MLL) offset and expected calibration error (ECE). All models beat chance. Even the earliest BAMBI stage detects impossible events, but separating plausible from implausible events needs more data. This clarifies how much semantic knowledge limited input can support.

Key points

  • Benchmark size: The first adapted dataset has 1,648 sentences in 824 minimal pairs, split evenly into 412 active and 412 passive items. Of these, 522 items form high/low-frequency near-synonym pairs.
  • Second dataset: It adds 62 AA items, 22 of which have passive versions. Totals are 272 AI items, 332 AA items and 282 AA-rev control items. 232 Italian-speaking participants validated plausibility on a 7-point scale.
  • Calibration: BAMBI models are well calibrated, with mean confidence within about 0.05–0.08 of accuracy. Minerva-3B and Cerbero-7B are the most accurate but underconfident. Velvet_2B has the lowest accuracy and the highest ECE, making it strongly overconfident.
  • Data source and stage effects: BAMBI trained on mC4 web text does slightly worse than BAMBI 0–18_CL trained on child-oriented data of equal size, which points to reporting bias. Curriculum learning makes little difference. Weak-violation sensitivity becomes calibrated only beyond the 0–6M-word stage.
  • Title discrepancy: The title in the PDF header frames the work as probing how Italian BabyLMs react to semantic violations, rather than using the "Development of Semantic Knowledge" wording given for the library entry. The citation should be checked.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.starsem-conference.24

@inproceedings{suozzi2026can,
  title = {{Can a Remedy Find a Researcher? Exploring the Development of Semantic Knowledge in Italian BabyLMs}},
  author = {Suozzi, Alice and Capone, Luca and Lebani, Gianluca and Lenci, Alessandro},
  booktitle = {Proceedings of the 15th Joint Conference on Lexical and Computational Semantics (*SEM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.starsem-conference.24},
  url = {https://doi.org/10.18653/v1/2026.starsem-conference.24}
}

References

51 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Matilde Barbini, Maria Letizia Piccini Bianchessi, Veronica Bressan, Achille Fusco, Sofia Neri, Sarah Rossi, Tommaso Sgrizzi, and Cristiano Chesi. 2025. BLiMP-IT: Harnessing Automatic Minimal Pair Generation for Italian Language Model Evaluation. In Proceedings of the Eleventh Italian Conference on… find our card
  2. Ezgi Başar, Francesca Padovani, Jaap Jumelet, and Arianna Bisazza. 2025. Turblimp: A turkish benchmark of linguistic minimal pairs. arXiv e-prints, pages arXiv–2506. find our card
  3. Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting linear mixed-effects models using lme4. Journal of Statistical Software, 67(1):1– 48. find our card
  4. Julie E. Boland, Michael K. Tanenhaus, and Susan M. Garnsey. 1990. Evidence for the immediate use of verb control information in sentence processing. Journal of Memory and Language, 29(4):413–432. find
  5. Julie E. Boland, Michael K. Tanenhaus, Susan M. Garnsey, and Greg N. Carlson. 1995. Verb argument structure in parsing and interpretation: Evidence from wh-questions. Journal of Memory and Language, 34(6):774–806. find
  6. Christopher Boseak. 2025. Evaluating log-likelihood for confidence estimation in llm-based multiple-choice question answering. Innovative Journal of Applied Science, 2(4):29. find
  7. Luca Capone, Alice Suozzi, Gianluca E. Lebani, and Alessandro Lenci. 2025. Bambi goes to school: Evaluating italian babylms with invalsi-ita. In Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025). find
  8. Rui P. Chaves and Stephanie N. Richter. 2021. Look at that! bert can be easily distracted from paying attention to morphosyntax. In Proceedings of the Society for Computation in Linguistics 2021, pages 28–38. find
  9. Won Ik Cho, Emmanuele Chersoni, Yu-Yin Hsu, and Chu-Ren Huang. 2021. Modeling the influence of verb aspect on the activation of typical event locations with BERT. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2922–2929. find
  10. Moreno I. Coco, Antje Nuthmann, and Olaf Dimigen. 2020. Fixation-related brain potentials during semantic integration of object–scene information. Journal of Cognitive Neuroscience, 32(4):571—-589. find
  11. Yanai Elazar, Hongming Zhang, Yoav Goldberg, and Dan Roth. 2021. Back to square one: Artifact detection, training and commonsense disentanglement in the Winograd schema. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10486–10500. find
  12. Allyson Ettinger. 2020. What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language models. Transactions of the Association for Computational Linguistics, 8:34–48. find our card
  13. Evelina Fedorenko, Idan Asher Blank, Matthew Siegelman, and Zachary Mineroff. 2020. Lack of selectivity for syntax relative to word meanings throughout the language network. Cognition, 203(104348). find
  14. Todd R. Ferretti, Ken McRae, and Andrea Hatherell. 2001. Integrating verbs, situation schemas, and thematic role concepts. Journal of Memory and Language, 44(4):516–547. find
  15. Federico A Galatolo and Mario GCA Cimino. 2023. Cerbero-7B: A Leap Forward in Language-Specific LLMs Through Enhanced Chat Corpus Generation and Evaluation. arXiv preprint arXiv:2311.15698. — [arXiv:2311.15698](https://arxiv.org/abs/2311.15698) find
  16. Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In *SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the s… find our card
  17. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017. On calibration of modern neural networks. In International conference on machine learning, pages 1321–1330. find
  18. Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, and Ethan Gotlieb Wilcox. 2024. Findings of the second BabyLM challenge: Sample-efficient pretraining on developmentally plausible corpora. In The 2nd BabyLM Challen… find
  19. Philip A Huebner, Elior Sulem, Fisher Cynthia, and Dan Roth. 2021. Babyberta: Learning more grammar with small-scale child-directed language. In Proceedings of the 25th conference on computational natural language learning, pages 624–646. find
  20. Anna A. Ivanova, Zachary Mineroff, Vitor Zimmerer, Nancy Kanwisher, Rosemary Varley, and Evelina Fedorenko. 2021. The language network is recruited but not required for nonverbal event semantics. Neurobiology of Language, 2. find
  21. Anne-Lise Jouen, Nicolas Cazin, Carol Madden-Lombardi Sullivan Hidot, Jocelyne Ventre-Dominey, and Peter Ford Dominey. 2019. Beyond the word and image: III. Neurodynamic properties of the semantic network. bioRxiv, 62:621–647. find
  22. Jaap Jumelet, Leonie Weissweiler, Joakim Nivre, and Arianna Bisazza. 2025. Multiblimp 1.0: A massively multilingual benchmark of linguistic minimal pairs. arXiv preprint arXiv:2504.02768. find
  23. Carina Kauf, Anna A. Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan Selena She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci. 2023. Event knowledge in large language models: The gap between the impossible and the unlikely. Cognitive Science, 47(11). find
  24. Marta Kutas and Kara D. Federmeier. 2011. Thirty years and counting: Finding meaning in the n400 component of the event-related brain potential (erp). Annual Review of Psychology, 62:621–647. find
  25. Marta Kutas and Steven A. Hillyard. 1980. Reading senseless sentences: Brain potentials reflect semantic incongruity. Science, 207(4427):203–205. find
  26. Russell V. Lenth and Julia Piaskowski. 2025. emmeans: Estimated Marginal Means, aka Least-Squares Means. R package version 2.0.0. find
  27. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR’12, page 552–561. find
  28. LimeSurvey GmbH. 2026. source survey tool. find
  29. Limesurvey: An open Yikang Liu, Yeting Shen, Hongao Zhu, Lilong Xu, Zhiheng Qian, Siyuan Song, Kejia Zhang, Jialong Tang, Pei Zhang, Baosong Yang, and 1 others. 2024. Zhoblimp: a systematic assessment of language models with linguistic minimal pairs in chinese. arXiv e-prints. find
  30. Zeyu Liu, Yizhong Wang, Jungo Kasai, Hannaneh Hajishirzi, and Noah A Smith. 2021. Probing across time: What does roberta know and when? Findings of the Association for Computational Linguistics: EMNLP 2021. find
  31. Rebecca Marvin and Tal Linzen. 2018. Targeted syntactic evaluation of language models. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1192–1202. find
  32. Ken McRae, Mary Hare, Jeffrey E. Elman, and Todd R. Ferretti. 2005. A basis for generating expectancies for verbs from nouns. Memory and Cognition, 33(7):1174–1184. find
  33. Ken McRae, Michael J. Spivey-Knowlton, and Michael K. Tanenhaus. 1998. Modeling the influence of thematic fit (and other constraints) in on-line sentence comprehension. Journal of Memory and Language, 38:283–312. find
  34. OpenAI. 2023. ChatGPT (March 14 version). find
  35. Riccardo Orlando, Luca Moroni, Pere-Lluís Huguet Cabot, Simone Conia, Edoardo Barba, Sergio Orlandini, Giuseppe Fiameni, and Roberto Navigli. 2024. Minerva llms: The first family of large language models trained from scratch on italian data. In Proceedings of the Tenth Italian Conference on Computat… find
  36. Maja Pavlovic. 2025. Understanding model calibration-a gentle introduction and visual exploration of calibration and the expected calibration error (ECE). In The Fourth Blogpost Track at ICLR 2025. find
  37. Paolo Pedinotti, Giulia Rambelli, Emmanuele Chersoni, Enrico Santus, Alessandro Lenci, and Philippe Blache. 2021. Did the cat drink the coffee? challenging transformers with generalized event knowledge. In Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics, p… find
  38. Prolific. 2026. Prolific. find
  39. R Core Team. 2024. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. Version 4.3.3. find
  40. Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2020. On the systematicity of probing contextualized word representations: The case of hypernymy in BERT. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 88–102. find
  41. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902– 4912. find
  42. Association for Computational Linguistics, 8:377– 392. find
  43. Jason Wei, Dan Garrette, Tal Linzen, and Ellie Pavlick. 2021. Frequency effects on syntactic rule learning in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 932–948. find
  44. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling… find
  45. Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. SWAG: A large-scale adversarial dataset for grounded commonsense inference. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 93–104. find
  46. Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel Bowman. 2021. When do you need billions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: L… find
  47. Xinyu Zhou, Delong Chen, Samuel Cahyawijaya, Xufeng Duan, and Zhenguang Cai. 2025. Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 6866–6888. find
  48. Julien Romero and Simon Razniewski. 2022. Do children texts hold the key to commonsense knowledge? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10954–10959. find
  49. Alice Suozzi, Luca Capone, Gianluca E Lebani, and Alessandro Lenci. 2025. Bambi: Developing baby language models for italian. Lingue e linguaggio, Rivista semestrale, 1/2025:83–102. find
  50. Alex Warstadt, Aaron Mueller, Leshem Choshen, Ethan Wilcox, Chengxu Zhuang, Juan Ciro, Rafael Mosquera, Bhargavi Paranjabe, Adina Williams, Tal Linzen, and 1 others. 2023. Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora. In Proceedings of the BabyL… find
  51. Alex Warstadt, Alicia Parrish, Haokun Liu, Anhad Mohananey, Wei Peng, Sheng-Fu Wang, and Samuel R Bowman. 2020. BLiMP: The benchmark of linguistic minimal pairs for English. Transactions of the *Figure 3: Distribution of MLL offsets across models and conditions.* find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/suozzi-2026-can-remedy-find-researcher-exploring/