Elamur Research⌕
☰
home / papers / 2026 / Kenji-Endo: a BabyLM @EVALITA

Kenji-Endo: a BabyLM @EVALITA

Paper · 2026 · Calogero Jerik Scozzaro, Matteo Rinaldi, Gianluca Mittone et al.

Summary

The paper asks whether a compact Italian language model trained on modest data can compete with much larger LLMs on real evaluation benchmarks. The authors built the Kenji-Endo corpus for pretraining. It mixes child language (CHILDES), spoken transcriptions (Kiparla, VoLip), tweets, legal texts (Eurlex, Gazzetta Ufficiale), easy-to-read news (DueParole), and books, forums and Wikipedia from TestiMole. Documents were filtered for readability using De Mauro's Nuovo Vocabolario di Base. Three decoder models were pretrained from scratch: a dense model, a Mixture-of-Experts model, and an ALiBi variant trained with the in-house Matformer library on consumer hardware. The models were adapted to classification either by fine-tuning with a classification head or by prompting. They were then entered in four EVALITA 2026 tasks. On certain subtasks the results were strong: the models came second on DeSegMa-IT's Subtask A and topped the ranking for Subtask 1 of IMPOLS. Performance dropped sharply on pragmatically abstract or noisy inputs. The work matters because it shows that small, cheaply trained models remain a practical option for Italian NLP.

Key points

  • Corpus: after readability filtering (third-quartile threshold per text type), the paper reports 184,000 documents and 83.8M tokens. The introduction gives a rough figure of 135M tokens, so the reported sizes do not fully agree.
  • Models:
    • Kenji-Endo vanilla: about 130M parameters, Qwen3ForCausalLM, 12 layers, hidden size 768.
    • Kenji-Endo MoE: about 240M parameters, Qwen2MoeForCausalLM, 7 experts with top-1 routing plus one shared expert.
    • Kenji-Endo AliBi: trained in 8 hours on a single RTX 3090 with the Muon optimizer.
  • Training setup: the vanilla and MoE models were pretrained on a single Grace Hopper (H100) node of the HPC4AI cluster, using bfloat16, the xFFL framework, a 1024-token context and 10 epochs.
  • IMPOLS: best F1 on Subtask 1 (0.9496, versus 0.9089 for the runner-up). Subtask 2 fell to 0.2226 because the model predicted only a single class.
  • FadeIT and MultiPRIDE: FadeIT Subtask A ranked 7th of 15 runs with micro F1 47.37 (baseline 20.84), but the system detected none of 14 out of 19 fallacy types. On MultiPRIDE Subtask B it scored F1 0.7489, below the 0.8981 baseline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of TurinTurin · Italy33 works · 6,637 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Calogero Jerik Scozzaro, Matteo Rinaldi, Gianluca Mittone and Marco Antonio Stranisci. 2026. Kenji-Endo: a BabyLM @EVALITA. Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{scozzaro2026kenji,
  title = {{Kenji-Endo: a BabyLM @EVALITA}},
  author = {Calogero Jerik Scozzaro, Matteo Rinaldi, Gianluca Mittone and Marco Antonio Stranisci},
  booktitle = {Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)},
  year = {2026},
  url = {https://aclanthology.org/2026.evalita-1.4/}
}

References

33 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, D. Amodei, Scaling laws for neural language models, arXiv preprint arXiv:2001.08361 (2020).
  2. J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short…
  3. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., Language models are unsupervised multitask learners, OpenAI blog 1 (2019) 9. find
  4. A. Meta, The llama 4 herd: The beginning of a new era of natively multimodal ai innovation, https://ai.meta. com/blog/llama-4-multimodal-intelligence/, checked on 4 (2025) 2025. ↗
  5. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, J. Dean, Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, in: International Conference on Learning Representations, 2017. find
  6. D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, et al., Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models, arXiv preprint arXiv:2401.06066 (2024). find
  7. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, Journal of machine learning research 21 (2020) 1–67.
  8. L. Soldaini, R. Kinney, A. Bhagia, D. Schwenk, D. Atkinson, R. Authur, B. Bogin, K. Chandu, J. Dumas, Y. Elazar, et al., Dolma: An open corpus of three trillion tokens for language model pretraining research, in: Proceedings of the 62nd annual meeting of the association for computational linguistics… find
  9. M. C. Rillig, M. Ågerstrand, M. Bi, K. A. Gould, U. Sauerland, Risks and benefits of large language models for the environment, Environmental science & technology 57 (2023) 3464–3466. find
  10. I. Lee, T. Berg-Kirkpatrick, Readability≠ learnability: Rethinking the role of simplicity in training small language models, in: Second Conference on Language Modeling, ???? find
  11. A. Warstadt, A. Mueller, L. Choshen, E. Wilcox, C. Zhuang, J. Ciro, R. Mosquera, B. Paranjabe, A. Williams, T. Linzen, et al., Findings of the babylm challenge: Sample-efficient pretraining on developmentally plausible corpora, in: Proceedings of the BabyLM challenge at the 27th conference on comput… find
  12. M. Y. Hu, A. Mueller, C. Ross, A. Williams, T. Linzen, C. Zhuang, R. Cotterell, L. Choshen, A. Warstadt, E. G. Wilcox, Findings of the second babylm challenge: Sample-efficient pretraining on developmentally plausible corpora, in: The 2nd BabyLM Challenge at the 28th Conference on Computational Natu… find
  13. L. G. G. Charpentier, L. Choshen, R. Cotterell, M. O. Gul, M. Y. Hu, J. Liu, J. Jumelet, T. Linzen, A. Mueller, C. Ross, et al., Proceedings of the first babylm workshop, in: Proceedings of the First BabyLM Workshop, 2025. find
  14. L. Capone, A. Suozzi, G. E. Lebani, A. Lenci, Bambi goes to school: Evaluating italian babylms with invalsi-ita (2025). find
  15. Z. Shen, A. Joshi, R.-C. Chen, Bambino-lm:(bilingual-) human-inspired continual pre-training of babylm, in: Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, 2024, pp. 1–7. find
  16. J. Jumelet, A. Fourtassi, A. Haga, B. Bunzeck, B. Shandilya, D. Galvan-Sosa, F. G. Haznitrama, F. Padovani, F. Meyer, H. Hu, et al., Babybabellm: A multilingual benchmark of developmentally plausible training data, arXiv preprint arXiv:2510.10159 (2025). find
  17. F. Cutugno, A. Miaschi, A. P. Aprosio, G. Rambelli, L. Siciliani, M. A. Stranisci, Evalita 2026: Overview of the 9th evaluation campaign of natural language processing and speech tools for italian, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for I… find
  18. G. Puccetti, A. Pedrotti, A. Esuli, Desegma-it at evalita 2026: Overview of the detection and segmentation of machine generated text in italian task, 2026. find
  19. C. Ferrando, L. Draetta, M. Madeddu, M. Sosto, V. Patti, P. Rosso, C. Bosco, J. Mata, E. Gualda, Multipride at evalita 2026: Overview of the multilingual automatic detection of slur reclamation in the lgbtq+ context task, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processin… find
  20. A. Ramponi, S. Tonelli, FadeIT at EVALITA 2026: Overview of the fallacy detection in italian social media texts task, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), CEUR.org, Bari, Italy, 2026.
  21. E. Goria, C. Mauri, Il corpus KIParla: una nuova risorsa per lo studio dell’italiano parlato, volume 2, CLUB- Circolo linguistico dell’Università di Bologna, Bologna, 2018, pp. 96–116. URL: https://hdl.handle.net/2318/1689340. ↗
  22. M. Voghera, C. Iacobini, F. Cutugno, R. Savy, I. Alfano, A. De Rosa, Il VoLIP: una risorsa per lo studio della variazione nel parlato della lingua italiana, in: Actes du XXVIIe Congrès international de linguistique et de philologie romanes, ELiPhi, Strasbourg, 2016, pp. 1651–1663. find
  23. M. A. Stranisci, S. Frenda, M. Lai, O. Araque, A. T. Cignarella, V. Basile, C. Bosco, V. Patti, O-dang! the ontology of dangerous speech messages, in: I. Kernerman, S. Carvalho, C. A. Iglesias, R. Sprugnoli (Eds.), Proceedings of the 2nd Workshop on Sentiment Analysis and Linguistic Linked Data, Eur… find
  24. A. Ramponi, C. Casula, DiatopIt: A corpus of social media posts for the study of diatopic language variation in Italy, in: Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), Association for Computational Linguistics, Dubrovnik, Croatia, 2023, pp. 187–199. URL: https:… find
  25. E. Federici, M. Ferraretto, N. Landro, Gazzetta Ufficiale: A dataset of legislative texts, public and private acts, 2024. URL: https://huggingface.co/datasets/mii-llm/gazzetta-ufficiale. ↗
  26. M. Rinaldi, R. Varvara, V. Patti, Testimole-conversational: A 30-billion-word italian discussion board corpus (1996-2024) for language modeling and sociolinguistic research, 2026. URL: https://arxiv.org/abs/2602.14819. arXiv:2602.14819 . ↗
  27. M. Aldinucci, S. Rabellino, M. Pironti, F. Spiga, P. Viviani, M. Drocco, M. Guerzoni, G. Boella, M. Mellia, P. Margara, I. Drago, R. Marturano, G. Marchetto, E. Piccolo, S. Bagnasco, S. Lusso, S. Vallero, G. Attardi, A. Barchiesi, A. Colla, F. Galeazzi, HPC4AI: an ai-on-demand federated platform end… — [doi:10.1145/3203217.3205340](https://doi.org/10.1145/3203217.3205340) find
  28. I. Colonnelli, R. Birke, G. Malenza, G. Mittone, A. Mulone, J. Galjaard, L. Y. Chen, S. Bassini, G. Scipione, J. Martinovič, V. Vondrák, M. Aldinucci, Cross-Facility Federated Learning, Procedia Computer Science 240 (2024) 3–12. doi: 10.1016/j.procs.2024.07.003 . — [doi:10.1016/j.procs.2024.07.003](https://doi.org/10.1016/j.procs.2024.07.003) find
  29. NVIDIA Developer, NVIDIA Hopper Architecture In Depth, 2023. URL: https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/, accessed: 2026-01-07. ↗
  30. O. Press, N. A. Smith, M. Lewis, Train short, test long: Attention with linear biases enables input length extrapolation, arXiv preprint arXiv:2108.12409 (2022). URL: https://arxiv.org/abs/2108.12409. arXiv:2108.12409 . ↗
  31. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, Y. Liu, Roformer: Enhanced transformer with rotary position embedding, Neurocomputing 568 (2024) 127063.
  32. K. Jordan, Y. Jin, V. Boza, Y. Jiacheng, F. Cesista, L. Newhouse, J. Bernstein, Muon: An optimizer for hidden layers in neural networks, 2024. URL: https://kellerjordan.github.io/posts/muon/. ↗
  33. A. Ramponi, A. Daffara, S. Tonelli, Fine-grained fallacy detection with human label variation, in: L. Chiruzzo, A. Ritter, L. Wang (Eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/scozzaro-2026-kenji-endo-babylm-evalita/