Elamur Research⌕
☰
home / papers / 2026 / ProverbIT - Easy to complete, hard to choose: A CALAMITA Challenge

ProverbIT - Easy to complete, hard to choose: A CALAMITA Challenge

Paper · 2026 · Enrico Mensa, Lorenzo Zane, Calogero Jerik Scozzaro et al.

Summary

ProverbIT tests whether LLMs can reject wrong answers or merely recognize familiar text. It was built for the CALAMITA initiative at EVALITA 2026 and contains 100 widely used Italian proverbs. Three native-speaker authors selected them from a list of 200. Each proverb is split into an opening and an ending. Models see the opening with four handcrafted false endings: a sound-alike (assonant), a literal non-assonant synonym, an inverted meaning, and a trivial or tautological one. A fifth option, "None of the other answers", is always correct. A free-generation completion task, scored with edit distance, serves as a baseline to separate missing knowledge from failed discrimination. In zero-shot tests at temperature 0, Llama-3.1-70B-Instruct completed 67% of proverbs but picked the right option only 14% of the time. Smaller 7B/8B models, including Italian-tuned ones, reached at most 5% on multiple choice. Errors cluster on the synonym and assonant distractors. The work shows that fluent completion can hide weak discriminative reasoning, and it releases a CC BY 4.0 dataset on Hugging Face.

Key points

  • The dataset has 100 Italian proverbs, each with four manually written distractors (assonant, synonym, inverse, trivial). The correct answer is always option E. It is distributed as CSV at huggingface.co/datasets/emensa/proverbIT.
  • Completion vs. multiple-choice accuracy:
    • Llama-3.1-70B-Instruct: 0.67 vs. 0.14
    • Llama-3.1-8B-Instruct: 0.20 vs. 0.03
    • LLaMAntino-3-ANITA-8B-Inst-DPO-ITA: 0.13 vs. 0.05
    • Minerva-7B-instruct-v1.0: 0.13 vs. 0.00
  • Distractor biases:
    • Llama 70B chose the synonym option 56 times, and Minerva 7B chose it 62 times.
    • Llama 8B favored the assonant option (43 times).
    • Minerva produced 33 invalid, non-letter outputs.
  • Completion accuracy relies on Python difflib edit distance, which heavily penalizes verbose answers from smaller models that ignore the output format.
  • Stated limitations are the small size and the manual construction of distractors, which makes the dataset hard to scale.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

Cite

Cite as Enrico Mensa, Lorenzo Zane, Calogero Jerik Scozzaro, Matteo Delsanto, Tommaso Milani and Daniele P. Radicioni. 2026. ProverbIT - Easy to complete, hard to choose: A CALAMITA Challenge. Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mensa2026proverbit,
  title = {{ProverbIT - Easy to complete, hard to choose: A CALAMITA Challenge}},
  author = {Enrico Mensa, Lorenzo Zane, Calogero Jerik Scozzaro, Matteo Delsanto, Tommaso Milani and Daniele P. Radicioni},
  booktitle = {Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)},
  year = {2026},
  url = {https://aclanthology.org/2026.evalita-1.72/}
}

References

8 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, et al., Solving quantitative reasoning problems with language models, Advances in Neural Information Processing Systems 35 (2022) 3843–3857.
  2. Y. Chang, X. Wang, J. Wang, Y. Wu, L. Yang, K. Zhu, H. Chen, X. Yi, C. Wang, Y. Wang, et al., A survey on evaluation of large language models, ACM transactions on intelligent systems and technology 15 (2024) 1–45.
  3. F. D. L. Fornaciari, B. Altuna, I. Gonzalez-Dios, M. Melero, A hard nut to crack: Idiom detection with conversational large language models, in: Proceedings of the 4th Workshop on Figurative Language Processing (FigLang 2024), 2024, pp. 35–44.
  4. G. Attanasio, P. Basile, F. Borazio, D. Croce, M. Francis, J. Gili, E. Musacchio, M. Nissim, V. Patti, M. Rinaldi, et al., Calamita: Challenge the abilities of language models in italian, in: Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024), Pisa, Italy, 2024. find
  5. M. Nissim, D. Croce, V. Patti, P. Basile, G. Attanasio, E. Musacchio, M. Rinaldi, F. Borazio, M. Francis, J. Gili, D. Scalena, B. Altuna, E. Azurmendi, V. Basile, L. Bentivogli, A. Bisazza, M. Bolognesi, D. Brunato, T. Caselli, S. Casola, M. Cassese, M. Cettolo, C. Collacciani, L. D. Cosmo, M. P. D.… — [arXiv:2512.04759](https://arxiv.org/abs/2512.04759) find
  6. E. Mensa, L. Zane, C. J. Scozzaro, M. Delsanto, T. Milani, D. P. Radicioni, Easy to complete, hard to choose: Investigating LLM performance on the ProverbIT benchmark, in: C. Bosco, E. Jezek, M. Polignano, M. Sanguinetti (Eds.), Proceedings of the Eleventh Italian Conference on Computational Linguis… find
  7. C. Bosco, E. Jezek, M. Polignano, M. Sanguinetti (Eds.), Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), CEUR Workshop Proceedings, Cagliari, Italy, 2025. URL: https://aclanthology.org/2025.clicit-1.0/. — [ACL 2025.clicit-1.0](https://aclanthology.org/2025.clicit-1.0/) ↗
  8. F. Caramagna, I 200 proverbi italiani più belli e famosi (con significato), 2025. URL: https://aforisticamente.com/i-200-proverbi-italiani-piu-belli-e-famosi-con-significato/. ↗

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mensa-2026-proverbit-easy-complete-hard-choose/