Elamur Research⌕
☰
home / papers / 2026 / UniTor at Cruciverb-IT: Retrieval-Augmented Two-Step Reasoning for Italian Crossword Clue Answering

UniTor at Cruciverb-IT: Retrieval-Augmented Two-Step Reasoning for Italian Crossword Clue Answering

Paper · 2026 · Andriy Shcherbakov, Danilo Croce and Roberto Basili

Summary

The paper addresses Italian crossword clue answering in Cruciverb-IT Task 1 at EVALITA 2026. Given a clue and the exact length of the answer, a system must return candidate solutions ranked by confidence. The authors present UniTor, an LLM-based solver that treats the task as constrained ranking and uses no task-specific fine-tuning.

UniTor first retrieves length-compatible examples from a large repository of clue–solution pairs. The authors compared lexical, neural and hybrid retrieval and chose neural retrieval for the final system. The retrieved examples serve as few-shot evidence in a prompt that runs in one model call and has two stages. The model first brainstorms a varied set of candidates. It then removes candidates of the wrong length, after normalizing them to uppercase A–Z, and re-ranks the rest with explicit confidence scores.

Ablations show that retrieval and two-step reasoning help in complementary ways, and that retrieval lets smaller models narrow the gap with larger ones. UniTor ranked first in the official evaluation. The results suggest that careful prompting and retrieval can compete with training-heavy approaches on knowledge-intensive tasks with loosely specified inputs.

Key points

  • Official results: UniTor scored Accuracy@1 0.69, Accuracy@10 0.83 and MRR 0.72. The second-best system scored 0.58 / 0.75 / 0.63, and the BM25 baseline scored 0.40 / 0.62 / 0.46.
  • Margins: UniTor beat the second-best system by +0.11 in Accuracy@1 and +0.09 in MRR. It beat the average of all submitted runs by +0.17 in Accuracy@1 and +0.15 in MRR.
  • Final configuration: the system uses k = 10 retrieved suggestions, two-step prompting and probabilistic output with the GLM model. Configurations based on GPT-OSS also beat the baseline and most competing systems.
  • Research questions: the study tests whether asking for probability scores beats plain list generation and whether two-step prompting beats single-pass inference. It also examines the effect of model scale and how retrieval interacts with scale and prompting.
  • Limitations and future work: the system answers isolated clues without grid or crossing-letter information. The authors propose integrating it into a full grid solver where UniTor's scores would act as soft constraints.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,426 worksInformation retrieval & search · 473 works

Cite

Cite as Andriy Shcherbakov, Danilo Croce and Roberto Basili. 2026. UniTor at Cruciverb-IT: Retrieval-Augmented Two-Step Reasoning for Italian Crossword Clue Answering. Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{shcherbakov2026unitor,
  title = {{UniTor at Cruciverb-IT: Retrieval-Augmented Two-Step Reasoning for Italian Crossword Clue Answering}},
  author = {Andriy Shcherbakov, Danilo Croce and Roberto Basili},
  booktitle = {Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026)},
  year = {2026},
  url = {https://aclanthology.org/2026.evalita-1.41/}
}

References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. C. Ciaccio, G. Sarti, A. Miaschi, F. Dell’Orletta, M. Nissim, Cruciverb-it @ evalita 2026: Overview of the crossword solving in italian task, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for Italian. Final Workshop (EVALITA 2026), CEUR.org, Bari, I… find
  2. F. Cutugno, A. Miaschi, A. P. Aprosio, G. Rambelli, L. Siciliani, M. A. Stranisci, Evalita 2026: Overview of the 9th evaluation campaign of natural language processing and speech tools for italian, in: Proceedings of the Ninth Evaluation Campaign of Natural Language Processing and Speech Tools for I… find
  3. C. Ciaccio, G. Sarti, A. Miaschi, F. Dell’Orletta, Crossword space: Latent manifold learning for Italian crosswords and beyond, in: C. Bosco, E. Jezek, M. Polignano, M. Sanguinetti (Eds.), Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), CEUR Workshop Proce… find
  4. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, D. Amodei, Scaling laws for neural language models, 2020. URL: https://arxiv.org/abs/2001.08361. arXiv:2001.08361 . ↗
  5. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. van den Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, L. Sifre, Training compute-optimal… — Training Compute-Optimal Large Language Models
  6. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, D. Kiela, Retrieval-augmented generation for knowledge-intensive nlp tasks, 2021. URL: https://arxiv.org/abs/2005.11401. arXiv:2005.11401 . ↗
  7. J. Wei, X. Wang, D. Schuurmans, M. Bosma, B. Ichter, F. Xia, E. Chi, Q. Le, D. Zhou, Chain-of-thought prompting elicits reasoning in large language models, 2023. URL: https://arxiv.org/abs/2201.11903. arXiv:2201.11903 . ↗
  8. X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, D. Zhou, Self-consistency improves chain of thought reasoning in language models, 2023. URL: https://arxiv.org/abs/2203. 11171. arXiv:2203.11171 . ↗
  9. J. Zhang, S. Yu, D. Chong, A. Sicilia, M. R. Tomz, C. D. Manning, W. Shi, Verbalized sampling: How to mitigate mode collapse and unlock llm diversity, 2025. URL: https://arxiv.org/abs/2510.01171. arXiv:2510.01171 . ↗
  10. M. L. Littman, G. A. Keim, N. Shazeer, A probabilistic approach to solving crossword puzzles, Artificial Intelligence 134 (2002) 23–55. URL: https://www.sciencedirect.com/science/article/pii/ S000437020100114X. doi: https://doi.org/10.1016/S0004-3702(01)00114-X . — [doi:10.1016/s0004-3702(01)00114-x](https://doi.org/10.1016/s0004-3702%2801%2900114-x) ↗
  11. G. Angelini, M. Ernandes, M. Gori, Webcrow: A web-based crosswords solver, in: M. Maybury, O. Stock, W. Wahlster (Eds.), Intelligent Technologies for Interactive Entertainment, Springer Berlin Heidelberg, Berlin, Heidelberg, 2005, pp. 295–298. find
  12. P. Basile, M. de Gemmis, P. Lops, G. Semeraro, Solving a complex language game by using knowledge-based word associations discovery, IEEE Transactions on Computational Intelligence and AI in Games 8 (2014) 1–1. doi: 10.1109/TCIAIG.2014.2355859 . — [doi:10.1109/tciaig.2014.2355859](https://doi.org/10.1109/tciaig.2014.2355859) find
  13. G. Barlacchi, M. Nicosia, A. Moschitti, Learning to rank answer candidates for automatic resolution of crossword puzzles, in: R. Morante, S. W.-t. Yih (Eds.), Proceedings of the Eighteenth Conference on Computational Natural Language Learning, Association for Computational Linguistics, Ann Arbor, Mi… find
  14. A. Moschitti, M. Nicosia, G. Barlacchi, SACRY: Syntax-based automatic crossword puzzle resolution sYstem, in: H.-H. Chen, K. Markert (Eds.), Proceedings of ACL-IJCNLP 2015 System Demonstrations, Association for Computational Linguistics and The Asian Federation of Natural Language Processing, Beijin… find
  15. S. Saha, S. Chakraborty, S. Saha, U. Garain, Language models are crossword solvers, in: L. Chiruzzo, A. Ritter, L. Wang (Eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Pape… find
  16. A. Sadallah, D. Kotova, E. Kochmar, What makes cryptic crosswords challenging for LLMs?, in: O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, S. Schockaert (Eds.), Proceedings of the 31st International Conference on Computational Linguistics, Association for Computational Linguisti… find
  17. J. Chen, S. Xiao, P. Zhang, K. Luo, D. Lian, Z. Liu, M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation, 2025. URL: https://arxiv.org/abs/2402.03216. arXiv:2402.03216 . ↗
  18. S. Robertson, H. Zaragoza, The probabilistic relevance framework: Bm25 and beyond, Found. Trends Inf. Retr. 3 (2009) 333–389. URL: https://dl.acm.org/doi/abs/10.1561/1500000019. ↗
  19. A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, al., The llama 3 herd of models, 2024. URL: https://arxiv.org/abs/2407.21783. arXiv:2407.21783 . ↗
  20. OpenAI, S. Agarwal, L. Ahmad, J. Ai, S. Altman, A. Applebaum, al., gpt-oss-120b & gpt-oss-20b model card, 2025. URL: https://arxiv.org/abs/2508.10925. arXiv:2508.10925 . ↗
  21. GLM-4.5 Team, A. Zeng, X. Lv, Q. Zheng, Z. Hou, B. Chen, C. Xie, C. Wang, D. Yin, H. Zeng, al., Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025. URL: https://arxiv.org/abs/ 2508.06471. arXiv:2508.06471 . ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shcherbakov-2026-unitor-cruciverb-it-retrieval-augmented/