Elamur Research⌕
☰
home / papers / 2026 / Vinclat: Evaluating Reasoning, Cognition and Culture in One Game

Vinclat: Evaluating Reasoning, Cognition and Culture in One Game

Paper · 2026 · Marc Pàmies, Javier Aula-Blasco, Aitor Gonzalez-Agirre et al.

Summary

Evaluating LLM reasoning usually relies on math and coding datasets. These miss creative, multi-step inference that depends on cultural background, especially outside English. The authors address this gap with Vinclat, a Catalan benchmark built from a daily word-association puzzle of the same name.

Each item gives four independent clues. The model must first work out a keyword behind each clue, using proverbs, wordplay, carol lyrics or category membership. It must then find the single term that links those keywords, guided by the given letter and word counts. The original puzzle creators wrote the items. The team then edited them so each is self-contained, removing dependence on dates or earlier answers.

Open-weight models of various sizes and architectures were tested against human baselines, with and without internet access. Even the strongest systems reached only modest accuracy, well below humans. Errors came from misreading clues, invented Catalan words and missing cultural knowledge. Scaling trends suggest the benchmark should stay challenging for a long time. This makes it useful for probing culturally situated reasoning in mid-resource languages.

Key points

  • The dataset has 1,000 human-authored and reviewed Catalan instances. Each has 4 hints, intermediate keywords and one final solution whose word/letter pattern is unique. It is published as projecte-aina/vinclat on Hugging Face, alongside a public leaderboard (projecte-aina/vinclat-leaderboard).
  • Contamination risk is argued to be negligible for three reasons:
    • Puzzles appeared only one per day on vinclat.cat and were never released in bulk.
    • The site stopped adding items on December 2, 2024 and later removed all of them.
    • Only the hints are released, not the answers.
  • Topics range across arts, science, history, geography, politics, religion and Catalan idioms and folklore, mixing local and global knowledge.
  • Mixture-of-experts and reasoning-enabled models outperformed dense and non-reasoning ones, and Qwen3 models were used to analyse scaling. Reasoners responded better to structured prompts and lost less accuracy when prompted in Catalan.
  • The authors list several limitations:
    • Only a single-turn setup was used.
    • No commercial models (GPT-4, Claude, Gemini) were tested.
    • The human baselines are indicative rather than true upper bounds.
    • It is hard to separate reasoning failures from gaps in cultural exposure.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Barcelona Supercomputing CenterBarcelona · Spain14 works · 53 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,685 works

Cite

Cite as Marc Pàmies, Javier Aula-Blasco, Aitor Gonzalez-Agirre and Marta Villegas. 2026. Vinclat: Evaluating Reasoning, Cognition and Culture in One Game. Proceedings of the First Workshop on Multilingual Multicultural Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.mme-main.4

@inproceedings{pamies2026vinclat,
  title = {{Vinclat: Evaluating Reasoning, Cognition and Culture in One Game}},
  author = {Marc Pàmies, Javier Aula-Blasco, Aitor Gonzalez-Agirre and Marta Villegas},
  booktitle = {Proceedings of the First Workshop on Multilingual Multicultural Evaluation},
  year = {2026},
  doi = {10.18653/v1/2026.mme-main.4},
  url = {https://doi.org/10.18653/v1/2026.mme-main.4}
}

References

45 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alan Agresti and Brent A. Coull. 1998. Approximate is better than "exact" for interval estimation of binomial proportions. The American Statistician, 52(2):119– 126. find
  2. Frederic C. Bartlett. 1932. Remembering: A study in experimental and social psychology. Cambridge University Press. find
  3. Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Scott Wen tau Yih, and Yejin Choi. 2020. Abductive commonsense reasoning. Preprint, arXiv:1908.05739.
  4. Ellen Bialystok. 1986. Factors in the growth of linguistic awareness. Child development, pages 498–510. find
  5. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. Preprint, a…
  6. Peter Clark, Isaac Cowhey, Oren Etzioni, and 1 others. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
  7. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. NAACL-HLT.
  9. Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph, Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, and 8 others. 2022. Glam: Efficient scalin…
  10. Mor Geva, Daniel Khashabi, Wen-tau Chen, and Dan Roth. 2021. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics. find
  11. Akash Ghosh, Debayan Datta, Sriparna Saha, and Chirag Agarwal. 2025. The multilingual mind: A survey of multilingual reasoning in language models. arXiv preprint arXiv:2502.09457. find
  12. Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, and 1 others. 2024. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint arXiv:2411.04872…
  13. Alex Gu, Baptiste Rozière, Hugh Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida I Wang. 2024. Cruxeval: A benchmark for code reasoning, understanding and execution. arXiv preprint arXiv:2401.03065.
  14. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  15. Dan Hendrycks, Steven Burns, Sam Basart, Collin Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  16. Chengpeng Hu, Yunlong Zhao, Ziqi Wang, Haocheng Du, and Jialin Liu. 2024. Games for artificial intelligence research: A review and perspectives. IEEE Transactions on Artificial Intelligence. find
  17. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022.… find
  18. David Klahr and Kevin Dunbar. 1988. The dual space search hypothesis: An alternative to the problem space hypothesis for scientific reasoning. In Proceedings of the 10th Annual Conference of the Cognitive Science Society. find
  19. David R. Krathwohl. 2002. A revision of Bloom’s Taxonomy: An overview. Theory Into Practice, 41(4):212–218. of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11116–11141, Bangkok, Thailand. Association for Computational Linguistics. find
  20. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, and 3… find
  21. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  22. Angel Yahir Loredo Lopez, Tyler McDonald, and Ali Emami. 2025. NYT-connections: A deceptively simple text classification task that stumps system-1 thinkers. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1952–1963, Abu Dhabi, UAE. Association for Computationa… find
  23. Nikolai Lörer, Norman Lee, and Douwe Kiela. 2023. Conceptarc: Evaluating abstract reasoning in language models with the abstraction and reasoning corpus. arXiv preprint arXiv:2306.01774. — [arXiv:2306.01774](https://arxiv.org/abs/2306.01774) find
  24. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association fo… find
  25. Swaroop Mishra, Matthew Finlayson, Pan Lu, Leonard Tang, Sean Welleck, Chitta Baral, Tanmay Rajpurohit, Oyvind Tafjord, Ashish Sabharwal, Peter Clark, and 1 others. 2022. Lila: A unified benchmark for mathematical reasoning. arXiv preprint arXiv:2210.17517.
  26. Willem J.M. Levelt. 1989. Speaking: From intention to articulation. MIT Press. find
  27. Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of th…
  28. Jiawei Li, Yizhe Yang, Yu Bai, Xiaofeng Zhou, Yinghao Li, Huashan Sun, Yuhang Liu, Xingpeng Si, Yuhao Ye, Yixiao Wu, , Bin Xu, Ren Bowen, Chong Feng, Yang Gao, and Heyan Huang. 2024. Fundamental capabilities of large language models and their applications in domain scenarios: A survey. In Proceeding… find
  29. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. Preprint, arXiv:2309.06180.
  30. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennigho… find
  31. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. Preprint, arXiv:1701.06538. find
  32. Parshin Shojaee, Iman Mirzadeh, Keivan Alizadeh, Maxwell Horton, Samy Bengio, and Mehrdad Farajtabar. 2025. The illusion of thinking: Understanding the strengths and limitations of reasoning models via the lens of problem complexity. arXiv preprint arXiv:2506.06941.
  33. David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, and 1 others. 2016. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484–489. find
  34. David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, and 1 others. 2017. Mastering the game of go without human knowledge. nature, 550(7676):354– 359. find
  35. Koustuv Sinha, Shagun Sodhani, and 1 others. 2019. Clutrr: A diagnostic benchmark for inductive reasoning from text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. find
  36. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, an… — Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
  37. Aarohi Srivastava and 1 others. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615.
  38. Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, and 1 others. 2019. Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782):350–354. find
  39. Alex Wang, Yada Pruksachatkun, Nikita Nangia, and 1 others. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In Advances in Neural Information Processing Systems.
  40. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emergent abilities of large language models. Preprint, arXiv:2…
  41. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022b. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, volume 35, pages 24824–24837. Curran Associates, Inc.
  42. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. Preprint, arXiv:2305.10601. find
  43. Xingyu Zheng, Yuye Li, Haoran Chu, Yue Feng, Xudong Ma, Jie Luo, Jinyang Guo, Haotong Qin, Michele Magno, and Xianglong Liu. 2025. An empirical study of qwen3 quantization. arXiv preprint arXiv:2505.02214. — [arXiv:2505.02214](https://arxiv.org/abs/2505.02214) find
  44. Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. Lima: Less is more for alignment. Preprint, arXiv:2305.11206. find
  45. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, and 1 others. 2024. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pamies-2026-vinclat-evaluating-reasoning-cognition-culture/