Elamur Research⌕
☰
home / papers / 2026 / Cards Against Contamination: TCG-Bench for Difficulty-Scalable Multilingual LLM Reasoning

Cards Against Contamination: TCG-Bench for Difficulty-Scalable Multilingual LLM Reasoning

Paper · 2026 · Sultan AlRashed, Jianghui Wang and Francesco Orabona

Summary

Game benchmarks for LLMs usually rely on well-known games such as chess, whose strategies are widely documented online. This makes it hard to tell memorization apart from genuine reasoning. TCG-Bench is a new two-player text trading card game built to address this. Its rule engine is open source, while the 30 card implementations used for scoring stay private and can be swapped out. Opponent strength is set by the rollout depth of a Monte Carlo player, which gives a continuous difficulty scale. The game exists in parallel English and Arabic versions. The authors also define a threat model and a procedure for refreshing the cards if they leak.

They tested 17 models over more than 50,000 games. Key findings:

  • Win rates drop exponentially as difficulty rises.
  • Parameter count predicts strategic skill only weakly.
  • The gap between English and Arabic grows with model scale.
  • Even the strongest models fall to almost zero against deep rollouts.

The benchmark is therefore far from saturated, and memorizing the public rules does not reveal the hidden content.

Key points

  • Game design:
    • Players start with 10 life points.
    • A turn proceeds through three phases: Draw, then Main, then Combat.
    • Cards are Champions, Spells or Tricks.
    • A game has about 3.5 legal moves per turn over roughly 12 turns, about 1.4×10⁶ nodes.
    • Submissions are scored on a leaderboard (tcg-bench.com).
  • Difficulty scaling: the exponential decay fit gives R² > 0.93, and the best models collapse at rollout-1000. Model size correlates with win rate at only r = 0.31.
  • Language gap: the English–Arabic gap reaches 47.4% for Qwen3-32B.
  • Contamination evidence:
    • Fine-tuning on only 2,000 game trajectories raised win rate from 16% to 30%.
    • The authors found Zork and other Jericho games in C4, using a fuzzy Elastic Search over the corpus.
    • They found traces of the TextWorld repository in The Stack.
  • Error analysis:
    • Late losses account for 59.9% of failures; models often miss lethal moves.
    • Games where no Tricks were played (19.2%) end early.
    • Thinking mode in Gemini 2.5 Flash helps against weak opponents (52.4% vs 46.0%) but loses that advantage, or falls behind, at higher difficulty.

Interesting ideas

  • I1. Separate a public, inspectable harness from a private, modular content pack that can be swapped cheaply when it leaks; scores stay comparable over time because the engine never changes.
  • I2. Expose difficulty as one numeric knob and report a decay curve with a fitted trend instead of a single accuracy number, so the benchmark cannot saturate and "leagues" of comparable strength become possible.
  • I3. Prove that contamination matters by deliberately fine-tuning on a small amount of related, non-test data and measuring the inflation.
  • I4. Build truly parallel versions of the same task in two languages, so that gaps reflect reasoning in that language rather than differences in content.
  • I5. Classify failures structurally (late collapses, entire action types never used) and break them down by system family, rather than only counting wins.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): keep the evaluation harness open but hold a private, rotatable pool of fables and word problems, so self-play and LLM-teacher data can never quietly absorb the test set.
  • I2 (answer): generate maths word problems with a controllable parameter (number of reasoning steps, distractor quantities) and track how accuracy falls along it.
  • I3 (learn): add this as a negative control, training on near-duplicate items to see how much a score can be inflated without real understanding.
  • I4 (back): render the same meta-world into parallel English and Ukrainian texts and check that both parse back to identical world graphs.
  • I5 (answer): tag wrong answers by which frame or world step was skipped, such as an unused holding change or a missed event ordering, to show where the pipeline's coverage is narrow.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓King Abdullah University of Science and TechnologyJeddah · Saudi Arabia27 works · 1,084 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Sultan AlRashed, Jianghui Wang and Francesco Orabona. 2026. Cards Against Contamination: TCG-Bench for Difficulty-Scalable Multilingual LLM Reasoning. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.353

@inproceedings{alrashed2026cards,
  title = {{Cards Against Contamination: TCG-Bench for Difficulty-Scalable Multilingual LLM Reasoning}},
  author = {Sultan AlRashed, Jianghui Wang and Francesco Orabona},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.353},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.353}
}

References

31 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, Cyril Zakka, Mathieu Morlon, and 3 others. 2025. SmolLM2: When smol goes big – data-centric training of a small language model. Preprint, arXiv:2502.02737. ↗
  2. Anton Bakhtin, David J Wu, Adam Lerer, Jonathan Gray, Athul Paul Jacob, Gabriele Farina, Alexander H Miller, and Noam Brown. 2022. Mastering the game of no-press diplomacy via human-regularized reinforcement learning and planning. Preprint, arXiv:2210.05492. ↗
  3. Elie Bakouch, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Lewis Tunstall, Carlos Miguel Patiño, Edward Beeching, Aymeric Roucher, Aksel Joonas Reedi, Quentin Gallouédec, Kashif Rasul, Nathan Habib, Clémentine Fourrier, Hynek Kydlicek, Guilherme Penedo, Hugo Larcher, Mathieu Morlon, Vaibhav Srivastav, Joshua Lochner, and 4 others. 2025. SmolLM3: smol, multilingual, long-context reasoner. https://huggingface.co/blog/smollm3. ↗
  4. Anne Beyer, Kranti Chalamalasetti, Sherzod Hakimov, Brielen Madureira, Philipp Sadler, and David Schlangen. 2024. clembench-2024: A challenging, dynamic, complementary, multilingual benchmark and underlying flexible framework for llms as multi-action agents. Preprint, arXiv:2405.20859. ↗
  5. Mirelle Candida Bueno, Roberto Lotufo, and Rodrigo Frassetto Nogueira. 2024. MLissard: Multilingual long and simple sequential reasoning benchmarks. In Proceedings of the 2nd GenBench Workshop on Generalisation (Benchmarking) in NLP, pages 86–95, Miami, Florida, USA. Association for Computational Linguistics. find
  6. Guillaume Chaslot, Sander Bakkes, Istvan Szita, and Pieter Spronck. 2008. Monte-carlo tree search: A new framework for game AI. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 4, pages 216– 217. find
  7. Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer, and Arjun Yadav. 2024. GameBench: Evaluating strategic reasoning abilities of LLM agents. Preprint, arXiv:2406.06613. ↗
  8. Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, and Adam Trischler. 2019. Interactive fiction games: A colossal adventure. Preprint, arXiv:1909.05398. ↗
  9. Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Ruo Yu Tao, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. 2018. TextWorld: A learning environment for text-based games. In Workshop on Computer Games at the 27th International Joint Conference on Artificial Intelligence (IJCAI).
  10. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8706–8719, Mexico City, Mexico. Association for Computational Linguistics. ↗
  11. Leon Guertler, Bobby Cheng, Simon Yu, Bo Liu, Leshem Choshen, and Cheston Tan. 2025. TextArena. Preprint, arXiv:2504.11442. ↗
  12. Hongyi Guo, Jiayang Wu, and Weiqin Wang. 2024. Can large language models play games? a case study of a self-play approach. Preprint, arXiv:2403.05632. ↗
  13. Alon Jacovi, Avi Caciularu, Omer Goldman, and Yoav Goldberg. 2023. Stop uploading test data in plain text: Practical strategies for mitigating data contamination by evaluation benchmarks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5075–5084, Singapore. Association for Computational Linguistics. find
  14. Denis Kocetkov, Raymond Li, Loubna Ben Allal, Jia Li, Chenghao Mou, Carlos Muñoz Ferrandis, Yacine Jernite, Margaret Mitchell, Sean Hughes, Thomas Wolf, Dzmitry Bahdanau, Leandro von Werra, and Harm de Vries. 2022. The Stack: 3 TB of permissively licensed source code. Preprint, arXiv:2211.15533. ↗
  15. Levente Kocsis and Csaba Szepesvári. 2006. Bandit based monte-carlo planning. In European conference on machine learning, pages 282–293. Springer. find
  16. Huiyuan Lai and Malvina Nissim. 2024. mCoT: Multilingual instruction tuning for reasoning consistency in language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12012– 12026, Bangkok, Thailand. Association for Computational Linguistics.
  17. Yucheng Li, Frank Guerin, and Chenghua Lin. 2024. LatestEval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction. Preprint, arXiv:2312.12343. ↗
  18. Wenye Lin, Jonathan Roberts, Yunhan Yang, Samuel Albanie, Zongqing Lu, and Kai Han. 2025. GAME-BoT: Transparent assessment of LLM reasoning in games. Preprint, arXiv:2412.13602. ↗
  19. Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. 2024. Quantifying variance in evaluation benchmarks. Preprint, arXiv:2406.10229. ↗
  20. OpenAI. 2023. GPT-4 technical report. Technical report, OpenAI. ↗
  21. Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1–22. find
  22. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. Preprint, arXiv:2202.03286. ↗
  23. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67. ↗
  24. Anian Ruoss, Grégoire Delétang, Sourabh Medapati, Jordi Grau-Moya, Li Kevin Wenliang, Elliot Catt, John Reid, and Tim Genewein. 2024. Grandmaster-level chess without search. Preprint, arXiv:2402.04494. ↗
  25. Yusuke Sakai, Hidetaka Kamigaito, and Taro Watanabe. 2024. mCSQA: Multilingual commonsense reasoning dataset with unified creation strategy by language models and humans. In Findings of the Association for Computational Linguistics: ACL 2024, pages 14182–14214, Bangkok, Thailand. Association for Computational Linguistics. find
  26. Václav Voráček. 2024. Treatment of statistical estimation problems in randomized smoothing for adversarial robustness. In Advances in Neural Information Processing Systems, volume 37, pages 133464– 133486. Curran Associates, Inc. find
  27. Hongyu Wang, Jiayu Xu, Senwei Xie, Ruiping Wang, Jialin Li, Zhaojie Xie, Bin Zhang, Chuyan Xiong, and Xilin Chen. 2024. M4U: Evaluating multilingual understanding and reasoning for large multimodal models. Preprint, arXiv:2405.15638. ↗
  28. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Benjamin Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, Shubh-Agrawal, Sandeep Singh Sandha, Siddartha Venkat Naidu, Chinmay Hegde, Yann LeCun, Tom Goldstein, Willie Neiswanger, and Micah Goldblum. 2025. LiveBench: A challenging, contamination-limited LLM benchmark. In The Thirteenth International Conference on Learning Representations. find
  29. Daijin Yang, Erica Kleinman, and Casper Harteveld. 2024. GPT for games: A scoping review (2020– 2023). Preprint, arXiv:2404.17794. ↗
  30. Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E Gonzalez, and Ion Stoica. 2023. Rethinking benchmark and contamination for language models with rephrased samples. Preprint, arXiv:2311.04850. ↗
  31. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. In Advances in Neural Information Processing Systems, volume 36. *Figure 10: Extended model comparison across rollout depths.* *Table 8: TCG-Bench Full Results with 95% Confidence Intervals* Organization Model Language Type Meta LLaMA LLaMA LLaMA EN EN EN 3.1-8B 3.2-11B 3.3-70B 28.8 30.0 45.0 [25.1%, 32.5%] [26.4%, 33.8%] [41.0%, 49.1%] – 14.4 30.4 – [11.6%, 17.4%] [26.7%, 34.2%] – 9.6 16.4 – [7.3%, 12.1%] [13.5%, 19.5%] – – – – – – Qwen Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 Qwen3 EN EN EN EN EN AR AR AR AR AR 4B 8B 30B 32B 235B 4B 8B 30B 32B 235B – – – 45.0 30.2 – – – – – – – – [41.0%, 49.1%] [26.5%, 34.0%] – – – – – 19.3 – 19.3 32.0 15.4 15.9 14.9 – – – [16.1%, 22.5%] – [16.1%, 22.5%] [28.3%, 35.9%] [12.5%, 18.5%] [13.0%, 19.0%] [12.1%, 17.9%] – – – 10.0 8.8 12.5 17.5 11.1 9.5 8.8 9.3 17.5 6.5 [7.7%, 12.7%] [6.5%, 11.2%] [10.0%, 15.4%] [14.3%, 20.7%] [8.6%, 13.5%] [7.3%, 12.1%] [6.5%, 11.2%] [7.0%, 11.8%] [14.3%, 20.7%] [4.7%, 8.8%] – – 7.80 13.3 6.9 – – 5.8 – – – – [5.7%, 10.1%] [10.6%, 16.1%] [4.7%, 8.8%] – – [4.0%, 7.8%] – – Google Gemini Gemini EN EN 2.5-Flash-Preview 2.5-Flash-Preview:thinking 46.0 52.4 [42.0%, 50.1%] [48.3%, 56.4%] 36.6 29.8 [32.6%, 40.5%] [26.0%, 33.5%] 17.2 11.0 [14.2%, 20.4%] [8.6%, 13.8%] 15.0 – [12.2%, 18.1%] – DeepSeek DeepSeek Grok EN EN EN R1-Distil-8B R1-Distil-70B 3-Mini 28.2 38.3 53.4 [25.0%, 31.4%] [34.4%, 42.3%] [49.3%, 57.4%] – 24.6 18.2 – [21.1%, 28.1%] [15.2%, 21.5%] – 12.4 10.2 – [9.7%, 15.1%] [7.9%, 12.9%] – – – – – – DeepSeek x-AI Rollout 1 (%) CI Rollout 2 (%) CI Rollout 5 (%) CI Rollout 10 (%) CI

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alrashed-2026-cards-against-contamination-tcg-bench/