Elamur Research⌕
☰
home / papers / 2026 / From Wordle to Fibble5: Evaluating LLM Reasoning Under Escalating Deception

From Wordle to Fibble5: Evaluating LLM Reasoning Under Escalating Deception

Paper · 2026 · Chang Liu

Summary

Game-based LLM benchmarks such as Wordle usually give the model honest feedback. Real reasoning tasks, like diagnosis, retrieval or working with other agents, often involve unreliable signals. This paper introduces WordleArenas, a platform that controls how much the feedback lies. It extends Fibble, a Wordle variant with one false tile per row, into Fibble^k. In Fibble^k, exactly k of the five colored tiles in each row are wrong, with k from 0 to 5. The model is told k but not which tiles are false. SHA-256 seeding fixes the lie positions and the false colors, so every game is reproducible.

The author evaluated 41 models from 10 providers, both API endpoints and local Ollama models, across 3,749 games. A live daily arena has also been running since February 2026. Win rates drop sharply as lies increase and fall to almost zero at 2–4 lies. They then partly recover when every tile lies, because the feedback can simply be inverted. Robustness to deception tracks ordinary Wordle skill only weakly. The paper concludes that truthful-feedback tests overstate LLM reasoning and that benchmarks should include deceptive feedback.

Key points

  • Average win rates (models with ≥10 games) for k = 0 to 5 were 41.1%, 18.7%, 2.9%, 0.3%, 0.4% and 9.5%. Fibble arenas allow 8 guesses instead of Wordle's 6.
  • Gemini 3.1 Pro was the most robust model:
    • 95.0% on Wordle, 87.5% on Fibble^1 and 31.6% on Fibble^2.
    • It was the only model with any wins at k = 4 (5.1%).
    • At k = 2, only GLM-5 (28.6%), Qwen3.5 122B (16.7%) and Kimi K2.5 (14.3%) also won any games.
  • Standard skill does not predict robustness. o3 reached 83.3% in standard Wordle yet won no games (0%) in any of the Fibble arenas. Claude Sonnet 4.6 fell from 56.7% to 0% at Fibble^1.
  • Thinking modes (Qwen3, Gemma3) behaved differently by arena:
    • They were 60–130× slower but gave large gains on Wordle.
    • They helped little under partial deception.
    • Under total deception they helped: Qwen3 8B scored 27.3% on Fibble^5 with thinking versus 0% without.
  • Setup and limitations:
    • The word list contains 1,674 five-letter words, and each model played 4–49 games per arena.
    • Usable information drops from 7.92 bits per row (Wordle) to 1.60 bits at k = 3–4. The author therefore recommends 13 guesses for those arenas.
    • Humans informally win about 99% of Wordle games.
    • Code and data are published at drchangliu.github.io/WordleArenas.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ohio UniversityAthens · United States2 works · 310 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Chang Liu. 2026. From Wordle to Fibble5: Evaluating LLM Reasoning Under Escalating Deception. Proceedings of the Workshop on Evaluating Evaluations (EvalEval).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.evaleval-1.5

@inproceedings{liu2026wordle,
  title = {{From Wordle to Fibble5: Evaluating LLM Reasoning Under Escalating Deception}},
  author = {Chang Liu},
  booktitle = {Proceedings of the Workshop on Evaluating Evaluations (EvalEval)},
  year = {2026},
  doi = {10.18653/v1/2026.evaleval-1.5},
  url = {https://doi.org/10.18653/v1/2026.evaleval-1.5}
}

References

10 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Schlangen. 2024. clembench-2024: A challenging, dynamic, complementary, multilingual benchmark and underlying flexible framework for LLMs as multi-action agents. arXiv preprint arXiv:2405.20859. — [arXiv:2405.20859](https://arxiv.org/abs/2405.20859) find
  2. Future Work. A key confound is the guess budget: an information-theoretic analysis (Table 4, Appendix) shows that usable bits per row fall from 7.92 (Wordle) to just 1.60 at k = 3–4, leaving only a +1.3-guess margin above the theoretical minimum. The near-zero win rates at k = 3–4 may partly reflect… find
  3. Filippo Momentè, Alessandro Suglia, Mario Giulianelli, Ambra Ferrari, Alexander Koller, Oliver Lemon, David Schlangen, Raquel Fernández, and Raffaella Bernardi. 2025. Triangulating LLM progress through benchmarks, games, and cognitive tests. In Findings of the Association for Computational Linguisti… find
  4. Suma Bailis, Jane Friedhoff, and Feiyang Chen. 2024. Werewolf Arena: A case study in LLM evaluation via social deduction. arXiv preprint arXiv:2407.13943. — [arXiv:2407.13943](https://arxiv.org/abs/2407.13943) find
  5. Zirui Song, Yuan Huang, Junchang Liu, Haozhe Luo, Chenxi Wang, Lang Gao, Zixiang Xu, Mingfei Han, Xiaojun Chang, and Xiuying Chen. 2025. Beyond survival: Evaluating LLMs in social deduction games with human-aligned strategies. arXiv preprint arXiv:2510.11389. \#\# Conclusion Mrinal Agarwal, Saad Rana,… — [arXiv:2510.11389](https://arxiv.org/abs/2510.11389) find
  6. We presented WordleArenas, a parametric deception benchmark that evaluates 41 models across 3,749 games and 6 deception levels. Our key findings are: (1) LLMs are brittle to deception, with average win rates dropping from 41.1% to 18.7% with just one lie per row; (2) a deception cliff at k = 2–3 whe… find
  7. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does LLM safety training fail? In Advances in Neural Information Processing Systems 36 (NeurIPS 2023).
  8. Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, et al. 2023. PromptRobust: Towards evaluating the robustness of large language models on adversarial prompts. In Proceedings of the 1st ACM Workshop on Large AI Systems and Models with Privacy and Safety Analysis…
  9. Yikang Pan, Liangming Pan, Wenhu Chen, Preslav Nakov, Min-Yen Kan, and William Wang. 2023. On the risk of misinformation pollution with large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 1389–1403.
  10. Krerkkiat Chusap, Colin Murphy, Clay Hess, Sai Deepa Kadaru, Rohit Buccapatnam, and Chang Liu. 2025. RL- and LLM-based AI solvers for the game of Wordle/Fibble. In 2025 IEEE Conference on Games (CoG), pages 1–8. IEEE. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-wordle-fibble-5-evaluating-llm/