Elamur Research⌕
☰
home / papers / 2025 / IOLBENCH: Benchmarking LLMs on Linguistic Reasoning

IOLBENCH: Benchmarking LLMs on Linguistic Reasoning

Paper · 2025 · Satyam Goyal and Soham Dan

Summary

Many LLM benchmarks make it hard to tell whether a model is actually reasoning or just recalling material seen during pre-training. IOLBENCH addresses this by turning problems from the International Linguistics Olympiad (IOL) into an evaluation set. Each puzzle centres on a low-resource, typologically diverse language and contains all the information needed to solve it, so prior knowledge gives little advantage. The authors digitized the IOL archive for 2004–2023 and kept problems with visual content as a separate multimodal split. They paired every item with its official solution and tested leading proprietary models (GPT-4 family, o1, Claude 4, Gemini 2.5, Grok-4) zero-shot. Scoring combines exact string matching, BLEU with an LLM judge, and manual expert grading. Even the strongest systems achieved low overall scores. A qualitative error analysis sorts failures into three dimensions: inducing rules from few examples, combining several rules into one consistent system, and handling irregular real-language complexity. The authors argue these weaknesses also matter for agents that must learn unfamiliar rules from sparse evidence.

Key points

  • About 1,500 problem instances were digitized from IOL PDFs (2004–2023). The released benchmark has 1198 text-only and 52 multimodal problems, along with code and zero-shot prompt templates.
  • The text-only problems fall into three answer types. There are 666 short-answer items (string match), 501 long-answer items such as translations (BLEU plus an LLM judge, 0–2 scale), and 31 explanatory items graded by an expert. A second grader reached Cohen's κ = 0.758 on the explanatory items.
  • Models evaluated: GPT-4, GPT-4o, GPT-4o mini, o1, Claude 4 Sonnet/Opus, Grok-4, Gemini 2.5 Pro/Flash. All were tested with zero-shot prompting only.
  • Illustrative failures include the following:
    • Claude 4 Opus segmented Guaraní verb morphemes correctly but wrongly stacked two future markers.
    • Grok-4 misread multi-part Icelandic matronymics in a family-tree puzzle.
  • Stated limitations: open-source models were not evaluated, and few-shot prompting and fine-tuning were not tested. The multimodal portion is also small.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓University of MichiganAnn Arbor · United States70 works · 13,693 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Satyam Goyal and Soham Dan. 2025. IOLBENCH: Benchmarking LLMs on Linguistic Reasoning. arXiv:2501.04249.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.04249 · arXiv 2501.04249

@misc{goyal2025iolbench,
  title = {{IOLBENCH: Benchmarking LLMs on Linguistic Reasoning}},
  author = {Satyam Goyal and Soham Dan},
  year = {2025},
  doi = {10.48550/arxiv.2501.04249},
  eprint = {2501.04249},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.04249}
}

References

11 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4, 2023. URL https:/…
  2. Nick Chater, Joshua B Tenenbaum, and Alan Yuille. Probabilistic models of cognition: Conceptual foundations. Trends in cognitive sciences, 10(7):287–291, 2006. find
  3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https://arxiv.org/ abs/2110.14168. ↗
  4. Jerry A Fodor and Zenon W Pylyshyn. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2):3–71, 1988. find
  5. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021. URL https://arxiv.org/abs/2009.03300. ↗
  6. Philip Nicholas Johnson-Laird. Mental models: Towards a cognitive science of language, inference, and consciousness. Number 6. Harvard University Press, 1983. find
  7. Najoung Kim and Tal Linzen. COGS: A compositional generalization challenge based on semantic interpretation. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9087–9105, Online, Novemb… find
  8. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openreview.… ↗
  9. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le, Ed Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. In Findings of the Association for Computational Linguistics: ACL 2023, pages…
  10. Petar Veličković, Adrià Puigdomènech Badia, David Budden, Razvan Pascanu, Andrea Banino, Misha Dashevskiy, Raia Hadsell, and Charles Blundell. The clrs algorithmic reasoning benchmark. In International Conference on Machine Learning, pages 22084–22102. PMLR, 2022. find
  11. Xiaoxuan Wang, Ziniu Hu, Pan Lu, Yanqiao Zhu, Jieyu Zhang, Satyen Subramaniam, Arjun R Loomba, Shichang Zhang, Yizhou Sun, and Wei Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models. In Forty-first International Conference on Machine Learning.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/goyal-2025-iolbench-benchmarking-llms-linguistic-reasoning/