Elamur Research⌕
☰
home / papers / 2026 / AlgBench: To What Extent Do Large Reasoning Models Understand Algorithms?

AlgBench: To What Extent Do Large Reasoning Models Understand Algorithms?

Paper · 2026 · Henan Sun, Kaichi Yu, Yuyao Wang et al.

Summary

Existing algorithmic reasoning benchmarks such as CLRS-Text, LeetCodeDataset, LiveCodeBench and LiveCodeBenchPro mostly reuse public contest or textbook tasks. Each task tends to mix several algorithms, covers a narrow algorithm set, and may have leaked into pretraining data. AlgBench addresses these gaps with an "algorithm-centric" design: each problem targets one algorithm, so a failure points to a specific missing skill. Experienced ACM competitors wrote over 3,000 new problems covering 27 algorithms. The problems follow a taxonomy with two axes. One is the optimization idea: none, local, global or heuristic. The other is the data structure: Euclidean or non-Euclidean. Leading reasoning models handle simple, non-optimized algorithms well but fall to roughly half accuracy on global optimization such as dynamic programming. Larger models do not close this gap. Error analysis shows models often drop a correct plan midway, which the authors tie to unavoidable low-entropy tokens. The authors argue that reinforcement learning built around individual problems is insufficient and that training should be organized around algorithms.

Key points

  • AlgBench contains 3,000+ expert-written, contamination-free problems across 27 algorithms, grouped into six categories. Examples include prefix sums, binary search, Tarjan, network flow, greedy, SSSP and A-star.
  • The evaluated models include Gemini-3-Pro, DeepSeek-v3.2-Speciale and GPT-o3. Accuracy reaches about 92% on non-optimized tasks but only about 49% on globally optimized ones.
  • Scaling up parameters helps local-optimization and non-Euclidean tasks. It brings little improvement on global and heuristic optimization, suggesting a plateau.
  • A detailed error taxonomy (shown for Qwen3-8B) separates algorithm-design, execution, instruction-inconsistency and knowledge-gap errors. It identifies "strategic over-shifts" linked to needed low-entropy tokens such as delimiters and numeric constants. The authors propose rewarding such tokens during RL or using tool-assisted agents.
  • Stated limitations: no complex algorithms like Segment Tree or Aho–Corasick, no rigorous difficulty quantification, and too few problems for fine-tuning.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations🎓Beijing Institute of TechnologyBeijing · China34 works · 3,652 citations🎓The Hong Kong University of Science and Technology (Guangzhou)Guangzhou · China53 works · 1,367 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Science Applications · 102 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Henan Sun, Kaichi Yu, Yuyao Wang, Bowen Liu, Xunkai Li, Rong-Hua Li, Nuo Chen and Jia Li. 2026. AlgBench: To What Extent Do Large Reasoning Models Understand Algorithms?. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1885

@inproceedings{sun2026algbench,
  title = {{AlgBench: To What Extent Do Large Reasoning Models Understand Algorithms?}},
  author = {Henan Sun, Kaichi Yu, Yuyao Wang, Bowen Liu, Xunkai Li, Rong-Hua Li, Nuo Chen and Jia Li},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1885},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1885}
}

References

48 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nuo Chen, Hongguang Li, Baoyuan Wang, and Jia Li. 2024a. From good to great: Improving math reasoning with tool-augmented interleaf prompting. In Proceedings of the 2nd Workshop on Natural Language Reasoning and Structured Explanations (@ ACL 2024), pages 64–79. find
  2. Nuo Chen, Yuhan Li, Jianheng Tang, and Jia Li. 2024b. Graphwiz: An instruction-following language model for graph computational problems. In Proceedings of Qiguang Chen, Libo Qin, Jinhao Liu, Dengyun Peng, Jiannan Guan, Peng Wang, Mengkang Hu, Yuhang Zhou, Te Gao, and Wanxiang Che. 2025. Towards rea…
  3. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588. find
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  5. Thomas H Cormen, Charles E Leiserson, Ronald L Rivest, and Clifford Stein. 2022. Introduction to algorithms. MIT press. find
  6. Google DeepMind. 2025. A new era of intelligence with gemini 3. https://blog.google/products/ gemini/gemini-3/. Accessed: Nov 18, 2025. ↗
  7. Elliot Glazer, Ege Erdil, Tamay Besiroglu, Diego Chicharro, Evan Chen, Alex Gunning, Caroline Falkman Olsson, Jean-Stanislas Denain, Anson Ho, Emily de Oliveira Santos, and 1 others. 2024. Frontiermath: A benchmark for evaluating advanced mathematical reasoning in ai. arXiv preprint arXiv:2411.04872… find
  8. Abhimanyu Gulati, Brando Miranda, Eric Chen, Emily Xia, Kai Fronsdal, Bruno de Moraes Dumont, and Sanmi Koyejo. 2024. Putnam-axiom: A functional & static benchmark for measuring higher level mathematical reasoning in llms. In Forty-second International Conference on Machine Learning. find
  9. Jiayan Guo, Lun Du, Hengyu Liu, Mengyu Zhou, Xinyi He, and Shi Han. 2023. Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305.15066. find
  10. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, and 1 others. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting…
  11. Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and 1 others. 2021a. Measuring coding challenge competence with apps. arXiv preprint arXiv:2105.09938.
  12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021b. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  13. Kaixuan Huang, Jiacheng Guo, Zihao Li, Xiang Ji, Jiawei Ge, Wenzhe Li, Yingqing Guo, Tianle Cai, Hui Yuan, Runzhe Wang, and 1 others. 2025. Math-perturb: Benchmarking llms’ math reasoning abilities against hard perturbations. arXiv preprint arXiv:2502.06453. find
  14. Zhen Huang, Zengzhi Wang, Shijie Xia, Xuefeng Li, Haoyang Zou, Ruijie Xu, Run-Ze Fan, Lyumanshan Ye, Ethan Chern, Yixin Ye, and 1 others. 2024. Olympicarena: Benchmarking multi-discipline cognitive reasoning for superintelligent ai. Advances in Neural Information Processing Systems, 37:19209– 19253. find
  15. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.
  16. Rik Koncel-Kedziorski, Subhro Roy, Aida Amini, Nate Kushman, and Hannaneh Hajishirzi. 2016. Mawps: A math word problem repository. In Proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pages 1152–1157. find
  17. Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09664. find
  18. Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. 2023. Taco: Topics in algorithmic code generation dataset. arXiv preprint arXiv:2312.14852. find
  19. Yuhan Li, Peisong Wang, Xiao Zhu, Aochuan Chen, Haiyun Jiang, Deng Cai, Victor W Chan, and Jia Li. 2024. Glbench: A comprehensive benchmark for graph with large language models. Advances in Neural Information Processing Systems, 37:42349– 42368. find
  20. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, and 1 others. 2022. Competition-level code generation with alphacode. Science, 378(6624):1092–1097.
  21. Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, and 1 others. 2025. Deepseek-v3. 2: Pushing the frontier of open large language models. arXiv preprint arXiv:2512.02556.
  22. Larisa Markeeva, Sean McLeish, Borja Ibarz, Wilfried Bounsi, Olga Kozlova, Alex Vitvitskyi, Charles Blundell, Tom Goldstein, Avi Schwarzschild, and Petar Veličković. 2024. The clrs-text algorithmic reasoning language benchmark. arXiv preprint arXiv:2406.04229. find
  23. Owen McLeod. 1995. Aristotle’s method. History of philosophy quarterly, 12(1):1–18. find
  24. Yifan Niu, Miao Peng, Nuo Chen, Yatao Bian, Tingyang Xu, and Jia Li. 2025. Reledit: Evaluating conceptual knowledge editing in language models via relational reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10220– 10238. find
  25. OpenAI. 2025. Learning to reason with llms. https://openai.com/index/ learning-to-reason-with-1lms/. Referenced for LRM capabilities in technical benchmarking. ↗
  26. Miao Peng, Nuo Chen, Zongrui Suo, and Jia Li. 2025a. Rewarding graph reasoning process makes llms more generalized reasoners. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 2257–2268. find
  27. Miao Peng, Nuo Chen, Jianheng Tang, and Jia Li. 2025b. How does misinformation affect large language model behaviors and preferences? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13711–13748. find
  28. Miao Peng, Weizhou Shen, Nuo Chen, Chenliang Li, Ming Yan, and Jia Li. 2026. Incentivizing in-depth reasoning over long contexts with process advantage shaping. arXiv preprint arXiv:2601.12465.
  29. Ivo Petrov, Jasper Dekoninck, Lyuben Baltadzhiev, Maria Drencheva, Kristian Minchev, Mislav Balunović, Nikola Jovanović, and Martin Vechev. 2025. Proof or bluff? evaluating llms on 2025 usa math olympiad. arXiv preprint arXiv:2503.21934. find
  30. Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379–423. find
  31. Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, and 1 others. 2025. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506.10947. find
  32. Jianheng Tang, Qifan Zhang, Yuhan Li, and Jia Li. 2024. Grapharena: Benchmarking large language models on graph computational problems. arXiv e-prints, pages arXiv–2407. find
  33. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, and 1 others. 2025. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939.
  34. Qifan Zhang, Xiaobin Hong, Jianheng Tang, Nuo Chen, Yuhan Li, Wenzhong Li, Jing Tang, and Jia Li. 2025b. Gcoder: Improving large language model for generalized graph reasoning. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, pages 4149–4159. find
  35. Anjiang Wei, Jiannan Cao, Ran Li, Hongyu Chen, Yuhui Zhang, Ziheng Wang, Yuan Liu, Thiago SFX Teixeira, Diyi Yang, Ke Wang, and 1 others. 2025. Equibench: Benchmarking large language models’ understanding of program semantics via equivalence checking. In NeurIPS 2025 Fourth Workshop on Deep Learning… find
  36. Qifan Zhang, Jianhao Ruan, Aochuan Chen, Kang Zeng, Nuo Chen, Jing Tang, and Jia Li. 2026. Exposing weaknesses of large reasoning models through graph algorithm problems. arXiv preprint arXiv:2602.06319. — [arXiv:2602.06319](https://arxiv.org/abs/2602.06319) find
  37. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Sreemanti Dey, and 1 others. 2024. Livebench: A challenging, contamination-limited llm benchmark. arXiv preprint arXiv:2406.19314. find
  38. Yunhui Xia, Wei Shen, Yan Wang, Jason Klein Liu, Huifeng Sun, Siyue Wu, Jian Hu, and Xiaolong Xu. 2025. Leetcodedataset: A temporal dataset for robust evaluation and efficient training of code llms. arXiv preprint arXiv:2504.14655. — [arXiv:2504.14655](https://arxiv.org/abs/2504.14655) find
  39. Beining Xu and Yongming Lu. 2025. Tecp: Token-entropy conformal prediction for llms. Mathematics, 13(20):3351. find
  40. Xin Xu, Jiaxin Zhang, Tianhao Chen, Zitong Chao, Jishan Hu, and Can Yang. 2025. Ugmathbench: A diverse and dynamic benchmark for undergraduate-level mathematical reasoning with large language models. arXiv preprint arXiv:2501.13766. — [arXiv:2501.13766](https://arxiv.org/abs/2501.13766) find
  41. Zihan Zheng, Zerui Cheng, Zeyu Shen, Shang Zhou, Kaiyuan Liu, Hansen He, Dongruixuan Li, Stanley Wei, Hangyi Hao, Jianzhu Yao, and 1 others. 2025. Livecodebench pro: How do olympiad medalists judge llms in competitive programming? arXiv preprint arXiv:2506.11928. — [arXiv:2506.11928](https://arxiv.org/abs/2506.11928) find
  42. A Evaluated LRMs. Our evaluation encompasses 25 state-of-the-art LRMs, including both closed-source commercial models and open-source LRMs. Leveraging AlgBench, we conduct a comprehensive assessment of the algorithmic reasoning capabilities of contemporary LRMs. The evaluated models are listed in Ta… — [arXiv:2502.20868](https://arxiv.org/abs/2502.20868) find
  43. Chuanyue Yu, Kuo Zhao, Yuhan Li, Heng Chang, Mingjian Feng, Xiangzhe Jiang, Yufei Sun, Jia Li, Yuzhi Zhang, Jianxin Li, and 1 others. 2025. Graphrag-r1: Graph retrieval-augmented generation with process-constrained reinforcement learning. arXiv preprint arXiv:2507.23581. — [arXiv:2507.23581](https://arxiv.org/abs/2507.23581) find
  44. Jifan Yu, Xiaozhi Wang, Shangbin Tu, Shulin Cao, Daniel Zhang-Li, Xin Lv, Hao Peng, Zijun Yao, Xiaohan Zhang, Hanming Li, and 1 others. 2023. Kola: Carefully benchmarking world knowledge of large language models. arXiv preprint arXiv:2306.09296. — [arXiv:2306.09296](https://arxiv.org/abs/2306.09296) find
  45. Zike Yuan, Ming Liu, Hui Wang, and Bing Qin. 2024. Gracore: Benchmarking graph comprehension and complex reasoning in large language models. arXiv preprint arXiv:2407.02936.
  46. Qifan Zhang, Nuo Chen, Zehua Li, Miao Peng, Jing Tang, and Jia Li. 2025a. Improving llms’ generalized reasoning abilities by graph problems. arXiv preprint arXiv:2507.17168. find
  47. Experimental Setup Model Name Organization Release Date DeepSeek-v3.2-Speciale gemini-3-pro grok4 gpt-o3 Qwen3-235B DeepSeek-R1 gemini-2.5-pro Claude Opus gpt-oss-120B DeepSeek Google DeepMind xAI OpenAI Alibaba DeepSeek Google DeepMind Anthropic OpenAI December 2025 November 2025 July 2025 April 20… find
  48. Qwen3-4B Qwen3-8B Qwen3-14B Qwen3-32B QwQ-32B DeepSeek-R1-0528-Qwen3-8B DeepSeek-R1-Distill-Qwen-32B OpenThinker2-32B Skyworker-OR1-32B Qwen3-Coder-30B-A3B-Instruct Llama-3.3-70b-instruct gemma3-12B gemma3-27B Llama3-8B Llama3.1-8B Mixtral-8x7B Alibaba Alibaba Alibaba Alibaba Alibaba DeepSeek DeepSe… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-algbench-what-extent-do-large/