Elamur Research⌕
☰
home / papers / 2026 / Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoning

Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoning

Paper · 2026 · Qisheng Su, Shiting Huang, Zhen Fang et al.

Summary

Agents that alternate between LLM reasoning and external tool calls are usually judged on accuracy. When efficiency is reported, it is often reduced to counting tokens or tool invocations. The authors argue that these counts misrepresent real inference latency in Tool-Integrated Reasoning (TIR). Pauses for tool execution can invalidate the KV-Cache, so the context must be prefilled again. Verbose tool outputs also lengthen the context, making every later memory-bound decode step more expensive.

As an alternative, the authors introduce a measure called PTE (Prefill Token Equivalents), which accounts for hardware behaviour by converting decode cost into prefill-token units via a model-specific coefficient γ. In a high-concurrency industrial deployment, PTE tracked wall-clock latency more closely than token counts and kept model rankings stable across hardware profiles.

Applied to thousands of trajectories on five benchmarks, PTE shows that models with near-identical accuracy can differ in cost by orders of magnitude. The authors also catalogue four recurring inefficiency patterns and find that wrong answers tend to come from costlier trajectories. This gives agent builders a cost yardstick closer to actual hardware behaviour.

Key points

  • Formula: For each turn, PTE adds the prefill context tokens to γ × cumulative sequence length × generated tokens, then sums over all turns.
  • γ values: γ depends on active parameters, for example 0.00329 for Qwen2.5-7B-Instruct, 0.00625 for Llama-3.1-8B-Instruct and 0.00068 for DeepSeek-V3.1-Terminus.
  • Benchmarks: MATH500, AIME24, AIME25, SimpleQA and WebInstruct, covering both open-source dense and MoE models.
  • Four inefficiency patterns:
    • Confirmatory Tool Usage: re-checking answers the model already knows.
    • Tool-Mixing: fragmented workflows that switch between tools without need.
    • Lack of Tool Priors: Qwen-2.5 loses accuracy with Python enabled, partly because it forgets to print results.
    • Tool Format Collapse: Tongyi-DeepResearch breaks on renamed tools or changed argument schemas.
  • Cost and correctness: Correct trajectories consistently show lower PTE than incorrect ones. The authors present this as a correlation, not a causal link. The code and a modular TIR framework are released on GitHub (sqs-ustc/tool-reasoning-framework-PTE).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Science and Technology of ChinaHefei · China108 works · 112,698 citations🎓Shanghai Innovation InstituteShanghai · China24 works · 262 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,426 works

Cite

Cite as Qisheng Su, Shiting Huang, Zhen Fang, Ziyan Chen, Zehui Chen and Feng Zhao. 2026. Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.339

@inproceedings{su2026beyond,
  title = {{Beyond Accuracy: Unveiling Inefficiency Patterns in Tool-Integrated Reasoning}},
  author = {Qisheng Su, Shiting Huang, Zhen Fang, Ziyan Chen, Zehui Chen and Feng Zhao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.339},
  url = {https://doi.org/10.18653/v1/2026.acl-long.339}
}

References

51 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kedi Chen, Zhikai Lei, Xu Guo, Xuecheng Wu, Siyuan Zeng, Jianghao Yin, Yinqi Zhang, Qin Chen, Jie Zhou, Liang He, Qipeng Guo, Kai Chen, and Wei Zhang. 2025a. Code-driven number sequence calculation: Enhancing the inductive reasoning abilities of large language models. Preprint, arXiv:2510.14620. — [arXiv:2510.14620](https://arxiv.org/abs/2510.14620) find
  2. Kedi Chen, Dezhao Ruan, Yuhao Dan, Yaoting Wang, Siyu Yan, Xuecheng Wu, Yinqi Zhang, Qin Chen, Jie Zhou, Liang He, Biqing Qi, Linyang Li, Qipeng Guo, Xiaoming Shi, and Wei Zhang. 2025b. A survey of inductive reasoning for large language models. Preprint, arXiv:2510.10182.
  3. Qianben Chen, Jingyi Cao, Jiayu Zhang, Tianrui Qin, Xiaowan Li, King Zhu, Dingfeng Shi, He Zhu, Minghao Liu, Xiaobo Liang, Xin Gui, Ge Zhang, Jian Yang, Yuchen Eleanor Jiang, and Wangchunshu Zhou. 2025c. A 2 fm: An adaptive agent foundation model for tool-aware hybrid reasoning. Preprint, arXiv:2510… — [arXiv:2510.12838](https://arxiv.org/abs/2510.12838) find
  4. Yifei Chen, Guanting Dong, and Zhicheng Dou. 2025d. Toward effective tool-integrated reasoning via self-evolved preference learning. Preprint, arXiv:2509.23285. — [arXiv:2509.23285](https://arxiv.org/abs/2509.23285) find
  5. Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, and 1 others. 2023. T-eval: Evaluating the tool utilization capability step by step. CoRR. find
  6. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  7. Lu Dai, Yijie Xu, Jinhui Ye, Hao Liu, and Hui Xiong. 2025. Seper: Measure retrieval utility through the lens of semantic perplexity reduction. Preprint, arXiv:2503.01478. — [arXiv:2503.01478](https://arxiv.org/abs/2503.01478) find
  8. Guanting Dong, Yifei Chen, Xiaoxi Li, Jiajie Jin, Hongjin Qian, Yutao Zhu, Hangyu Mao, Guorui Zhou, Zhicheng Dou, and Ji-Rong Wen. 2025a. Tool-star: Empowering LLM-brained multi-tool reasoner via reinforcement learning. Preprint, arXiv:2505.16410. — [arXiv:2505.16410](https://arxiv.org/abs/2505.16410) find
  9. Guanting Dong, Hangyu Mao, Kai Ma, Licheng Bao, Yifei Chen, Zhongyuan Wang, Zhongxia Chen, Jiazhen Du, Huiyang Wang, Fuzheng Zhang, Guorui Zhou, Yutao Zhu, Ji-Rong Wen, and Zhicheng Dou. 2025b. Agentic reinforced policy optimization. Preprint, arXiv:2507.19849. — [arXiv:2507.19849](https://arxiv.org/abs/2507.19849) find
  10. Xuanqi Gao, Siyi Xie, Juan Zhai, Shiqing Ma, and Chao Shen. 2025. Mcp-radar: A multi-dimensional benchmark for evaluating tool use capabilities in large language models. Preprint, arXiv:2505.16700. — [arXiv:2505.16700](https://arxiv.org/abs/2505.16700) find
  11. Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlie Huang, Nan Duan, and Weizhu Chen. 2024. Tora: A tool-integrated reasoning agent for mathematical problem solving. Preprint, arXiv:2309.17452.
  12. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. Preprint, arXiv:2103.03874.
  13. Shiting Huang, Zhen Fang, Zehui Chen, Siyu Yuan, Junjie Ye, Yu Zeng, Lin Chen, Qi Mao, and Feng Zhao. 2025. Critictool: Evaluating self-critique capabilities of large language models in tool-calling error scenarios. Preprint, arXiv:2506.13977.
  14. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? Preprint, arXiv:2310.06770.
  15. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. Preprint, arXiv:2001.08361.
  16. Hanchen Li, Qiuyang Mang, Runyuan He, Qizheng Zhang, Huanzhi Mao, Xiaokun Chen, Alvin Cheung, Joseph Gonzalez, and Ion Stoica. 2025a. Continuum: Efficient and robust multi-turn LLM agent scheduling with kv cache time-to-live. Preprint, arXiv:2511.02230. find
  17. Kuan Li, Zhongwang Zhang, Huifeng Yin, Rui Ye, Yida Zhao, Liwen Zhang, Litu Ou, Dingchu Zhang, Xixi Wu, Jialong Wu, Xinyu Wang, Zile Qiao, Zhen Zhang, Yong Jiang, Pengjun Xie, Fei Huang, and Jingren Zhou. 2025b. Websailor-v2: Bridging the chasm to proprietary agents via synthetic data and scalable r… — [arXiv:2509.13305](https://arxiv.org/abs/2509.13305) find
  18. Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, Weizhou Shen, Junkai Zhang, Dingchu Zhang, Xixi Wu, Yong Jiang, Ming Yan, Pengjun Xie, Fei Huang, and Jingren Zhou. 2025c. Websailor: Navigating super-human reasoning for web a… find
  19. Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. Api-bank: A comprehensive benchmark for tool-augmented LLMs. Preprint, arXiv:2304.08244.
  20. Xuefeng Li, Haoyang Zou, and Pengfei Liu. 2025d. ToRL: Scaling tool-integrated RL. Preprint, arXiv:2503.23383. find
  21. Heng Lin and Zhongwen Xu. 2025. Understanding tool-integrated reasoning. Preprint, arXiv:2508.19201. — [arXiv:2508.19201](https://arxiv.org/abs/2508.19201) find
  22. Shuhang Lin, Wenyue Hua, Lingyao Li, Che-Jui Chang, Lizhou Fan, Jianchao Ji, Hang Hua, Mingyu Jin, Jiebo Luo, and Yongfeng Zhang. 2024. Battleagent: Multi-modal dynamic emulation on historical battles to complement historical analysis. Preprint, arXiv:2404.15532. find
  23. Jiayu Liu, Cheng Qian, Zhaochen Su, Qing Zong, Shijue Huang, Bingxiang He, and Yi R. Fung. 2025. Cost-bench: Evaluating multi-turn cost-optimal planning and adaptation in dynamic environments for LLM tool-use agents. Preprint, arXiv:2511.02734. — [arXiv:2511.02734](https://arxiv.org/abs/2511.02734) find
  24. Yi Lu, Jianing Wang, Linsen Guo, Wei He, Hongyin Tang, Tao Gui, Xuanjing Huang, Xuezhi Cao, Wei Wang, and Xunliang Cai. 2025. R-horizon: How far can your large reasoning model really go in breadth and depth? Preprint, arXiv:2510.08189. — [arXiv:2510.08189](https://arxiv.org/abs/2510.08189) find
  25. Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. 2025. General-reasoner: Advancing LLM reasoning across all domains. Preprint, arXiv:2505.14652.
  26. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. Preprint, arXiv:2311.12983.
  27. Zaifeng Pan, Ajjkumar Patel, Zhengding Hu, Yipeng Shen, Yue Guan, Wan-Lu Li, Lianhui Qin, Yida Wang, and Yufei Ding. 2025. Kvflow: Efficient prefix caching for accelerating LLM-based multi-agent workflows. Preprint, arXiv:2507.07400. find
  28. Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfcl): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning. find
  29. Han Peng, Peiyu Liu, Zican Dong, Daixuan Cheng, Junyi Li, Yiru Tang, Shuo Wang, and Wayne Xin Zhao. 2025. How efficient are diffusion language models? a critical examination of efficiency evaluation practices. Preprint, arXiv:2510.18480. — [arXiv:2510.18480](https://arxiv.org/abs/2510.18480) find
  30. Cheng Qian, Emre Can Acikgoz, Hongru Wang, Xiusi Chen, Avirup Sil, Dilek Hakkani-Tür, Gokhan Tur, and Heng Ji. 2025. Smart: Self-aware agent for tool overuse mitigation. Preprint, arXiv:2502.11435. find
  31. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. ToolLLM: Facilitating large language models to master 16000+ real…
  32. Joykirat Singh, Raghav Magazine, Yash Pandya, and Akshay Nambi. 2025. Agentic reasoning and tool integration for LLMs via reinforcement learning. Preprint, arXiv:2505.01441. — [arXiv:2505.01441](https://arxiv.org/abs/2505.01441) find
  33. Tongyi DeepResearch Team. 2025. Tongyi deepresearch technical report. Preprint, arXiv:2510.24701. — [arXiv:2510.24701](https://arxiv.org/abs/2510.24701) find
  34. Hongru Wang, Cheng Qian, Wanjun Zhong, Xiusi Chen, Jiahao Qiu, Shijue Huang, Bowen Jin, Mengdi Wang, Kam-Fai Wong, and Heng Ji. 2025a. Acting less is reasoning more! teaching model to act efficiently. Preprint, arXiv:2504.14870. — [arXiv:2504.14870](https://arxiv.org/abs/2504.14870) find
  35. Ke Wang, Houxing Ren, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2023. Mathcoder: Seamless code integration in LLMs for enhanced mathematical reasoning. Preprint, arXiv:2310.03731.
  36. Ningning Wang, Xavier Hu, Pai Liu, He Zhu, Yue Hou, Heyuan Huang, Shengyu Zhang, Jian Yang, Jiaheng Liu, Ge Zhang, Changwang Zhang, Jun Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou. 2025b. Efficient agents: Building effective agents while reducing cost. Preprint, arXiv:2508.02694. — [arXiv:2508.02694](https://arxiv.org/abs/2508.02694) find
  37. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. Measuring short-form factuality in large language models. Preprint, arXiv:2411.04368.
  38. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. 2025a. Browsecomp: A simple yet challenging benchmark for browsing agents. Preprint, arXiv:2504.12516. find
  39. Yifan Wei, Xiaoyan Yu, Yixuan Weng, Tengfei Pan, Angsheng Li, and Li Du. 2025b. Autotir: Autonomous tools integrated reasoning via reinforcement learning. Preprint, arXiv:2507.21836. — [arXiv:2507.21836](https://arxiv.org/abs/2507.21836) find
  40. Samuel Williams, Andrew Waterman, and David Patterson. 2009. Roofline: an insightful visual performance model for multicore architectures. Commun. ACM, 52(4):65–76. find
  41. Ryan Wong, Jiawei Wang, Junjie Zhao, Li Chen, Yan Gao, Long Zhang, Xuan Zhou, Zuo Wang, Kai Xiang, Ge Zhang, Wenhao Huang, Yang Wang, and Ke Wang. 2025. Widesearch: Benchmarking agentic broad info-seeking. Preprint, arXiv:2508.07999.
  42. Dian Yu, Yulai Zhao, Kishan Panaganti, Linfeng Song, Haitao Mi, and Dong Yu. 2025a. Every question has its own value: Reinforcement learning with explicit human values. Preprint, arXiv:2510.20187. — [arXiv:2510.20187](https://arxiv.org/abs/2510.20187) find
  43. Yuanqing Yu, Zhefan Wang, Weizhi Ma, Shuai Wang, Chuhan Wu, Zhiqiang Guo, and Min Zhang. 2025b. Steptool: Enhancing multi-step tool usage in LLMs via step-grained reinforcement learning. In Proceedings of the 34th ACM International Conference on Information and Knowledge Management, CIKM ’25, page 3…
  44. Zhihang Yuan, Yuzhang Shang, Yang Zhou, Zhen Dong, Zhe Zhou, Chenhao Xue, Bingzhe Wu, Zhikai Li, Qingyi Gu, Yong Jae Lee, Yan Yan, Beidi Chen, Guangyu Sun, and Kurt Keutzer. 2024. LLM inference unveiled: Survey and roofline model insights. Preprint, arXiv:2402.16363. find
  45. Michael J. Zellinger and Matt Thomson. 2025. Economic evaluation of LLMs. Preprint, arXiv:2507.03834. — [arXiv:2507.03834](https://arxiv.org/abs/2507.03834) find
  46. Yufeng Zhao, Junnan Liu, Hongwei Liu, Dongsheng Zhu, Yuan Shen, Songyang Zhang, and Kai Chen. 2025. Dissecting tool-integrated reasoning: An empirical study and analysis. Preprint, arXiv:2508.15754. — [arXiv:2508.15754](https://arxiv.org/abs/2508.15754) find
  47. Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. 2024. {DistServe}: Disaggregating prefill and decoding for goodput-optimized large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 193–210… find
  48. Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. Toolqa: A dataset for LLM question answering with external tools. Preprint, arXiv:2306.13304.
  49. Yang Xiao, Mohan Jiang, Jie Sun, Keyu Li, Jifan Lin, Yumin Zhuang, Ji Zeng, Shijie Xia, Qishuo Hua, Xuefeng Li, Xiaojie Cai, Tongyu Wang, Yue Zhang, Liming Liu, Xia Wu, Jinlong Hou, Yuan Cheng, Wenjie Li, Xiang Wang, and 2 others. 2025. Limi: Less is more for agency. Preprint, arXiv:2509.17567. — [arXiv:2509.17567](https://arxiv.org/abs/2509.17567) find
  50. Frank F. Xu, Yufan Song, Boxuan Li, Yuxuan Tang, Kritanjali Jain, Mengxue Bao, Zora Z. Wang, Xuhui Zhou, Zhitong Guo, Murong Cao, Mingyang Yang, Hao Yang Lu, Amaad Martin, Zhe Su, Leander Maben, Raj Mehta, Wayne Chi, Lawrence Jang, Yiqing Xie, and 2 others. 2025. Theagentcompany: Benchmarking LLM ag… find
  51. Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35:20744–20757.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/su-2026-beyond-accuracy-unveiling-inefficiency-patterns/