Elamur Research⌕
☰
home / papers / 2025 / TRAIL: Trace Reasoning and Agentic Issue Localization

TRAIL: Trace Reasoning and Agentic Issue Localization

Paper · 2025 · Darshan Deshpande, Varun Gangal, Hersh Mehta et al.

Summary

Agentic LLM systems produce long, structured execution logs, and diagnosing what went wrong inside them is still mostly done by hand by domain experts. This does not scale. The authors from Patronus AI propose a fine-grained error taxonomy for agents, grouped into reasoning, planning and execution failures. The taxonomy covers issues such as hallucinations, goal deviation, tool selection mistakes, API problems and task orchestration errors. Using it, they built TRAIL, a benchmark of expert-annotated OpenTelemetry traces. The traces come from single- and multi-agent runs on GAIA (open-world information retrieval) and SWE-Bench (software engineering). Each error is labeled with its category, the span where it occurs, supporting evidence and an impact level. When strong long-context LLMs act as judges that must find and classify these errors, they do poorly. Some cannot even fit a full trace into their context window. Models that reason more help somewhat. The results show that automated, scalable debugging of agent workflows remains an open problem, and TRAIL gives a realistic testbed for it.

Key points

  • TRAIL contains 148 traces with 1987 OpenTelemetry spans. Of these spans, 575 contain at least one error, and the dataset has 841 annotated errors in total. It is released under the MIT License on HuggingFace (PatronusAI/TRAIL).
  • The best model, GEMINI-2.5-PRO, reaches only about 11% joint accuracy (category plus location) across both splits. Its accuracy is 18% on GAIA and 5% on SWE-Bench.
  • Three of the eight evaluated models could not process the full trace context. Solving the task also demands a large share of models' maximum output length.
  • Results vary widely by category:
    • Context Handling Failures get roughly 0.00 F1 from almost every model, with CLAUDE-3.7-SONNET as the exception at 0.18.
    • For Tool Selection Errors, O3 leads with 0.53.
    • Language-only hallucinations are comparatively easy for all models.
  • Annotators were computer-science experts paid $12.66 per trace, with each trace taking 30–40 minutes. Stated limitations include text-only scope and many rare tail categories.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksInformation Systems · 1,888 worksSoftware · 150 worksComputer Networks and Communications · 115 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan and Rebecca Qian. 2025. TRAIL: Trace Reasoning and Agentic Issue Localization. arXiv:2505.08638.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.08638 · arXiv 2505.08638

@misc{deshpande2025trail,
  title = {{TRAIL: Trace Reasoning and Agentic Issue Localization}},
  author = {Darshan Deshpande, Varun Gangal, Hersh Mehta, Jitin Krishnan, Anand Kannappan and Rebecca Qian},
  year = {2025},
  doi = {10.48550/arxiv.2505.08638},
  eprint = {2505.08638},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.08638}
}

References

112 references, 43 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Siraaj Akhtar, Saad Khan, and Simon Parkinson. 2025. Llm-based event log analysis techniques: A survey. arXiv preprint arXiv:2502.00677. — [arXiv:2502.00677](https://arxiv.org/abs/2502.00677) find
  2. Reem Aleithan, Haoran Xue, Mohammad Mahdi Mohajer, Elijah Nnorom, Gias Uddin, and Song Wang. 2024. Swe-bench+: Enhanced coding benchmark for llms. arXiv preprint arXiv:2410.06992.
  3. Anthropic. 2025. Claude 3.7 sonnet. https://www.anthropic.com/news/claude-3-7-sonnet. Accessed: May 9, 2025. ↗
  4. Anthropic. 2025. Model context protocol: Transparency and control for ai inputs and outputs. Accessed: 2025-05-08. find
  5. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal A. Patwardhan, Lil ian Weng, and Aleksander Mkadry. 2024. Mle-bench: Evaluating machine learning agents on machine learning engineering. ArXiv, abs/2410.07095.
  6. Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024. Mllm-asa-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Proceedings of the 41st International Conference on Machine Learning (ICML). find
  7. Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. 2025. Judgelrm: Large reasoning models as a judge. find
  8. Anthony Costarelli, Mat Allen, Roman Hauksson, Grace Sodunke, Suhas Hariharan, Carlson Cheng, Wenjie Li, Joshua Clymer, and Arjun Yadav. 2024. Gamebench: Evaluating strategic reasoning abilities of llm agents. arXiv preprint arXiv:2406.06613.
  9. Google DeepMind. 2025. Gemini model thinking updates: March 2025. https://blog.google/technology/google-deepmind/ gemini-model-thinking-updates-march-2025/. Accessed: May 11, 2025. ↗
  10. Darshan Deshpande, Selvan Sunitha Ravi, Sky CH Wang, Bartosz Mielczarek, Anand Kannappan, and Rebecca Qian. 2024a. Glider: Grading llm interactions and decisions using explainable ranking. arXiv preprint arXiv:2412.14140. — [arXiv:2412.14140](https://arxiv.org/abs/2412.14140) find
  11. Darshan Deshpande, Zhivar Sourati, Filip Ilievski, and Fred Morstatter. 2024b. Contextualizing argument quality assessment with relevant knowledge. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volum… find
  12. Xueying Du, Mingwei Liu, Kaixin Wang, Hanlin Wang, Junwei Liu, Yixuan Chen, Jiayi Feng, Chaofeng Sha, Xin Peng, and Yiling Lou. 2024. Evaluating large language models in class-level code generation. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, ICSE ’24, New Y… find
  13. Will Epperson, Gagan Bansal, Victor Dibia, Adam Fourney, Jack Gerrits, Erkang Zhu, and Saleema Amershi. 2025. Interactive debugging and steering of multi-agent ai systems. arXiv preprint arXiv:2503.02068. — [arXiv:2503.02068](https://arxiv.org/abs/2503.02068) find
  14. Ficlive. 2025. Fiction.livebench (april 6, 2025). https://fiction.live/ stories/Fiction-livebench-April6-2025/ oQdzQvKHw8JyXbN87. Accessed: 2025-05-12. ↗
  15. Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 61–71, Online. Association for Computational Linguistics.
  16. Xiaohan Fu, Shuheng Li, Zihan Wang, Yihao Liu, Rajesh K Gupta, Taylor Berg-Kirkpatrick, and Ear-lence Fernandes. 2024. Imprompter: Tricking llm agents into improper tool use. arXiv preprint arXiv:2410.14923. — [arXiv:2410.14923](https://arxiv.org/abs/2410.14923) find
  17. Yunfan Gao, Yun Xiong, Yijie Zhong, Yuxi Bi, Ming Xue, and Haofen Wang. 2025. Synergizing rag and reasoning: A systematic review. find
  18. Yingqiang Ge, Yujie Ren, Wenyue Hua, Shuyuan Xu, Juntao Tan, and Yongfeng Zhang. 2023. Llm as os, agents as apps: Envisioning aios, agents and the aios-agent ecosystem. arXiv preprint arXiv:2312.03815. — [arXiv:2312.03815](https://arxiv.org/abs/2312.03815) find
  19. Jiayan Guo, Lun Du, Hengyu Liu, Mengyu Zhou, Xinyi He, and Shi Han. 2023. Gpt4graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking. arXiv preprint arXiv:2305.15066.
  20. Michael Hanna and Ondřej Bojar. 2021. A fine-grained analysis of BERTScore. In Proceedings of the Sixth Conference on Machine Translation, pages 507–517, Online. Association for Computational Linguistics.
  21. Juyeon Heo, Miao Xiong, Christina Heinze-Deml, and Jaya Narain. 2024. Do llms estimate uncertainty well in instruction-following? arXiv preprint arXiv:2410.14582. — [arXiv:2410.14582](https://arxiv.org/abs/2410.14582) find
  22. Mengkang Hu, Pu Zhao, Can Xu, Qingfeng Sun, Jian-guang Lou, Qingwei Lin, Ping Luo, and Saravan Rajmohan. 2024a. Agentgen: Enhancing planning abilities for large language model based agent via environment and task generation. arXiv preprint arXiv:2408.00764. — [arXiv:2408.00764](https://arxiv.org/abs/2408.00764) find
  23. Yucheng Hu and Yuxing Lu. 2024. Rag and rau: A survey on retrieval-augmented language model in natural language processing. find
  24. Zhengyu Hu, Linxin Song, Jieyu Zhang, Zheyuan Xiao, Jingang Wang, Zhenyu Chen, Jieyu Zhao, and Hui Xiong. 2024b. Rethinking llm-based preference evaluation. arXiv e-prints, pages arXiv–2407. find
  25. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2).
  26. Hugging Face. 2024. open deep research: An open-source replication of openai’s deep research agent. https://github.com/huggingface/ smolagents/tree/main/examples/open_deep_ research. ↗
  27. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12).
  28. Zhihan Jiang, Junjie Huang, Zhuangbin Chen, Yichen Li, Guangba Yu, Cong Feng, Yongqiang Yang, Zengyin Yang, and Michael R. Lyu. 2025. L4: Diagnosing large-scale llm training failures via automated log analysis. arXiv preprint arXiv:2503.20263. — [arXiv:2503.20263](https://arxiv.org/abs/2503.20263) find
  29. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.
  30. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. 2024a. Ai agents that matter. arXiv preprint arXiv:2407.01502.
  31. Sayash Kapoor, Benedikt Stroebl, Zachary S. Siegel, Nitya Nadgir, and Arvind Narayanan. 2024b. Ai agents that matter. arXiv preprint arXiv:2407.01502.
  32. Marzena Karpinska, Katherine Thai, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. One thousand and one pairs: A" novel" challenge for long-context language models. arXiv preprint arXiv:2406.16264. find
  33. Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, et al. 2025. A survey of frontiers in llm reasoning: Inference scaling, learning to reason, and agentic systems. arXiv preprint arXiv:2504.09037.
  34. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. arXiv preprint arXiv:2310.08491.
  35. Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, Sue Hyun Park, Hyeonbin Hwang, Jinkyung Jo, Hyowon Cho, Haebin Shin, Seongyun Lee, Hanseok Oh, Noah Lee, Namgyu Ho, Se June Joo, Miyoung Ko, Yoonjoo Lee, Hyungjoo… find
  36. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Metho…
  37. Yafu Li, Zhilin Wang, Leyang Cui, Wei Bi, Shuming Shi, and Yue Zhang. 2024. Spotting AI‘s touch: Identifying LLM-paraphrased spans in text. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7088–7107, Bangkok, Thailand. Association for Computational Linguistics. find
  38. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step.
  39. Michael Xieyang Liu, Frederick Liu, Alexander J Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J Cai. 2024a. " we need structured output": Towards user-centered constraints on large language model output. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems,… find
  40. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, et al. 2023a. Agentbench: Evaluating llms as agents. arXiv preprint arXiv:2308.03688.
  41. Yu Lu Liu, Su Lin Blodgett, Jackie Chi Kit Cheung, Q Vera Liao, Alexandra Olteanu, and Ziang Xiao. 2024b. Ecbd: Evidence-centered benchmark design for nlp. arXiv preprint arXiv:2406.08723. find
  42. Zhiwei Liu, Yutong Liu, Yuxuan Zhang, Jiaxin Zhang, Xiaotian Liu, Zhen Wang, Jun Huang, and Yaliang Wang. 2023b. Bolaa: Benchmarking and orchestrating llm-augmented autonomous agents. arXiv preprint arXiv:2308.05960.
  43. Qitan Lv, Jie Wang, Hanzhu Chen, Bin Li, Yongdong Zhang, and Feng Wu. 2024. Coarse-to-fine highlighting: Reducing knowledge hallucination in large language models. In International Conference on Machine Learning (ICML). find
  44. Kaixin Ma, Hongming Zhang, Hongwei Wang, Xiaoman Pan, Wenhao Yu, and Dong Yu. 2023. Laser: Llm agent with state-space exploration for web navigation. arXiv preprint arXiv:2309.08172. — [arXiv:2309.08172](https://arxiv.org/abs/2309.08172) find
  45. Shirley Kokane, Ming Zhu, Tulika Awalgaonkar, Jianguo Zhang, Thai Hoang, Akshara Prabhakar, Zuxin Liu, Tian Lan, Liangwei Yang, Juntao Tan, et al. 2024. Spectool: A benchmark for characterizing errors in tool-use llms. arXiv preprint arXiv:2411.13547. — [arXiv:2411.13547](https://arxiv.org/abs/2411.13547) find
  46. Xinbei Ma, Yiting Wang, Yao Yao, Tongxin Yuan, Aston Zhang, Zhuosheng Zhang, and Hai Zhao. 2024a. Caution for the environment: Multimodal agents are susceptible to environmental distractions. arXiv preprint arXiv:2408.02544. — [arXiv:2408.02544](https://arxiv.org/abs/2408.02544) find
  47. Harsha Kokel, Michael Katz, Kavitha Srinivas, and Shirin Sohrabi. 2025. Acpbench: Reasoning about action, change, and planning. In AAAI. AAAI Press. find
  48. Zeyang Ma, An Ran Chen, Dong Jae Kim, Tse-Hsun Chen, and Shaowei Wang. 2024b. Llmparser: An exploratory study on using large language models for log parsing. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13. find
  49. Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. 2025. Llms get lost in multi-turn conversation. arXiv preprint arXiv:2505.06120.
  50. Yukyung Lee, Joonghoon Kim, Jaehee Kim, Hyowon Cho, Jaewook Kang, Pilsung Kang, and Najoung Kim. 2025. Checkeval: A reliable llm-as-a-judge framework for evaluating text generation using checklists. arXiv preprint arXiv:2403.18771.
  51. Zeyang Ma, An Ran Chen, Dong Jae Kim, Tse-Husn Chen, and Shaowei Wang. 2024c. Llmparser: An exploratory study on using large language models for log parsing. 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE), pages 1209–1221. find
  52. Meta AI. 2025. Llama 4: Advancing multimodal intelligence. https://ai.meta.com/blog/ llama-4-multimodal-intelligence/. Accessed: May 11, 2025. ↗
  53. Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. 2024. Training software engineering agents and verifiers with swe-gym. arXiv preprint arXiv:2412.21139.
  54. Gregoire Mialon, Clementine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2023. Gaia: a benchmark for general ai assistants. arXiv preprint arXiv:2311.12983.
  55. Shishir G Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2024. Gorilla: Large language model connected with massive APIs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  56. Ivan Milev, Mislav Balunović, Maximilian Baader, and Martin Vechev. 2025. Toolfuzz–automated agent tool testing. arXiv preprint arXiv:2503.04479. — [arXiv:2503.04479](https://arxiv.org/abs/2503.04479) find
  57. Dany Moshkovich, Hadar Mulian, Sergey Zeltyn, Natti Eder, Inna Skarbovsky, and Roy Abitbol. 2025. Beyond black-box benchmarking: Observability, analytics, and optimization of agentic systems. arXiv preprint arXiv:2503.06745. — [arXiv:2503.06745](https://arxiv.org/abs/2503.06745) find
  58. Imran Nasim. 2025. Governance in agentic workflows: Leveraging llms as oversight agents. In AAAI 2025 Workshop on AI Governance: Alignment, Morality, and Law. find
  59. Dang Nguyen, Viet Dac Lai, Seunghyun Yoon, Ryan A. Rossi, Handong Zhao, Ruiyi Zhang, Puneet Mathur, Nedim Lipka, Yu Wang, Trung Bui, Franck Dernoncourt, and Tianyi Zhou. 2024. Dynasaur: Large language agents beyond predefined actions. arXiv preprint arXiv:2411.01747. — [arXiv:2411.01747](https://arxiv.org/abs/2411.01747) find
  60. Augustus Odena, Charles Sutton, David Martin Dohan, Ellen Jiang, Henryk Michalewski, Jacob Austin, Maarten Paul Bosma, Maxwell Nye, Michael Terry, and Quoc V. Le. 2021. Program synthesis with large language models. In n/a, page n/a, n/a. N/a.
  61. OpenAI. 2024. Introducing deep research. OpenAI Blog. Accessed: 2025-05-12. find
  62. OpenAI. 2025a. Introducing GPT-4.1. https://openai.com/index/gpt-4-1/. Accessed: May 11, 2025. ↗
  63. OpenAI. 2025b. Introducing O1: A state-of-the-art multimodal ai model. https://openai.com/o1/. Accessed: May 11, 2025. ↗
  64. OpenAI. 2025c. Introducing o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/. Accessed: May 11, 2025. ↗
  65. OpenAI. 2025d. Introducing o3-mini: A smaller, faster and more cost-effective model. https://openai.com/index/openai-o3-mini/. Accessed: May 9, 2025. ↗
  66. OpenManus. 2024. Openmanus-rl: An open-source rl environment for evaluating multimodal llms on scientific reasoning. https://github.com/OpenManus/ OpenManus-RL. ↗
  67. OpenTelemetry. 2025. OpenTelemetry — opentelemetry.io. https://opentelemetry.io/. [Accessed 07-05-2025]. ↗
  68. Patronus AI. 2025. Modeling statistical risk in ai products. https://www.patronus.ai/blog/ modeling-statistical-risk-in-ai-products. Blog post. ↗
  69. Long Phan et al. 2025. Humanity’s last exam.
  70. Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2024. ChatDev: Communicative agents for software development. In Proceedings of the 62nd Annual Meeting of the Association for Computat…
  71. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, et al. 2023. Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789.
  72. Devjeet Roy, Xuchao Zhang, Rashi Bhave, Chetan Bansal, Pedro Las-Casas, Rodrigo Fonseca, and Saravan Rajmohan. 2024a. Exploring llm-based agents for root cause analysis. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, pages 208– 219. find
  73. Devjeet Roy, Xuchao Zhang, Rashi Bhave, Chetan Bansal, Pedro Las-Casas, Rodrigo Fonseca, and Saravan Rajmohan. 2024b. Exploring llm-based agents for root cause analysis. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, page 208–2… find
  74. Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, and Tianlu Wang. 2025. Learning to plan & reason for evaluation with thinking-llm-as-a-judge. find
  75. Natalie Schluter. 2017. The limits of automatic summarisation according to ROUGE. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 41–45, Valencia, Spain. Association for Computational Linguistics.
  76. Zhuocheng Shen. 2024. Llm with tools: A survey. arXiv preprint arXiv:2409.18807. — [arXiv:2409.18807](https://arxiv.org/abs/2409.18807) find
  77. Connor Shorten, Charles Pierse, Thomas Benjamin Smith, Erika Cardenas, Akanksha Sharma, John Trengrove, and Bob van Luijt. 2024. Structuredrag: Json response formatting with large language models. arXiv preprint arXiv:2408.11061. — [arXiv:2408.11061](https://arxiv.org/abs/2408.11061) find
  78. Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. 2024c. Executable code actions elicit better llm agents. In Proceedings of the 41st International Conference on Machine Learning (ICML 2024).
  79. Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. 2024. Chain of thoughtlessness? an analysis of cot in planning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  80. Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, a… find
  81. Hongjin Su, Howard Yen, Mengzhou Xia, Weijia Shi, Niklas Muennighoff, Han yu Wang, Haisu Liu, Quan Shi, Zachary S. Siegel, Michael Tang, Ruoxi Sun, Jinsung Yoon, Sercan O. Arik, Danqi Chen, and Tao Yu. 2025. Bright: A realistic and challenging benchmark for reasoning-intensive retrieval.
  82. Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. In WSDM ’24: Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 1620–1629. Assoc…
  83. Runchu Tian, Yining Ye, Yujia Qin, Xin Cong, Yankai Lin, Yinxu Pan, Yesai Wu, Hui Haotian, Liu Weichuan, Zhiyuan Liu, and Maosong Sun. 2024. DebugBench: Evaluating debugging capability of large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 4173– 4198,… find
  84. Prapti Trivedi, Aditya Gulati, Oliver Molenschot, Meghana Arakkal Rajeev, Rajkumar Ramamurthy, Keith Stevens, Tanveesh Singh Chaudhery, Jahnavi Jambholkar, James Zou, and Nazneen Rajani. 2024. Self-rationalization improves llm as a fine-grained judge. find
  85. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2024a. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6):186345.
  86. Minzheng Wang, Longze Chen, Fu Cheng, Shengyi Liao, Xinghua Zhang, Bingli Wu, Haiyang Yu, Nan Xu, Lei Zhang, Run Luo, Yunshui Li, Min Yang, Fei Huang, and Yongbin Li. 2024b. Leave no document behind: Benchmarking long-context LLMs with extended multi-doc QA. In Proceedings of the 2024 Conference on…
  87. Sky CH Wang, Darshan Deshpande, Smaranda Muresan, Anand Kannappan, and Rebecca Qian. 2025a. Browsing lost unformed recollections: A benchmark for tip-of-the-tongue search and reasoning. arXiv preprint arXiv:2503.19193. find
  88. Yutong Wang, Pengliang Ji, Chaoqun Yang, Kaixin Li, Ming Hu, Jiaoyang Li, and Guillaume Sartoretti. 2025b. Mcts-judge: Test-time scaling in llm-as-a-judge for code correctness evaluation. arXiv preprint arXiv:2502.12468. — [arXiv:2502.12468](https://arxiv.org/abs/2502.12468) find
  89. Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. 2024e. Helpsteer2-preference: Complementing ratings with preferences.
  90. Hui Wei, Shenghua He, Tian Xia, Fei Liu, Andy Wong, Jingyang Lin, and Mei Han. 2024. Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates. arXiv preprint arXiv:2408.13006. — [arXiv:2408.13006](https://arxiv.org/abs/2408.13006) find
  91. Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Ziv, Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. 2024. Livebench: A challenging, contamination-free llm benchmark. arXiv preprint arXiv:2406.19314. find
  92. Shirley Wu, Shiyu Zhao, Michihiro Yasunaga, Kexin Huang, Kaidi Cao, Qian Huang, Vassilis Ioannidis, Karthik Subbian, James Y Zou, and Jure Leskovec. 2024. Stark: Benchmarking llm retrieval on textual and relational knowledge bases. Advances in Neural Information Processing Systems, 37:127129–127153. find
  93. Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, and Lingming Zhang. 2024. Agentless: Demystifying llm-based software engineering agents. arXiv preprint arXiv:2407.01489.
  94. Austin Xu, Srijan Bansal, Yifei Ming, Semih Yavuz, and Shafiq Joty. 2025. Does context matter? con-textualjudgebench for evaluating llm-based judges in contextual settings. find
  95. Hongshen Xu, Zichen Zhu, Lei Pan, Zihan Wang, Su Zhu, Da Ma, Ruisheng Cao, Lu Chen, and Kai Yu. 2024. Reducing tool hallucination via reliability alignment. arXiv preprint arXiv:2412.04141. — [arXiv:2412.04141](https://arxiv.org/abs/2412.04141) find
  96. Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. 2024. τ -bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprint arXiv:2406.12045.
  97. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
  98. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, et al. 2024a. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736.
  99. Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. 2024b. Flask: Fine-grained language model evaluation based on alignment skill sets. In International Conference on Learning Representations (ICLR).
  100. Asaf Yehudai, Lilach Eden, Alan Li, Guy Uziel, Yilun Zhao, Roy Bar-Haim, Arman Cohan, and Michal Shmueli-Scheuer. 2025. Survey on evaluation of llm-based agents. arXiv preprint arXiv:2503.16416. — [arXiv:2503.16416](https://arxiv.org/abs/2503.16416) find
  101. Ori Yoran, Samuel Joseph Amouyal, Chaitanya Malaviya, Ben Bogin, Ofir Press, and Jonathan Berant. 2024. AssistantBench: Can web agents solve realistic and time-consuming tasks? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8938–8968, Miami, Florida,…
  102. Chenhan Yuan, Qianqian Xie, Jimin Huang, and Sophia Ananiadou. 2024. Back to the future: Towards explainable temporal reasoning with large language models. In Proceedings of the ACM Web Conference 2024, pages 1963–1974. find
  103. Boyang Zhang, Yicong Tan, Yun Shen, Ahmed Salem, Michael Backes, Savvas Zannettou, and Yang Zhang. 2024a. Breaking agents: Compromising autonomous llm agents through malfunction amplification. arXiv preprint arXiv:2407.20859.
  104. Yuxiang Zhang, Jing Chen, Junjie Wang, Yaxin Liu, Cheng Yang, Chufan Shi, Xinyu Zhu, Zihao Lin, Hanwen Wan, Yujiu Yang, Tetsuya Sakai, Tian Feng, and Hayato Yamana. 2024b. Toolbehonest: A multi-level hallucination diagnostic benchmark for tool-augmented large language models. find
  105. Zeyu Zhang, Xiaohe Bo, Chen Ma, Rui Li, Xu Chen, Quanyu Dai, Jieming Zhu, Zhenhua Dong, and Ji-Rong Wen. 2024c. A survey on the memory mechanism of large language model based agents. find
  106. Qi Zhao, Haotian Fu, Chen Sun, and George Konidaris. 2024. Epo: Hierarchical llm agents with environment preference optimization. arXiv preprint arXiv:2408.16090. find
  107. Yong Zhao, Kai Xu, Zhengqiu Zhu, Yue Hu, Zhiheng Zheng, Yingfeng Chen, Yatai Ji, Chen Gao, Yong Li, and Jincai Huang. 2025. Cityeqa: A hierarchical llm agent on embodied question answering benchmark in city space. arXiv preprint arXiv:2502.12532. find
  108. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems (Neur…
  109. Yilun Zhou, Austin Xu, Peifeng Wang, Caiming Xiong, and Shafiq Joty. 2025. Evaluating judges as evaluators: The jetts benchmark of llm-as-judges as test-time scaling evaluators. find
  110. Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2025. Judgelm: Fine-tuned large language models are scalable judges. In Proceedings of the International Conference on Learning Representations (ICLR). Spotlight.
  111. Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. 2024. Agent-as-a-judge: Evaluate agents with agents. arXiv preprint arXiv:2410.10934.
  112. Egor Zverev, Sahar Abdelnabi, Soroush Tabesh, Mario Fritz, and Christoph H Lampert. 2024. Can llms separate instructions from data? and what do we even mean by that? arXiv preprint arXiv:2403.06833. — [arXiv:2403.06833](https://arxiv.org/abs/2403.06833) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/deshpande-2025-trail-trace-reasoning-agentic-issue/