Elamur Research⌕
☰
home / papers / 2025 / DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents

DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents

Paper · 2025 · Mingxuan Du, Benfeng Xu, Chiwei Zhu et al.

Summary

Deep Research Agents (DRAs) are LLM systems that search the web over many steps and turn what they find into long, cited reports. Until now there was no dedicated way to evaluate them. Earlier benchmarks tested browsing or writing separately, and long research questions have no single correct answer. The authors present DeepResearch Bench, which contains 100 PhD-level tasks across 22 fields. Domain experts wrote and refined the tasks. The number of tasks per topic follows the topic mix found in tens of thousands of real chatbot queries.

The benchmark uses two automatic evaluation frameworks:

  • RACE scores report quality. A Judge LLM sets task-specific weights and criteria, then rates the target report against a high-quality reference report.
  • FACT measures retrieval. It extracts each claim and its cited URL, then checks whether the cited page actually backs the claim.

Human studies show that RACE agrees with expert raters more often than the experts agree with one another. The benchmark therefore offers a reproducible, human-aligned standard for comparing research agents. The authors also argue that RACE and FACT are useful beyond deep research.

Key points

  • Dataset construction: The authors started from 96,147 anonymized user queries. DeepSeek-V3-0324 filtered these down to 44,019 deep-research queries and sorted them into 22 domains using the WebOrganizer taxonomy. The final benchmark contains 100 tasks, divided evenly between the two languages: half (50) are in Chinese and the other half (50) are in English. Task authors were PhDs or practitioners with more than five years of domain experience.
  • RACE design: It weights four dimensions per task: Comprehensiveness, Insight/Depth, Instruction-Following and Readability. The final score is the target report's share of the combined target-plus-reference score.
  • FACT design: Page text is fetched with the Jina Reader API. A binary support/no-support judgment then yields two metrics: citation accuracy and the average number of effective citations.
  • Human agreement:
    • Full RACE reaches a Pairwise Agreement Rate of 71.33%, above the 68.44% agreement between human experts.
    • A vanilla judging prompt reaches only 58.89%.
    • On the overall consistency score, RACE scores 72.56 versus 60.46 for the vanilla prompt.
  • Judge model and release: Gemini 2.5 Pro Preview was chosen as the Judge LLM because it had the best overall consistency at about $0.13 per query; o4-mini was cheaper at $0.04. The benchmark and evaluation code are released at github.com/Ayanami0730/deep_research_bench.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang and Zhendong Mao. 2025. DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents. arXiv:2506.11763.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2506.11763 · arXiv 2506.11763

@misc{du2025deepresearch,
  title = {{DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents}},
  author = {Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang and Zhendong Mao},
  year = {2025},
  doi = {10.48550/arxiv.2506.11763},
  eprint = {2506.11763},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2506.11763}
}

References

38 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs, August 2024. arXiv:2408.07055 [cs].
  2. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, Lilian Weng, and Aleksander M adry. MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering, February 2025. arXiv:2410.07095 [cs].
  3. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. ScienceAgentBench: Toward Rigorous Assessment of… find
  4. DeepSeek-AI. deepseek-ai/DeepSeek-V3-0324 · Hugging Face, March 2025. find
  5. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chengg… — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  6. Shihan Deng, Weikai Xu, Hongda Sun, Wei Liu, Tao Tan, Jianfeng Liu, Ang Li, Jian Luan, Bin Wang, Rui Yan, and Shuo Shang. Mobile-Bench: An Evaluation Benchmark for LLM-based Mobile Agents, July 2024. arXiv:2407.00993 [cs]. find
  7. Gemini Google. Deep Research is now available on Gemini 2.5 Pro Experimental., April 2025. find
  8. Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework, November 2024. ar…
  9. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code, June 2024. arXiv:2403.07974 [cs].
  10. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?, November 2024. arXiv:2310.06770 [cs].
  11. Jon M. Laurent, Joseph D. Janizek, Michael Ruzo, Michaela M. Hinks, Michael J. Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D. White, and Samuel G. Rodriques. LAB-Bench: Measuring Capabilities of Language Models for Biology Research, July 2024. arXiv:2407.10362 [cs]. find
  12. Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. WebThinker: Empowering Large Reasoning Models with Deep Research Capability, April 2025. arXiv:2504.21776 [cs]. find
  13. Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. AgentBench: Evaluating LLMs as Age… — AgentBench: Evaluating LLMs as Agents
  14. Tula Masterman, Sandi Besen, Mason Sawtell, and Alex Chao. The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey, April 2024. arXiv:2404.11584 [cs]. find
  15. Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom. GAIA: a benchmark for General AI Assistants, November 2023. arXiv:2311.12983 [cs].
  16. Ludovico Mitchener, Jon M. Laurent, Benjamin Tenmann, Siddharth Narayanan, Geemi P. Wellawatte, Andrew White, Lorenzo Sani, and Samuel G. Rodriques. BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology, March 2025. arXiv:2503.00096 [q-bio]. find
  17. Ruaridh Mon-Williams, Gen Li, Ran Long, Wenqian Du, and Christopher G. Lucas. Embodied large language models enable robots to complete complex tasks in unpredictable environments. Nature Machine Intelligence, 7(4):592–601, April 2025. Publisher: Nature Publishing Group. find
  18. OpenAI. Introducing deep research \| OpenAI. find
  19. OpenAI, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, Alex Iftimie, Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Sar… — OpenAI o1 System Card
  20. Long Phan, Alice Gatti, Ziwen Han, Nathaniel Li, Josephina Hu, Hugh Zhang, Chen Bo Calvin Zhang, Mohamed Shaaban, John Ling, Sean Shi, Michael Choi, Anish Agrawal, Arnav Chopra, Adam Khoja, Ryan Kim, Richard Ren, Jason Hausenloy, Oliver Zhang, Mantas Mazeika, Dmitry Dodonov, Tung Nguyen, Jaeho Lee,… — Humanity's Last Exam
  21. Shanghaoran Quan, Jiaxi Yang, Bowen Yu, Bo Zheng, Dayiheng Liu, An Yang, Xuancheng Ren, Bofei Gao, Yibo Miao, Yunlong Feng, Zekun Wang, Jian Yang, Zeyu Cui, Yang Fan, Yichang Zhang, Binyuan Hui, and Junyang Lin. CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable El… — CodeElo: Benchmarking Competition-level Code Generation of LLMs with Human-comparable Elo Ratings
  22. Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, Junran Peng, Zhaoxiang Zhang, Songyang Zhang, and Kai Chen. HelloBench: Evaluating Long Text Generation Capabilities of Large Language Models, September 2024. arXiv:2409.161… — [arXiv:2409.16191](https://arxiv.org/abs/2409.16191) find
  23. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. Agent Laboratory: Using LLM Agents as Research Assistants, January 2025.
  24. Yijia Shao, Yucheng Jiang, Theodore A. Kanell, Peter Xu, Omar Khattab, and Monica S. Lam. Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Models, April 2024. arXiv:2402.14207 [cs].
  25. Luyuan Wang, Yongyu Deng, Yiwei Zha, Guodong Mao, Qinmin Wang, Tianchen Min, Wei Chen, and Shoufa Chen. MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents, June 2024. arXiv:2406.08184 [cs]. — [arXiv:2406.08184](https://arxiv.org/abs/2406.08184) find
  26. Wenxuan Wang, Zizhan Ma, Zheng Wang, Chenghan Wu, Wenting Chen, Xiang Li, and Yixuan Yuan. A Survey of LLM-based Agents in Medicine: How far are we from Baymax?, February 2025. arXiv:2502.11211 [cs] version: 1. find
  27. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents, April 2025. arXiv:2504.12516 [cs]. find
  28. Alexander Wettig, Kyle Lo, Sewon Min, Hannaneh Hajishirzi, Danqi Chen, and Luca Soldaini. Organize the Web: Constructing Domains Enhances Pre-Training Data Curation, February 2025. arXiv:2502.10341 [cs]. find
  29. Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, and Fei Huang. WebWalker: Benchmarking LLMs in Web Traversal, January 2025. arXiv:2501.07572 [cs].
  30. Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang. WritingBench: A Comprehensive Benchmark for Generative Writing, March 2025. arXiv:2503.05244 [cs]. — [arXiv:2503.05244](https://arxiv.org/abs/2503.05244) find
  31. Yijia Xiao, Runhui Wang, Luyang Kong, Davor Golac, and Wei Wang. CSR-Bench: Benchmarking LLM Agents in Deployment of Computer Science Research Repositories. In Luis Chiruzzo, Alan Ritter, and Lu Wang, editors, Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associati… find
  32. Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. Berkeley Function Calling Leaderboard, February 2024. bibtex[howpublished] = {\\url{https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html}} citationkey: ber… find
  33. John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, November 2024. arXiv:2405.15793 [cs].
  34. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, December 2023. arXiv:2306.05685 [cs].
  35. Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deep-Researcher: Scaling Deep Research via Reinforcement Learning in Real-world Environments, April 2025. arXiv:2504.03160 [cs]. — [arXiv:2504.03160](https://arxiv.org/abs/2504.03160) find
  36. Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, Yuxin Gu, Sixin Hong, Jing Ren, Jian Chen, Chao Liu, and Yining Hua. BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinese, May 2025. arXiv:2504.19… — [arXiv:2504.19314](https://arxiv.org/abs/2504.19314) find
  37. Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, Uri Alon, and Graham Neubig. WebArena: A Realistic Web Environment for Building Autonomous Agents, April 2024. arXiv:2307.13854 [cs].
  38. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex G… — BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/du-2025-deepresearch-bench-comprehensive-benchmark-deep/