Elamur Research⌕
☰
home / papers / 2025 / WideSearch: Benchmarking Agentic Broad Info-Seeking

WideSearch: Benchmarking Agentic Broad Info-Seeking

Paper · 2025 · Ryan Wong, Jiawei Wang, Junjie Zhao et al.

Summary

Many everyday and professional information needs, such as listing every job opening or admission threshold that matches a set of conditions, are hard because of their scale and the need for exactness, not because of deep reasoning. Existing agent benchmarks test either finding one obscure fact (DeepSearch-style, e.g., BrowseComp) or writing synthesized reports (DeepResearch-style). This broad, collection-heavy setting has gone unmeasured.

ByteDance Seed researchers built WideSearch to fill that gap. It has 200 hand-curated tasks in English and Chinese, derived from real user requests. Each task gives the agent a natural-language query and a fixed table schema. The agent must fill the table from the live web, and every cell is an atomic fact that can be checked objectively by a hybrid automatic evaluator.

The authors tested more than ten single-agent, multi-agent and commercial systems. Table-level success was near zero, peaking around 5%. Agents can retrieve most individual facts, so the real weakness is completeness and accuracy at scale. This points toward multi-agent designs with parallel search and cross-checking.

Key points

  • Dataset: 200 tasks (100 English, 100 Chinese) covering 15+ domains, grounded in real user queries and filtered through a five-stage quality-control pipeline.
  • Task format: each instance pairs a query (Q) with a table schema (S). The agent must identify the full set of entities and fill every attribute. A single missing, extra or wrong cell means the task fails.
  • Results: the best system, a multi-agent framework, reached about 5.1% success. Lone human annotators managed roughly 20%, while cross-validation by several humans approaches 100%.
  • Test-time scaling: the authors ran Kimi K2 in single-agent mode with up to 128 attempts per task. Item-level F1 rose close to 80, but table-level success stayed below 20.
  • Error analysis:
    • Higher-level failures include incomplete query decomposition, no reflection after failed searches, and misuse of retrieved evidence.
    • Basic failures include tool-call errors, malformed tables, context-window overflow and refusals.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Ryan Wong, Jiawei Wang, Junjie Zhao, Li Chen, Yan Gao, Long Zhang, Xuan Zhou, Zuo Wang, Kai Xiang, Ge Zhang, Wenhao Huang, Yang Wang and Ke Wang. 2025. WideSearch: Benchmarking Agentic Broad Info-Seeking. arXiv:2508.07999.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2508.07999 · arXiv 2508.07999

@misc{wong2025widesearch,
  title = {{WideSearch: Benchmarking Agentic Broad Info-Seeking}},
  author = {Ryan Wong, Jiawei Wang, Junjie Zhao, Li Chen, Yan Gao, Long Zhang, Xuan Zhou, Zuo Wang, Kai Xiang, Ge Zhang, Wenhao Huang, Yang Wang and Ke Wang},
  year = {2025},
  doi = {10.48550/arxiv.2508.07999},
  eprint = {2508.07999},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2508.07999}
}

References

32 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Introducing Claude 4. https://www.anthropic.com/news/claude-4, 2025. ↗
  2. Kaiyuan Chen, Yixin Ren, Yang Liu, Xiaobo Hu, Haotong Tian, Tianbao Xie, Fangfu Liu, Haoye Zhang, Hongzhang Liu, Yuan Gong, et al. xbench: Tracking agents productivity scaling with profession-aligned real-world evaluations. arXiv preprint arXiv:2506.13651, 2025. — [arXiv:2506.13651](https://arxiv.org/abs/2506.13651) find
  3. Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763, 2025.
  4. Google Gemini. Deep Research is now available on Gemini 2.5 Pro Experimental. https://blog.google/ products/gemini/deep-research-gemini-2-5-pro-experimental/, 2025. ↗
  5. Google Gemini. Gemini 2.5: Our most intelligent AI model. https://blog.google/technology/ google-deepmind/gemini-model-thinking-updates-march-2025/, 2025. ↗
  6. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  7. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, 2020.
  8. Ziyang Huang, Xiaowei Yuan, Yiming Ju, Jun Zhao, and Kang Liu. Reinforced internal-external knowledge synergistic reasoning for efficient adaptive search agent. arXiv preprint arXiv:2505.07596, 2025. — [arXiv:2505.07596](https://arxiv.org/abs/2505.07596) find
  9. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. Search-r1: Training llms to reason and leverage search engines with reinforcement learning. arXiv preprint arXiv:2503.09516, 2025.
  10. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, 2017.
  11. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:4…
  12. Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin SU, ZHAOQING SUO, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows. In The… find
  13. Kuan Li, Zhongwang Zhang, Huifeng Yin, Liwen Zhang, Litu Ou, Jialong Wu, Wenbiao Yin, Baixuan Li, Zhengwei Tao, Xinyu Wang, et al. Websailor: Navigating super-human reasoning for web agent. arXiv preprint arXiv:2507.02592, 2025. find
  14. Xiaoxi Li, Jiajie Jin, Guanting Dong, Hongjin Qian, Yutao Zhu, Yongkang Wu, Ji-Rong Wen, and Zhicheng Dou. Webthinker: Empowering large reasoning models with deep research capability. arXiv preprint arXiv:2504.21776, 2025. find
  15. Manus. Leave it to Manus. https://manus.im, 2025. ↗
  16. Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. Gaia: a benchmark for general ai assistants. In The Twelfth International Conference on Learning Representations, 2023.
  17. Moonshot AI. Kimi-Researcher: End-to-End RL Training for Emerging Agentic Capabilities. https://moonshotai.github.io/Kimi-Researcher/, 2025. ↗
  18. OpenAI. Deep Research System Card. https://openai.com/index/deep-research-system-card/, 2024. ↗
  19. OpenAI. OpenAI o3 and o4-mini System Card. https://cdn.openai.com/pdf/ 2221c875-02dc-4789-800b-e7758f3722c1/o3-and-o4-mini-system-card.pdf, 2025. ↗
  20. Jiahao Qiu, Xuan Qi, Tongcheng Zhang, Xinzhe Juan, Jiacheng Guo, Yifu Lu, Yimin Wang, Zixin Yao, Qihan Ren, Xun Jiang, et al. Alita: Generalist agent enabling scalable agentic reasoning with minimal predefinition and maximal self-evolution. arXiv preprint arXiv:2505.20286, 2025. — [arXiv:2505.20286](https://arxiv.org/abs/2505.20286) find
  21. ByteDance Seed, Jiaze Chen, Tiantian Fan, Xin Liu, Lingjun Liu, Zhiqi Lin, Mingxuan Wang, Chengyi Wang, Xiangpeng Wei, Wenyuan Xu, et al. Seed1. 5-thinking: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504.13914, 2025.
  22. Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592, 2025.
  23. Huatong Song, Jinhao Jiang, Wenqing Tian, Zhipeng Chen, Yuhuan Wu, Jiahao Zhao, Yingqian Min, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. R1-searcher++: Incentivizing the dynamic knowledge acquisition of llms via reinforcement learning. arXiv preprint arXiv:2505.17005, 2025. — [arXiv:2505.17005](https://arxiv.org/abs/2505.17005) find
  24. Hao Sun, Zile Qiao, Jiayan Guo, Xuanbo Fan, Yingyan Hou, Yong Jiang, Pengjun Xie, Yan Zhang, Fei Huang, and Jingren Zhou. Zerosearch: Incentivize the search capability of llms without searching. arXiv preprint arXiv:2505.04588, 2025.
  25. Kimi Team, Yifan Bai, Yiping Bao, Guanduo Chen, Jiahao Chen, Ningxin Chen, Ruijue Chen, Yanru Chen, Yuankun Chen, Yutian Chen, et al. Kimi k2: Open agentic intelligence. arXiv preprint arXiv:2507.20534, 2025.
  26. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Musique: Multihop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554, 2022. — ♫ MuSiQue: Multihop Questions via Single-hop Question Composition
  27. Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, and Amelia Glaese. Browsecomp: A simple yet challenging benchmark for browsing agents. arXiv preprint arXiv:2504.12516, 2025. find
  28. Jialong Wu, Baixuan Li, Runnan Fang, Wenbiao Yin, Liwen Zhang, Zhengwei Tao, Dingchu Zhang, Zekun Xi, Yong Jiang, Pengjun Xie, et al. Webdancer: Towards autonomous information seeking agency. arXiv preprint arXiv:2505.22648, 2025.
  29. x.ai. Grok 3 beta — the age of reasoning agents. https://x.ai/news/grok-3, 2025. ↗
  30. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
  31. Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160, 2025.
  32. Peilin Zhou, Bruce Leon, Xiang Ying, Can Zhang, Yifan Shao, Qichen Ye, Dading Chong, Zhiling Jin, Chenxuan Xie, Meng Cao, et al. Browsecomp-zh: Benchmarking web browsing ability of large language models in chinese. arXiv preprint arXiv:2504.19314, 2025. — [arXiv:2504.19314](https://arxiv.org/abs/2504.19314) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wong-2025-widesearch-benchmarking-agentic-broad-info/