Elamur Research⌕
☰
home / papers / 2025 / BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents

BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents

Paper · 2025 · Jason Wei, Zhiqing Sun, Spencer Papay et al.

Summary

Most earlier retrieval and question-answering benchmarks ask for facts a person could find quickly, so current language models have mostly maxed them out. OpenAI introduces BrowseComp (short for "Browsing Competition"), a set of 1,266 fact-seeking questions. It tests whether an agent can keep searching the web, rephrase its queries in creative ways, and piece together clues spread across many pages. Human trainers built each question backwards. They started from a known fact and added several constraints that each cover a huge search space. The result is a short answer that is easy to check but very expensive to find. Every question had to stump GPT-4o, o1 and an early Deep Research model. A model-based grader checks whether a predicted answer means the same as the reference string. Human trainers working without AI assistants gave up on about seven questions in ten after two hours. Deep Research's accuracy rises steadily as it gets more test-time compute, and it rises further with parallel sampling. The benchmark is deliberately narrow: it does not test long answers or how ambiguous requests are handled. Still, it gives a cheap and reliable measure of how persistently an agent can browse.

Key points

  • Dataset: 1,266 short-answer questions, released in the openai/simple-evals repository with a canary string so the data can be filtered out of training corpora. The largest topics are TV shows & movies (205), science & technology (173), art, history and sports. The set started with 1,287 questions. The authors reviewed the 118 that Deep Research never solved and removed 21 with flawed answers.
  • Difficulty checks during creation: each question had to defeat GPT-4o (with and without browsing), o1 and early Deep Research. The answer could not appear on the first results page of five Google searches. If a second trainer solved a writer's questions more than 40% of the time, the writer had to revise them.
  • Human baseline: trainers attempted 1,255 questions. They solved 367 (29.2%) and gave up on 70.8% after two hours. When they did find an answer, it matched the reference 86.4% of the time.
  • Using more compute: Deep Research produced 64 samples per question. Majority voting, confidence-weighted voting and best-of-N all scored 15–25% higher than a single attempt, and best-of-N did best. This suggests the model can often tell when its answer is right, even though its confidence scores are poorly calibrated.
  • How hard individual tasks are: over 64 trials, Deep Research solved 16% of tasks every time and 14% never. When the model was given the correct answer for the never-solved tasks, it could usually find supporting evidence on the web. So those tasks are very hard, not impossible.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus and Amelia Glaese. 2025. BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv:2504.12516.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2504.12516 · arXiv 2504.12516

@misc{wei2025browsecomp,
  title = {{BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents}},
  author = {Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus and Amelia Glaese},
  year = {2025},
  doi = {10.48550/arxiv.2504.12516},
  eprint = {2504.12516},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2504.12516}
}

References

24 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. E. Dinan, S. Roller, K. Shuster, A. Fan, M. Auli, and J. Weston. Wizard of wikipedia: Knowledge-powered conversational agents. ICLR, 2019.
  2. A. Fan, Y. Jernite, E. Perez, D. Grangier, J. Weston, and M. Auli. ELI5: Long form question answering. In ACL, 2019.
  3. D. Ferrucci, E. Brown, J. Chu-Carroll, J. Fan, D. Gondek, A. Kalyanpur, A. Lally, J. W. Murdock, E. Nyberg, J. Prager, N. Schlaefer, and C. Welty. Building Watson: An overview of the DeepQA project. AI Magazine, 2010. find
  4. Google. Try Deep Research and our new experimental model in Gemini, your AI assistant. Google blog, 2024. find
  5. K. Guu, K. Lee, Z. Tung, P. Pasupat, and M.-W. Chang. REALM: Retrieval-augmented language model pre-training. In ICML, 2020.
  6. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL, 2017.
  7. K. Lee, M.-W. Chang, and K. Toutanova. Latent retrieval for weakly supervised open domain question answering. In ACL, 2019.
  8. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurIPS, 2020.
  9. J. Menick, T. Miller, T. Ribeiro, L. G. Brigato, A. Bakhtin, P. Chatelain, M. Moore, J. Kramár, A. Joulin, K. Shuster, P. Stenetorp, and D. Kiela. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022.
  10. G. Mialon, C. Fourrier, T. Wolf, Y. LeCun, and T. Scialom. GAIA: a benchmark for general AI assistants. In ICLR, 2023.
  11. R. Nakano, J. Hilton, S. Balaji, J. Wu, L. Ouyang, C. Kim, C. Hesse, S. Jain, V. Kosaraju, W. Saunders, X. Jiang, K. Cobbe, T. Eloundou, G. Krueger, K. Button, M. Knight, B. Chess, and J. Schulman. WebGPT: Browser-assisted question-answering with human feedback. arXiv, 2021.
  12. OpenAI. Introducing deep research. OpenAI blog, 2025a. find
  13. OpenAI. Introducing operator. OpenAI blog, 2025b. perplexity.AI. Introducing perplexity deep research. perplexity.AI blog, 2025. find
  14. F. Petroni, A. Piktus, A. Fan, P. Lewis, M. Yazdani, N. De Cao, J. Thorne, Y. Jernite, V. Karpukhin, J. Maillard, V. Plachouras, T. Rocktäschel, and S. Riedel. KILT: a benchmark for knowledge intensive language tasks. In NAACL, 2021.
  15. L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al. Humanity’s last exam. arXiv, 2025. — Humanity's Last Exam
  16. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. GPQA: A graduate-level google-proof Q&A benchmark. In COLM, 2024.
  17. T. Schick, J. Dwivedi-Yu, R. Dessı̀, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom. Toolformer: Language models can teach themselves to use tools. NeurIPS, 2023.
  18. Y. Song, K. Thai, C. M. Pham, Y. Chang, M. Nadaf, and M. Iyyer. BEARCUBS: A benchmark for computer-using web agents. arXiv, 2025. find
  19. A. Srivastava, D. Kleyko, and Z. Wu. Beyond the imitation game: Quantifying and extrapolatingthe capabilities of language models. TMLR, 2023. find
  20. J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal. FEVER: a large-scale dataset for fact extraction and verification. In NAACL, 2018.
  21. J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. Measuring short-form factuality in large language models. arXiv, 2024. x.AI. Grok 3 beta — the age of reasoning agents. x.AI blog, 2025.
  22. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, 2018.
  23. S. Yao, H. Chen, J. Yang, and K. Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. In NeurIPS, 2022.
  24. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y. Cao. ReAct: Synergizing reasoning and acting in language models. In ICLR, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wei-2025-browsecomp-simple-yet-challenging-benchmark/