Elamur Research⌕
☰
home / papers / 2025 / SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge

SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge

Paper · 2025 · Lukas Haas, Gal Yona, Giovanni D'Antonio et al.

Summary

OpenAI's SimpleQA became a standard test of whether LLMs can recall short facts from their own weights. It has flaws, though. Some reference answers are wrong, individual raters contributed clusters of near-identical questions, and dates, person names and science/technology topics dominate.

Researchers at Google DeepMind and Google Research rebuilt it as SimpleQA Verified, a 1,000-question subset. Their pipeline:

  • kept a single question per source URL,
  • removed near-duplicates using Gemini Embeddings and TF-IDF similarity,
  • dropped items whose sources block AI-training crawlers via robots.txt,
  • rebalanced topics and answer types,
  • resolved conflicting sources and gave numeric answers tolerance ranges,
  • preferred questions that frontier models get wrong.

They also revised the grading prompt so that responses listing several hedged guesses count as not attempted.

Models were evaluated without tools. Gemini 2.5 Pro leads with an F1 of 55.6, ahead of GPT-5 (52.3) and o3 (51.9). The released dataset, code and Kaggle leaderboard give a cleaner signal of real progress in parametric factuality and make it harder to gain by overfitting to dataset quirks.

Key points

  • Curation shrank the dataset from 4,326 to 1,000 items (−76.9%). The robots.txt publisher filter alone removed about 30% (2,664 → 1,855).
  • De-duplication used a cosine cutoff of 0.77 on Gemini Embeddings, then TF-IDF similarity above 0.4 with manual review. The original set had, for example, 119 questions about founding dates of Colombian municipalities.
  • Numeric answers received acceptable error margins (roughly ±1% or ±5% depending on magnitude). The autorater prompt (run with gpt-4.1-2025-04-14) grew from 963 to 1,124 words.
  • 13 models were compared. GPT-4o, Claude Opus 4 and Claude Sonnet 4 score significantly lower than on SimpleQA, while o4-mini improves. Claude models abstain often; Opus 4 attempts only 35.5% of questions.
  • With search tools enabled, scores approach perfection, so the benchmark is intended strictly for tool-free evaluation. It is hosted on Kaggle (deepmind/simpleqa-verified).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Lukas Haas, Gal Yona, Giovanni D'Antonio, Sasha Goldshtein and Dipanjan Das. 2025. SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge. arXiv:2509.07968.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2509.07968 · arXiv 2509.07968

@misc{haas2025simpleqa,
  title = {{SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge}},
  author = {Lukas Haas, Gal Yona, Giovanni D'Antonio, Sasha Goldshtein and Dipanjan Das},
  year = {2025},
  doi = {10.48550/arxiv.2509.07968},
  eprint = {2509.07968},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2509.07968}
}

References

29 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Introducing claude 4, 2025a. URL https://www.anthropic.com/news/claude-4 . ↗
  2. Anthropic. Claude 3.7 Sonnet and Claude Code, 2025b. URL https://www.anthropic.com/ news/claude-3-7-sonnet . ↗
  3. S. Chen, Y. Zhao, J. Zhang, I.-C. Chern, S. Gao, P. Liu, and J. He. Felm: Benchmarking factuality evaluation of large language models. arXiv preprint arXiv:2310.00741, 2023. find
  4. DeepSeek-AI, D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi, X. Zhang, X. Yu, Y. Wu, Z. F. Wu, Z. Gou, Z. Shao, Z. Li, Z. Gao, A. Liu, B. Xue, B. Wang, B. Wu, B. Feng, C. Lu, C. Zhao, C. Deng, C. Zhang, C. Ruan, D. Dai, D. Chen, D. Ji, E. Li, F. Lin, F. Dai, F. Lu… — DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
  5. Gemini Team, Google. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261, 2025.
  6. M. Geva, D. Khashabi, E. Segal, T. Khot, D. Roth, and J. Berant. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. arXiv preprint arXiv:2101.02235, 2021. find
  7. A. Jacovi, A. Wang, C. Alberti, C. Tao, J. Lipovetz, K. Olszewska, L. Haas, M. Liu, N. Keating, A. Bloniarz, C. Saroufim, C. Fry, D. Marcus, D. Kukliansky, G. S. Tomar, J. Swirhun, J. Xing, L. Wang, M. Gurumurthy, M. Aaron, M. Ambar, R. Fellinger, R. Wang, Z. Zhang, S. Goldshtein, and D. Das. The fa… — The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input
  8. K. S. Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of Documentation, 28:11–21, 1972. find
  9. M. Joshi, E. Choi, D. Weld, and L. Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611. Association for Computational…
  10. J. Kasai, K. Sakaguchi, Y. Takahashi, R. L. Bras, A. Asai, X. Yu, D. Radev, N. A. Smith, Y. Choi, and K. Inui. Realtime qa: What’s the answer right now? arXiv preprint arXiv:2207.13332, 2024.
  11. S. Krishna, K. Krishna, A. Mohananey, S. Schwarcz, A. Stambler, S. Upadhyay, and M. Faruqui. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation. arXiv preprint arXiv:2409.12941, 2025. find
  12. T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural questions: A benchmark for question answering research. Transactions of th…
  13. J. Lee, F. Chen, S. Dua, D. Cer, M. Shanbhogue, I. Naim, G. H. Ábrego, Z. Li, K. Chen, H. S. Vera, X. Ren, S. Zhang, D. Salz, M. Boratko, J. Han, B. Chen, S. Huang, V. Rao, P. Suganthan, F. Han, A. Doumanoglou, N. Gupta, F. Moiseev, C. Yip, A. Jain, S. Baumgartner, S. Shahi, F. P. Gomez, S. Mariserl… — Gemini Embedding: Generalizable Embeddings from Gemini
  14. S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2022.
  15. S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In H. Bouamor, J. Pino, and K. Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in…
  16. OpenAI. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024a.
  17. OpenAI. Learning to reason with LLMs, 2024b. find
  18. URL https://openai.com/index/learning-to-reason-with-llms . OpenAI. Introducing gpt-4.1 in the api, 2025a. URL https://openai.com/index/gpt-4-1/ . ↗
  19. OpenAI. Gpt-5 system card, 2025b. URL — OpenAI GPT-5 System Card https://openai.com/index/gpt-5-system-card/ . ↗
  20. L. Pan, X. Wu, X. Lu, A. T. Luu, W. Y. Wang, M.-Y. Kan, and P. Nakov. Fact-checking complex claims with program-guided reasoning. arXiv preprint arXiv:2305.12744, 2023.
  21. S. Pichai, D. Hassabis, and K. Kavukcuoglu. Introducing Gemini 2.0: our new ai model for the agentic era, 2024. URL https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024 . ↗
  22. H. Rashkin, V. Nikolaev, M. Lamm, L. Aroyo, M. Collins, D. Das, S. Petrov, G. S. Tomar, I. Turc, and D. Reitter. Measuring attribution in natural language generation models. arXiv preprint arXiv:2112.12870, 2022.
  23. Y. Song, Y. Kim, and M. Iyyer. Veriscore: Evaluating the factuality of verifiable claims in long-form text generation. arXiv preprint arXiv:2406.19276, 2024.
  24. T. Vu, M. Iyyer, X. Wang, N. Constant, J. Wei, J. Wei, C. Tar, Y.-H. Sung, D. Zhou, Q. Le, and T. Luong. Freshllms: Refreshing large language models with search engine augmentation. arXiv preprint arXiv:2310.03214, 2023.
  25. J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024a.
  26. J. Wei, C. Yang, X. Song, Y. Lu, N. Hu, J. Huang, D. Tran, D. Peng, R. Liu, D. Huang, C. Du, and Q. V. Le. Long-form factuality in large language models. arXiv preprint arXiv:2403.18802, 2024b.
  27. X. Yang, K. Sun, H. Xin, Y. Sun, N. Bhalla, X. Chen, S. Choudhary, R. D. Gui, Z. W. Jiang, Z. Jiang, L. Kong, B. Moran, J. Wang, Y. E. Xu, A. Yan, C. Yang, E. Yuan, H. Zha, N. Tang, L. Chen, N. Scheffer, Y. Liu, N. Shah, R. Wanga, A. Kumar, W. tau Yih, and X. L. Dong. Crag – comprehensive rag benchm… — [arXiv:2406.04744](https://arxiv.org/abs/2406.04744) find
  28. Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. W. Cohen, R. Salakhutdinov, and C. D. Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018.
  29. A. SimpleQA Verified Evaluation Prompt The following is the updated SimpleQA Verified evaluation prompt, where differences from the original SimpleQA (Wei et al., 2024a) prompt are highlighted in red. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/haas-2025-simpleqa-verified-reliable-factuality-benchmark/