Elamur Research⌕
☰
home / papers / 2025 / The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input

The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input

Paper · 2025 · Alon Jacovi, Andrew Wang, Chris Alberti et al.

Summary

The paper presents FACTS Grounding, a benchmark and Kaggle-hosted leaderboard from Google DeepMind, Google Research, Google Cloud and Kaggle. It tests whether LLMs write long answers that rely strictly on a supplied document and do not add outside facts. Human annotators wrote realistic requests over web documents of up to 32k tokens. The documents cover enterprise domains such as finance, technology, retail, medicine and law. The requests include fact finding, summarization, comparison and rewriting. Prompts needing creativity, expert knowledge or mathematical reasoning were filtered out.

Scoring is automatic and runs in two stages. First, responses that ignore the user's request are disqualified. Second, the remaining responses are checked for full support by the source text. Three different LLM judges are averaged to offset each judge's tendency to favour its own outputs. Among the nine models tested, Gemini 2.0 Flash Experimental ranks first, while the OpenAI o1 models come last. The benchmark matters because it measures grounding across many task types rather than summarization alone. Its hidden split also helps protect the leaderboard from overfitting.

Key points

  • The data has two splits: 860 public "Open" examples and 859 private "Blind" examples. Each example combines a system instruction, a context document of up to 32k tokens, and a user request.
  • The judges are Gemini 1.5 Pro, GPT-4o and Claude 3.5 Sonnet. Each judge rated its own family's outputs about 3.23% higher on average, which is why the final score pools all three.
  • Treating ineligible responses as inaccurate lowers final scores by roughly 1–5%. It also changes the ranking slightly, for example moving Gemini 1.5 Flash from first to second place.
  • Final average factuality scores:
    • Gemini 2.0 Flash Experimental: 83.6
    • Gemini 1.5 Flash: 82.9
    • Gemini 1.5 Pro: 80.0
    • Claude 3.5 Sonnet: 79.4
    • GPT-4o: 78.8
    • OpenAI o1-mini: 62.0
    • OpenAI o1-preview: 61.7
  • A Condorcet-based fusion of the six split-by-judge rankings gives exactly the same order as the averaged score.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States492 works · 440,462 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, Carl Saroufim, Corey Fry, Dror Marcus, Doron Kukliansky, Gaurav Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, Michael Aaron, Moran Ambar, Rachana Fellinger, Rui Wang, Zizhao Zhang, Sasha Goldshtein and Dipanjan Das. 2025. The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input. arXiv:2501.03200.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2501.03200 · arXiv 2501.03200

@misc{jacovi2025facts,
  title = {{The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input}},
  author = {Alon Jacovi, Andrew Wang, Chris Alberti, Connie Tao, Jon Lipovetz, Kate Olszewska, Lukas Haas, Michelle Liu, Nate Keating, Adam Bloniarz, Carl Saroufim, Corey Fry, Dror Marcus, Doron Kukliansky, Gaurav Singh Tomar, James Swirhun, Jinwei Xing, Lily Wang, Madhu Gurumurthy, Michael Aaron, Moran Ambar, Rachana Fellinger, Rui Wang, Zizhao Zhang, Sasha Goldshtein and Dipanjan Das},
  year = {2025},
  doi = {10.48550/arxiv.2501.03200},
  eprint = {2501.03200},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2501.03200}
}

References

39 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. Anthropic. The Claude 3 model family: Opus, Sonnet, Haiku, 2024. find
  3. URL https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_ Card_Claude_3.pdf . J. A. Bishop, Q. Xie, and S. Ananiadou. LongDocFACTScore: Evaluating the factuality of long document abstractive summarisation. arXiv preprint arXiv:2309.12455, 2023. ↗
  4. Y. Chang, K. Lo, T. Goyal, and M. Iyyer. Booookscore: A systematic exploration of book-length summarization in the era of LLMs. arXiv preprint arXiv:2310.00785, 2023.
  5. Z. Gekhman, J. Herzig, R. Aharoni, C. Elkind, and I. Szpektor. Trueteacher: Learning factual consistency evaluation with large language models. arXiv preprint arXiv:2305.11171, 2023.
  6. Introducing gemini 2.0: our new ai model for the agentic URL https://blog.google/technology/google-deepmind/ google-gemini-ai-update-december-2024 . Accessed: 2024-12-11. ↗
  7. Gemini Team: R. Anil, S. Borgeaud, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, K. Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  8. O. Honovich, R. Aharoni, J. Herzig, H. Taitelbaum, D. Kukliansy, V. Cohen, T. Scialom, I. Szpektor, A. Hassidim, and Y. Matias. TRUE: Re-evaluating factual consistency evaluation. arXiv preprint arXiv:2204.04991, 2022.
  9. C.-W. Huang and Y.-N. Chen. FactAlign: Long-form factuality alignment of large language models. arXiv preprint arXiv:2410.01691, 2024. find
  10. A. Jacovi, M. Ambar, E. Ben-David, U. Shaham, A. Feder, M. Geva, D. Marcus, and A. Caciularu. Cover-bench: A challenging benchmark for complex claim verification. arXiv preprint arXiv:2408.03325, 2024. — [arXiv:2408.03325](https://arxiv.org/abs/2408.03325) find
  11. A. T. Kalai and S. S. Vempala. Calibrated language models must hallucinate. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, pages 160–171, 2024. find
  12. M. Karpinska, K. Thai, K. Lo, T. Goyal, and M. Iyyer. One thousand and one pairs: A "novel" challenge for long-context language models. arXiv preprint arXiv:2406.16264, 2024. find
  13. Y. Kim, Y. Chang, M. Karpinska, A. Garimella, V. Manjunatha, K. Lo, T. Goyal, and M. Iyyer. FA-BLES: Evaluating faithfulness and content selection in book-length summarization. arXiv preprint arXiv:2404.01261, 2024. — [arXiv:2404.01261](https://arxiv.org/abs/2404.01261) find
  14. K. Krishna, E. Bransom, B. Kuehl, M. Iyyer, P. Dasigi, A. Cohan, and K. Lo. LongEval: Guidelines for human evaluation of faithfulness in long-form summarization. arXiv preprint arXiv:2301.13298, 2023.
  15. Z. Lan, W. Li, J. Su, X. Xiao, J. Liu, W. Wu, and Y. Lyu. Factgen: Faithful text generation by factuality-aware pre-training and contrastive ranking fine-tuning. Journal of Artificial Intelligence Research, 76:1281–1303, 2023. find
  16. S. Lee, H. Hsu, and C.-F. Chen. LLM hallucination reasoning with zero-shot knowledge test. arXiv preprint arXiv:2411.09689, 2024. — [arXiv:2411.09689](https://arxiv.org/abs/2411.09689) find
  17. J. Li, X. Cheng, W. X. Zhao, J.-Y. Nie, and J.-R. Wen. HaluEval: A large-scale hallucination evaluation benchmark for large language models. arXiv preprint arXiv:2305.11747, 2023.
  18. Y. Liu, N. Moosavi, and C. Lin. LLMs as narcissistic evaluators: When ego inflates evaluation scores. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 12688–12701, Bangkok, Thailand, Aug. 2024. Association for Computational… find
  19. S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, P. W. Koh, M. Iyyer, L. Zettlemoyer, and H. Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251, 2023.
  20. OpenAI. Learning to reason with LLMs, 2024. find
  21. URL — Fact-Checking Complex Claims with Program-Guided Reasoning https://openai.com/index/learning-to-reason-with-llms . L. Pan, X. Wu, X. Lu, A. T. Luu, W. Y. Wang, M.-Y. Kan, and P. Nakov. Fact-checking complex claims with program-guided reasoning. arXiv preprint arXiv:2305.12744, 2023. ↗
  22. S. Ramprasad and B. C. Wallace. Do automatic factuality metrics measure factuality? A critical evaluation. arXiv preprint arXiv:2411.16638, 2024. — [arXiv:2411.16638](https://arxiv.org/abs/2411.16638) find
  23. S. Ramprasad, K. Krishna, Z. C. Lipton, and B. C. Wallace. Evaluating the factuality of zero-shot summarizers across varied domains. arXiv preprint arXiv:2402.03509, 2024. find
  24. H. Rashkin, V. Nikolaev, M. Lamm, L. Aroyo, M. Collins, D. Das, S. Petrov, G. S. Tomar, I. Turc, and D. Reitter. Measuring attribution in natural language generation models. Computational Linguistics, 49(4):777–840, 2023.
  25. P. Roit, J. Ferret, L. Shani, R. Aharoni, G. Cideron, R. Dadashi, M. Geist, S. Girgin, L. Hussenot, O. Keller, et al. Factually consistent summarization via reinforcement learning with textual entailment feedback. arXiv preprint arXiv:2306.00186, 2023.
  26. O. Sainz, I. García-Ferrero, A. Jacovi, J. A. Campos, Y. Elazar, E. Agirre, Y. Goldberg, W.-L. Chen, J. Chim, L. Choshen, et al. Data contamination report from the 2024 CONDA shared task. arXiv preprint arXiv:2407.21530, 2024. find
  27. Y. Song, Y. Kim, and M. Iyyer. VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation. arXiv preprint arXiv:2406.19276, 2024.
  28. W. Su, C. Wang, Q. Ai, Y. Hu, Z. Wu, Y. Zhou, and Y. Liu. Unsupervised real-time hallucination detection based on the internal states of large language models. arXiv preprint arXiv:2403.06448, 2024.
  29. L. Tang, P. Laban, and G. Durrett. MiniCheck: Efficient fact-checking of LLMs on grounding documents. arXiv preprint arXiv:2404.10774, 2024.
  30. Vectara. Hallucination evaluation model (revision 7437011), 2024. URL https://huggingface.co/vectara/hallucination_evaluation_model . ↗
  31. K. Wataoka, T. Takahashi, and R. Ri. Self-preference bias in LLM-as-a-judge. arXiv preprint arXiv:2410.21819, 2024.
  32. J. Wei, N. Karina, H. W. Chung, Y. J. Jiao, S. Papay, A. Glaese, J. Schulman, and W. Fedus. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368, 2024a.
  33. J. Wei, C. Yang, X. Song, Y. Lu, N. Hu, D. Tran, D. Peng, R. Liu, D. Huang, C. Du, et al. Long-form factuality in large language models. arXiv preprint arXiv:2403.18802, 2024b.
  34. W. Xu, G. Zhu, X. Zhao, L. Pan, L. Li, and W. Wang. Pride and prejudice: LLM amplifies self-bias in self-refinement. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15474–15492,…
  35. J. Ye, Y. Wang, Y. Huang, D. Chen, Q. Zhang, N. Moniz, T. Gao, W. Geyer, C. Huang, P.-Y. Chen, N. V. Chawla, and X. Zhang. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736, 2024.
  36. W. Zhao, T. Goyal, Y. Y. Chiu, L. Jiang, B. Newman, A. Ravichander, K. Chandu, R. L. Bras, C. Cardie, Y. Deng, et al. WildHallucinations: Evaluating long-form factuality in LLMs with real-world entity queries. arXiv preprint arXiv:2407.17468, 2024a. — [arXiv:2407.17468](https://arxiv.org/abs/2407.17468) find
  37. Y. Zhao, J. Zhang, I. Chern, S. Gao, P. Liu, J. He, et al. FELM: Benchmarking factuality evaluation of large language models. Advances in Neural Information Processing Systems, 36, 2024b.
  38. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Informati…
  39. Z. Zhu, Y. Yang, and Z. Sun. HaluEval-Wild: Evaluating hallucinations of language models in the wild. arXiv preprint arXiv:2403.04307, 2024. — [arXiv:2403.04307](https://arxiv.org/abs/2403.04307) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jacovi-2025-facts-grounding-leaderboard-benchmarking-llms/