Elamur Research⌕
☰
home / papers / 2025 / VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts

VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts

Paper · 2025 · Xin Liu, Lechen Zhang, Sheza Munir et al.

Summary

Judging whether long LLM answers are factually correct is hard because individual claims often depend on context spread across several sentences. Existing checkers such as SAFE split a response into standalone claims and verify each one, but they tend to drop conditions, comparisons, and causal or temporal links. An annotation study of 451 SAFE-extracted facts confirmed these gaps.

The authors propose VeriFact, which keeps the decompose-decontextualize-verify pipeline but adds a stage in which several LLM judges flag incomplete or missing facts. Flagged facts are rewritten into self-contained statements and then checked against Google Search evidence.

They also release FactRBench, which provides reference fact sets so that recall can be measured alongside precision. VeriFact produces noticeably fewer incomplete and missing facts. Benchmarking twelve LLMs shows that precision and recall can diverge. This matters because a terse answer can be perfectly accurate yet leave out most of the relevant information.

Key points

  • Annotation study: Responses from GPT4o-mini and Llama3.1-8B to 14 FactBench prompts were analysed. Of SAFE's extracted facts, 12.68% were incomplete, and about 1.24 relational facts were missing per response (out of roughly 24.5 extracted).

    • Causes of incompleteness: omitted conditions (51%), ambiguous entities (38%), missing comparanda (6%).

  • VeriFact results: It reduces incomplete facts by 19.2% compared with the strongest baseline. Its refinement stage cuts missing facts by 37%.

  • FactRBench queries: These combine in-the-wild FactBench prompts with Reddit questions (e.g., r/askscience) posted after January 2024, which limits pretraining leakage.

  • FactRBench references: Reference facts come from two sources:

    • VeriFact-verified outputs of Claude3.5-Sonnet, GPT-4o, Gemini-1.5 and Llama3.1-405B-instruct;
    • highly upvoted human answers from Reddit.

    The full retrieved webpages are included to make verification reproducible.

  • Benchmark of 12 LLMs (9 open-weight, 3 closed):

    • Larger models within a family generally score higher on both precision and recall.
    • Closed models tend to lead on recall.
    • Large open models such as Mistral-123B, Llama3.1-405B and Qwen2.5-72B are competitive, especially on precision.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of MichiganAnn Arbor · United States86 works · 15,493 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSociology and Political Science · 626 works

Cite

Cite as Xin Liu, Lechen Zhang, Sheza Munir, Yiyang Gu and Lu Wang. 2025. VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.905

@inproceedings{liu2025verifact,
  title = {{VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts}},
  author = {Xin Liu, Lechen Zhang, Sheza Munir, Yiyang Gu and Lu Wang},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.905},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.905}
}

References

14 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2024. Claude 3.5 sonnet. Available at: https://www.anthropic.com/news/ claude-3-5-sonnet. tational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pages 3751–3768. Association for Computational Linguistics. ↗
  2. Qisheng Hu, Quanyu Long, and Wenya Wang. 2024. Decomposition dilemmas: Does claim decomposition boost or burden fact-checking performance? CoRR, abs/2411.02400. find
  3. Aaron Hurst, Adam, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Madry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Al… — GPT-4o System Card
  4. Eleni Miltsakaki, Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2004. The Penn Discourse Treebank. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC’04), Lisbon, Portugal. European Language Resources Association (ELRA).
  5. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural L…
  6. Mistral AI. 2024. Large enough. Accessed: 2025-05-09. find
  7. Mistral AI. 2025. Mistral small 3.1. Accessed: 2025-05-09. find
  8. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean-Baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sotti… — Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
  9. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  10. Yuxia Wang, Revanth Gangi Reddy, Zain Muhammad Mujahid, Arnav Arora, Aleksandr Rubashevskii, Jiahui Geng, Osama Mohammed Afzal, Liangming Pan, Nadav Borenstein, Aditya Pillai, Isabelle Augenstein, Iryna Gurevych, and Preslav Nakov. 2023. Factcheck-gpt: End-to-end fine-grained document-level fact-che… — [arXiv:2311.09000](https://arxiv.org/abs/2311.09000) find
  11. Junyi Li, Xiaoxue Cheng, Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464, Singapore. Association for Computa…
  12. Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. 2024. Long-form factuality in large language models.
  13. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
  14. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2025-verifact-enhancing-long-form-factuality/