Elamur Research⌕
☰
home / papers / 2026 / Truth or Mirage? Towards End-To-End Factuality Evaluation with LLM-O asis

Truth or Mirage? Towards End-To-End Factuality Evaluation with LLM-O asis

Paper · 2026 · Alessandro Scirè, Andrei Stefan Bejgu, Simone Tedeschi et al.

Summary

LLMs still hallucinate, and their fluent output makes such errors easy to miss. Existing factuality benchmarks tend to be limited to one domain or task (e.g., news or dialogue summarization), are too small to train evaluators, or reduce the problem to checking single claims, as FEVER does. The authors present LLM-Oasis, a large resource for end-to-end factuality evaluation, where a whole natural-language passage must be judged true or false. It is built from passages taken from popular English Wikipedia pages. A single unified LLM prompt splits each passage into atomic claims, alters one claim so it becomes false, and writes two passages: a faithful paraphrase and a version that contains the falsified fact. Human annotators checked the quality of this pipeline and also built a gold-standard test set. Even strong LLMs find the benchmark hard. Smaller models trained on LLM-Oasis perform competitively. The method needs only raw text, so it could be applied to other corpora, domains, or languages.

Key points

  • Scale: about 81k factual/unfactual passage pairs (162,550 passages), with 681,201 claims drawn from 81,275 Wikipedia pages.
  • Tasks: two benchmark tasks are defined, end-to-end factuality evaluation and evidence-based claim verification.
  • Gold set: a manually curated test set of roughly 2k instances is used to benchmark state-of-the-art models.
  • Results: GPT-4o reaches only about 60% accuracy zero-shot and 68% with RAG.
  • Release: data and code are available at github.com/Babelscape/LLM-Oasis. The paper appeared in Computational Linguistics (2026) under a CC BY-NC-ND 4.0 license.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sapienza University of RomeRome · Italy57 works · 9,564 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Alessandro Scirè, Andrei Stefan Bejgu, Simone Tedeschi, Karim Ghonim, Federico Martelli and Roberto Navigli. 2026. Truth or Mirage? Towards End-To-End Factuality Evaluation with LLM-O asis. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli.a.575

@article{scire2026truth,
  title = {{Truth or Mirage? Towards End-To-End Factuality Evaluation with LLM-O asis}},
  author = {Alessandro Scirè, Andrei Stefan Bejgu, Simone Tedeschi, Karim Ghonim, Federico Martelli and Roberto Navigli},
  journal = {Computational Linguistics},
  year = {2026},
  doi = {10.1162/coli.a.575},
  url = {https://doi.org/10.1162/coli.a.575}
}

References

13 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdin, Marah, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, et al. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905. https://arxiv.org/abs/2412.08905 ↗
  2. Alhindi, Tariq, Savvas Petridis, and Smaranda Muresan. 2018. Where is your evidence: Improving fact-checking by justification modeling. In Proceedings of the First Workshop on Fact Extraction and VERification (FEVER), pages 85–90. https://doi.org/10.18653/v1/W18-5513 Alves, Duarte M., José Pombal,… ↗
  3. Goyal, Tanya, Junyi Li, and Greg Durrett. 2022. News summarization and evaluation in the era of GPT-3. arXiv preprint arXiv:2209.12356. https://doi.org/10.48550/arXiv.2209.12356 Grattafiori, Aaron, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhi… ↗
  4. Kamradt, Gregory. 2023. Needle in a Haystack – Pressure testing LLMs. https://github.com/gkamradt/LLMTest NeedleInAHaystack Karpukhin, Vladimir, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answe… ↗
  5. McNemar, Quinn. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. https://doi.org/10.1007/bf02295996 PubMed: 20254758 Microsoft, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao… ↗
  6. Min, Sewon, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural…
  7. Rasool, Zafaryab, Stefanus Kurniawan, Sherwin Balugo, Scott Barnett, Rajesh Vasa, Courtney Chesser, Benjamin M. Hampstead, Sylvie Belleville, Kon Mouzakis, and Alex Bahar-Fuchs. 2024. Evaluating LLMs on document-based QA: Exact answer selection and numerical extraction using CogTale dataset. Natural…
  8. Tam, Derek, Anisha Mascarenhas, Shiyue Zhang, Sarah Kwan, Mohit Bansal, and Colin Raffel. 2022. Evaluating the factual consistency of large language models through summarization. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5220–5255. https://doi.org/10.18653/v1 /202… find
  9. NAACL-HLT 2024: 4455–4480. https://doi.org/10.18653/v1/2024.naacl -long.251 Thorne, James, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: A large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of… ↗
  10. Zha, Yuheng, Yichi Yang, Ruichen Li, and Zhiting Hu. 2023. AlignScore: Evaluating factual consistency with a unified alignment function. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11328–11348. https://doi.org/10.18653/v1… ↗
  11. PMLR. Zhang, Shiyue and Mohit Bansal. 2021. Finding a balanced degree of automation for summary evaluation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6617–6632. https://doi.org/10.18653/v1/2021.emnlp-main.531 Zhang, Tianyi, Varsha Kishore, Feli… ↗
  12. ICLR 2020. https://arxiv.org/abs/1904.09675 ↗
  13. Zhang, Tianyi, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori Hashimoto. 2023. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39–57. https://doi.org/10 .1162/tacl a 00632 Zuo, Jingwei, Maksim Veli…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/scire-2026-truth-or-mirage-end-end/