Elamur Research⌕
☰
home / papers / 2025 / SciClaims: An End-to-End Generative System for Biomedical Claim Analysis

SciClaims: An End-to-End Generative System for Biomedical Claim Analysis

Paper · 2025 · Raul Ortega and Jose Manuel Gomez-Perez

Summary

Checking whether statements in scientific texts are backed by prior peer-reviewed work is central to systematic literature reviews and patent validation. Done manually, it is slow, error-prone and hard to scale. Earlier tools usually covered only part of the job, often assuming the claims were already identified. SciClaims is an interactive web demo that runs the whole chain in one system. It pulls claims out of raw text, finds candidate evidence among PubMed abstracts, and labels each claim-evidence pair as SUPPORT, REFUTE or NEI, with a natural-language rationale and highlighted key sentences. A single Llama3 8B Instruct model handles every generative step through prompting alone, with no fine-tuning, served via vLLM. Retrieval uses Elasticsearch. The system fits on one 24GB GPU. Against baselines on SciFact and under an LLM judge, it offers a strong balance of accuracy and speed. Human reviewers rated its outputs well. Because it is cheap to run and its decisions are explainable, it suits practical, high-stakes biomedical use. Code and a live demo are released.

Key points

  • The evidence corpus has 4.7 million PubMed abstracts (2000–2022), kept only if they have at least three Highly Influential Citations according to Semantic Scholar.
  • Claim extraction calls the LLM twice: once to draft claims and once to refine and filter them. On SciFact it reached a Rouge1 of 0.3387, edging out a sentence-tokenizer baseline (0.3313) and the methods of Pan et al. and Wright et al.
  • With Qwen2.5 72B AWQ as judge, the full SciClaims+CDP+CR pipeline scored 0.5020 at the system level, taking about 9 s per document. Of all tested combinations, only this setup fits on a single 24GB GPU.
  • Five NLP experts reviewed 154 PubMed documents:
    • 70.47% of claims satisfied all eight quality criteria.
    • In 60% of cases, the evidence that was retrieved proved adequate for checking the claim.
    • Label accuracy averaged 4.40 on a 5-point scale.
    • 74.61% of rationales were judged correct, with completeness being the weakest aspect.
  • The interface accepts inputs of up to 10,000 characters and offers more than 30 preloaded examples (papers, COVID news, social media, patents). Code is published on GitHub (expertailab/sciclaims-backend and expertailab/sciclaims-frontend).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

Cite

Cite as Raul Ortega and Jose Manuel Gomez-Perez. 2025. SciClaims: An End-to-End Generative System for Biomedical Claim Analysis. arXiv:2503.18526.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-demos.11 · arXiv 2503.18526

@misc{ortega2025sciclaims,
  title = {{SciClaims: An End-to-End Generative System for Biomedical Claim Analysis}},
  author = {Raul Ortega and Jose Manuel Gomez-Perez},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-demos.11},
  eprint = {2503.18526},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2025.emnlp-demos.11}
}

References

22 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. I-Chun Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, and Pengfei Liu. 2023. Factool: Factuality detection in generative ai – a tool augmented framework for multi-task and multi-domain scenarios. Preprint, arXiv:2307.13528.
  2. Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2021. Climate-fever: A dataset for verification of real-world climate claims. Preprint, arXiv:2012.00614.
  3. Islam Eldifrawi, Shengrui Wang, and Amine Trabelsi. 2024. Automated justification production for claim veracity in fact checking: A survey on architectures and approaches. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6679–6… find
  4. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao,… — The Llama 3 Herd of Models
  5. Nazanin Jafari and James Allan. 2024. Robust claim verification through fact detection. Preprint, arXiv:2407.18367. — [arXiv:2407.18367](https://arxiv.org/abs/2407.18367) find
  6. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. Preprint, arXiv:2004.12832.
  7. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. Preprint, arXiv:2309.06180.
  8. Xiangci Li, Gully Burns, and Nanyun Peng. 2021. A paragraph-level multi-task learning model for scientific fact-verification. Preprint, arXiv:2012.14500. — [arXiv:2012.14500](https://arxiv.org/abs/2012.14500) find
  9. Siting Liang and Daniel Sonntag. 2025. Explainable biomedical claim verification with large language models. find
  10. Hao Liu, Ali Soroush, Jordan G Nestor, Elizabeth Park, Betina Idnay, Yilu Fang, Jane Pan, Stan Liao, Marguerite Bernard, Yifan Peng, and Chunhua Weng. 2024. Retrieval augmented scientific claim verification. JAMIA Open, 7(1):ooae021. find
  11. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. Preprint, arXiv:1907.11692.
  12. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural L…
  13. Liangming Pan, Wenhu Chen, Wenhan Xiong, Min-Yen Kan, and William Yang Wang. 2021. Zero-shot fact verification by claim generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V… find
  14. Ronak Pradeep, Xueguang Ma, Rodrigo Nogueira, and Jimmy Lin. 2021. Scientific claim verification with VerT5erini. In Proceedings of the 12th International Workshop on Health Text Mining and Information Analysis, pages 94–103, online. Association for Computational Linguistics. find
  15. Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3:333–389. find
  16. Alessandro Scirè, Andrei Stefan Bejgu, Simone Tedeschi, Karim Ghonim, Federico Martelli, and Roberto Navigli. 2024. Truth or mirage? towards end-to-end factuality evaluation with llm-oasis. Preprint, arXiv:2411.19655. — [arXiv:2411.19655](https://arxiv.org/abs/2411.19655) find
  17. Marco Valenzuela, Vu Ha, and Oren Etzioni. 2015. Identifying meaningful citations. In AAAI Workshops. find
  18. Juraj Vladika, Ivana Hacajova, and Florian Matthes. 2025. Step-by-step fact verification system for medical claims with explainable reasoning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (V… find
  19. Juraj Vladika and Florian Matthes. 2023. Scientific fact-checking: A survey of resources and approaches. Preprint, arXiv:2305.16859. find
  20. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association…
  21. David Wadden, Kyle Lo, Lucy Lu Wang, Arman Cohan, Iz Beltagy, and Hannaneh Hajishirzi. 2022. Multivers: Improving scientific claim verification with weak supervision and full-document context. Preprint, arXiv:2112.01640. find
  22. Dustin Wright, David Wadden, Kyle Lo, Bailey Kuehl, Arman Cohan, Isabelle Augenstein, and Lucy Lu Wang. 2022. Generating scientific claims for zero-shot scientific fact checking. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ortega-2025-sciclaims-end-end-generative-system/