Elamur Research⌕
☰
home / papers / 2026 / Towards Unified Factuality Evaluation for Biomedical QA and Summarization: Aligning Metrics with Clinical Use-Cases

Towards Unified Factuality Evaluation for Biomedical QA and Summarization: Aligning Metrics with Clinical Use-Cases

Paper · 2026 · Mahule Roy and Subhas Roy

Summary

Standard overlap metrics such as ROUGE, BLEU and BERTScore can give biomedical LLM outputs high marks while missing clinically important factual mistakes. The authors propose an evaluation framework that adds evidence-based checking to these reference-based scores. Each generated answer is split into atomic claims with spaCy. For every claim, the top five PubMed abstracts (2014–2024) are retrieved, and BioBERT-NLI, trained on MedNLI, labels the claim as supported, contradicted or unsupported. Separate rule-based checks against RxNorm and an FDA contraindication list flag dosage and medication hazards. The results are combined into a composite Fact-Aligned Score (FAS).

They tested Llama-2-13B-chat, Llama-2-7B-chat, Mistral-7B-Instruct-v0.2 and an encoder-only PubMedBERT baseline on BioASQ, PubMedQA and MedLFQA. Even the strongest model had a sizeable share of contradicted claims. On MedLFQA, FAS tracked expert claim-level labels much more closely than lexical metrics did. The work matters because rewarding surface overlap can hide errors that are unsafe in clinical use. An open-source toolkit with outputs and scripts is released.

Key points

  • FAS definition: FAS = 0.3·BERTScore + 0.4·Support@5 + 0.3·(1−Contra@5). It uses a 0.7 NLI confidence threshold, chosen through a sensitivity sweep from 0.5 to 0.9.
  • Benchmarks: BioASQ Task B, 10th edition (492 questions), PubMedQA (500 instances) and a 200-item MedLFQA subset. MedLFQA gold labels were derived from its Must Have / Nice to Have annotations.
  • Contradiction rates on BioASQ: Contra@5 ranged from 13.4% (Llama-2-13B, Support@5 71.2%) to 24.7% (PubMedBERT).
  • Correlation with gold labels on MedLFQA (Spearman ρ): FAS 0.68, Support@5 alone 0.61, ROUGE-L 0.41, BLEU-4 0.36.
  • Ablations: removing retrieval lowers ρ to 0.52. Using a general MNLI model instead of BioBERT-NLI lowers it to 0.45. Retrieving only a single abstract per claim lowers it to 0.56.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom104 works · 157,183 citations

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 3Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksQuestion answering · 441 worksSummarization · 232 works

Cite

Cite as Mahule Roy and Subhas Roy. 2026. Towards Unified Factuality Evaluation for Biomedical QA and Summarization: Aligning Metrics with Clinical Use-Cases. BioNLP 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.bionlp-1.2

@inproceedings{roy2026unified,
  title = {{Towards Unified Factuality Evaluation for Biomedical QA and Summarization: Aligning Metrics with Clinical Use-Cases}},
  author = {Mahule Roy and Subhas Roy},
  booktitle = {BioNLP 2026},
  year = {2026},
  doi = {10.18653/v1/2026.bionlp-1.2},
  url = {https://doi.org/10.18653/v1/2026.bionlp-1.2}
}

References

16 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nentidis, A., Katsimpras, G., Krithara, A., Lima López, S., Farré-Maduell, E., Gasco, L., ... & Paliouras, G. (2023, September). Overview of bioasq 2023: The eleventh bioasq challenge on large-scale biomedical semantic indexing and question answering. In International Conference of the Cross-Languag… find
  2. Bird, S., Klein, E., & Loper, E. (2009). Natural language processing with Python: analyzing text with the natural language toolkit. " O’Reilly Media, Inc.". find
  3. Cock, P. J., Antao, T., Chang, J. T., Chapman, B. A., Cox, C. J., Dalke, A., ... & De Hoon, M. J. (2009). Biopython: freely available Python tools for computational molecular biology and bioinformatics. Bioinformatics, 25(11), 1422. find
  4. Jin, Q., Dhingra, B., Liu, Z., Cohen, W., & Lu, X. (2019, November). Pubmedqa: A dataset for biomedical research question answering. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EM…
  5. Jeong, M., Hwang, H., Yoon, C., Lee, T., & Kang, J. (2024). Olaph: Improving factuality in biomedical long-form question answering. arXiv preprint arXiv:2405.12701. — [arXiv:2405.12701](https://arxiv.org/abs/2405.12701) find
  6. Shivade, C. (2017). Mednli-a natural language inference dataset for the clinical domain. find
  7. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2019). Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. find
  8. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., ... & Scialom, T. (2023). Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  9. Jiang, Y., Li, X., Zhu, G., Li, H., Deng, J., Han, K., ... & Zhang, R. (2023). 6G non-terrestrial networks enabled low-altitude economy: Opportunities and challenges. arXiv preprint arXiv:2311.09047. — [arXiv:2311.09047](https://arxiv.org/abs/2311.09047) find
  10. Gu, Y., Tinn, R., Cheng, H., Lucas, M., Usuyama, N., Liu, X., ... & Poon, H. (2021). Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1), 1-23.
  11. Efron, B., & Tibshirani, R. (1986). Bootstrap methods for standard errors, confidence intervals, and other measures of statistical accuracy. Statistical science, 54-75. find
  12. Honnibal, M., Montani, I., Van Landeghem, S., & Boyd, A. (2020). spaCy: Industrial-strength natural language processing in Python. find
  13. Liu, S., Ma, W., Moore, R., Ganesan, V., & Nelson, S. (2005). RxNorm: prescription for electronic drug information exchange. IT professional, 7(5), 17-23. find
  14. Papineni, K., Roukos, S., Ward, T., & Zhu, W. J. (2002, July). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics (pp. 311-318).
  15. Lin, C. Y. (2004, July). Rouge: A package for automatic evaluation of summaries. In Text summarization branches out (pp. 74-81).
  16. FDA, U. (2023). FDA adverse event reporting system (FAERS) public dashboard. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/roy-2026-unified-factuality-evaluation-biomedical-qa/