Elamur Research⌕
☰
home / papers / 2026 / Evaluating Professional Acceptability of LLM-Generated Systematic Review Summaries in Healthcare: Psychiatrists' Perspectives

Evaluating Professional Acceptability of LLM-Generated Systematic Review Summaries in Healthcare: Psychiatrists' Perspectives

Paper · 2026 · Paul Thompson, Artemis Boulogeorgou, Fotini Kaponi et al.

Summary

Cochrane systematic review abstracts have grown long (often approaching 1000 words), so busy clinicians struggle to appraise them quickly. The authors ask whether LLMs can shorten these abstracts while keeping them accurate and acceptable to professionals. They built an evaluation framework based on PRISMA-A and Cochrane handbook requirements. They then used it to create PsycSumEval, a dataset of 460 summaries that four LLMs produced for 115 mental-health Cochrane abstracts. Two psychiatrists rated every summary on nine content dimensions. For each dimension they gave an ordinal score. When a score fell short, they also noted the fix needed (add, remove or modify), marked the faulty summary sentences, linked them to the source abstract, and wrote a free-text rationale. The resource shows how experts judge compressed medical evidence and where those judgments agree or diverge. Its structured scores and rationales could help train explainable LLM judges. This matters because standard overlap metrics and current LLM-as-judge methods fail to reliably catch clinically important errors in medical summaries.

Key points

  • PsycSumEval: 115 Cochrane mental-health abstracts × 4 LLMs = 460 summaries. GPT-4o is among the models, as shown in the worked example. Two psychiatrists evaluated the summaries.
  • Nine dimensions: the four PICO elements (Population, Intervention, Comparator, Outcome), plus Methodology, Main Findings and Conclusions, among others. All are derived from PRISMA-A and Cochrane guidelines.
  • Annotation per dimension: up to four pieces of information, namely
    • an ordinal score,
    • a corrective action (add/remove/modify),
    • the problematic sentences, traced to the source abstract,
    • a rationale.
  • Motivation: the median length of Cochrane abstracts rose from 353 words (2000) to 838 words (2022) after PRISMA-A was introduced. Metrics such as ROUGE, BLEU, METEOR and BERTScore cannot verify compliance with clinically detailed criteria.
  • Example of errors: in the GPT-4o summary, typical problems were omitting the number of included studies and adding future-work recommendations that the source abstract does not support.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ManchesterManchester · United Kingdom87 works · 44,938 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Biochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 worksDecision Sciences · 209 worksMedicine · 241 worksHealth Informatics · 184 worksStatistics, Probability and Uncertainty · 61 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Paul Thompson, Artemis Boulogeorgou, Fotini Kaponi, Efstathia Soufleri and Sophia Ananiadou. 2026. Evaluating Professional Acceptability of LLM-Generated Systematic Review Summaries in Healthcare: Psychiatrists' Perspectives. Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4exjqeuaw9ti

@inproceedings{thompson2026evaluating,
  title = {{Evaluating Professional Acceptability of LLM-Generated Systematic Review Summaries in Healthcare: Psychiatrists' Perspectives}},
  author = {Paul Thompson, Artemis Boulogeorgou, Fotini Kaponi, Efstathia Soufleri and Sophia Ananiadou},
  booktitle = {Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026},
  year = {2026},
  doi = {10.63317/4exjqeuaw9ti},
  url = {https://doi.org/10.63317/4exjqeuaw9ti}
}

References

51 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gerd Antes, David Tovey, and Nancy Owens. 2022. Cochrane: Trusted evidence. informed decisions. better health. Systematic Reviews in Health Research: Meta-Analysis in Context, pages 413–423. find
  2. Felipe Arias-Russi, Carolina Salazar-Lara, and Rubén Manrique. 2025a. Bridging the gap in health literacy: Harnessing the power of large language models to generate plain language summaries from biomedical texts. In Proceedings of the Second Workshop on Patient-Oriented Language Processing (CL4Healt… find
  3. Felipe Arias-Russi, Carolina Salazar Lara, and Ruben Manrique. 2025b. A multi-agent framework with diagnostic feedback for iterative plain language summary generation from cochrane medical abstracts. In Proceedings of the Fourth Workshop on Text Simplification, Accessibility and Readability (TSAR 20… find
  4. Howard Balshem, Mark Helfand, Holger J. Schünemann, Andrew D. Oxman, Regina Kunz, Jan Brozek, Gunn E. Vist, Yngve Falck-Ytter, Joerg Meerpohl, Susan Norris, and Gordon H. Guyatt. 2011. GRADE guidelines: 3. Rating the quality of evidence. Journal of Clinical Epidemiology, 64(4):401–406. find
  5. Lydie Bednarczyk, Daniel Reichenpfader, Christophe Gaudet-Blavignac, Amon Kenna Ette, Jamil Zaghir, Yuanyuan Zheng, Adel Bensahla, Mina Bjelogrlic, and Christian Lovis. 2025. Scientific evidence for clinical text summarization using large language models: scoping review. Journal of Medical Internet… find
  6. Elaine M Beller, Paul P Glasziou, Douglas G Altman, Sally Hopewell, Hilda Bastian, Iain Chalmers, Peter C Gøtzsche, Toby Lasserson, David Tovey, and PRISMA for Abstracts Group. 2013. PRISMA for abstracts: reporting systematic reviews in journal and conference abstracts. PLoS medicine, 10(4):e1001419… find
  7. Elaine M Beller, Paul P Glasziou, Sally Hopewell, and Douglas G Altman. 2011. Reporting of effect direction and size in abstracts of systematic reviews. Jama, 306(18):1981–1982. find
  8. BE Belsher, E Beech, D Evatt, DJ Smolenski, MT Shea, JL Otto, CS Rosen, and PP Schnurr. Present‐centered therapy (pct) for 2019. post‐traumatic stress disorder (ptsd) in adults. Cochrane Database of Systematic Reviews, (11). find
  9. Iain Chalmers and Douglas G Altman. 1995. Systematic reviews. BMJ Publishing London. find
  10. Bohao Chu, Meijie Li, Sameh Frihat, Chengyu Gu, Georg Lodde, Elisabeth Livingstone, and Norbert Fuhr. 2025. TracSum: A new benchmark for aspect-based summarization with sentence-level traceability in medical domain. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Proce… find
  11. Declan Devane, Johanna Pope, Paula Byrne, Evan Forde, Isabel O’Byrne, Steven Woloshin, Eileen Culloty, Darren Dahly, Ingeborg Hess Elgersma, Heather Munthe-Kaas, Conor Judge, Martin O’Donnell, Finn Krewer, Sandra Galvin, Nikita N. Burke, Theresa Tierney, KM Saif-Ur-Rahman, Tom Conway, and James Thom… find
  12. Kay Dickersin, Roberta Scherer, and Carol Lefebvre. 1994. Identifying relevant studies for systematic reviews. BMJ, 309(6964):1286–1291. find
  13. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled AlpacaEval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. find
  14. Zelalem Gero, Chandan Singh, Yiqing Xie, Sheng Zhang, Praveen Subramanian, Paul Vozila, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2024. Attribute structuring improves LLM-based evaluation of clinical text summaries. arXiv preprint arXiv:2403.01002. find
  15. Jasmin Helbach, Falk Hoffmann, Dawid Pieper, and Katharina Allers. 2023a. Reporting according to the preferred reporting items for systematic reviews and meta-analyses for abstracts (PRISMA-A) depends on abstract length. Journal of Clinical Epidemiology, 154:167–177. find
  16. Jasmin Helbach, Kathrin Wandscher, Dawid Pieper, and Falk Hoffmann. 2023b. Cochrane review abstracts are getting longer, but this has no large impact on the reporting quality. Journal of Evidence-Based Medicine, 16(3):294–302. find
  17. JPT Higgins, J Thomas, J Chandler, M Cumpston, T Li, MJ Page, and VA Welch. 2024. Cochrane Handbook for Systematic Reviews of Interventions version 6.5 (updated August 2024). Cochrane. find
  18. Zhenyu Huang, Xianlai Chen, Yunbo Wang, Jincai Huang, and Xing Zhao. 2025. A survey on biomedical automatic text summarization with large language models. Information Processing & Management, 62(5):104216. find
  19. Sagar Jose, Khanh T.P Nguyen, Kamal Medjaher, Ryad Zemouri, Mélanie Lévesque, and Antoine Tahan. 2024. Advancing multimodal diagnostics: Integrating industrial textual data and domain knowledge with large language models. Expert Systems with Applications, 255:124603. find
  20. Sebastian Joseph, Lily Chen, Jan Trienes, Hannah Göke, Monika Coers, Wei Xu, Byron Wallace, and Junyi Jessy Li. 2024. FactPICO: Factuality evaluation for plain language summarization of medical evidence. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu… find
  21. Hita Kambhamettu, Jamie Flores, and Andrew Head. 2025. Traceable texts and their effects: A study of summary-source links in ai-generated summaries. In Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–7. find
  22. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations.
  23. Mahira Kirmani, Gagandeep Kour, Mudasir Mohd, Nasrullah Sheikh, Dawood Ashraf Khan, Zahid Maqbool, Mohsin Altaf Wani, and Abid Hussain Wani. 2024. Biomedical semantic text summarizer. BMC Bioinformatics, 25(1):152. find
  24. Alon Lavie and Abhaya Agarwal. 2007. ME-TEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, pages 228– 231, Prague, Czech Republic. Association for Computational Linguistics. find
  25. Alessandro Liberati, Douglas G Altman, Jennifer Tetzlaff, Cynthia Mulrow, Peter C Gøtzsche, John PA Ioannidis, Mike Clarke, Philip J Devereaux, Jos Kleijnen, and David Moher. 2009. find
  26. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ, 339. find
  27. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  28. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2024. Factual consistency evaluation of summarization in the era of large language models. Expert Systems with Applications, 254:124456. find
  29. Katarina Maticic, Marina Krnic Martinic, and Livia Puljak. 2019. Assessment of reporting quality of abstracts of systematic reviews with meta-analysis using PRISMA-A and discordance in assessments between raters without prior experience. BMC Medical Research Methodology, 19(1):32. find
  30. Sally Morton, Alfred Berg, Laura Levit, and Jill Eden. 2011. Finding what works in health care: standards for systematic reviews. National Academies Press. find
  31. Ian G Needleman. 2002. A guide to systematic reviews. Journal of clinical periodontology, 29:6– 9. find
  32. Matthew J Page, Joanne E McKenzie, Patrick M Bossuyt, Isabelle Boutron, Tammy C Hoffmann, Cynthia D Mulrow, Larissa Shamseer, Jennifer M Tetzlaff, Elie A Akl, Sue E Brennan, et al. 2021a. find
  33. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ, 372. find
  34. W Scott Richardson, Mark C Wilson, Jim Nishikawa, and Robert S Hayward. 1995. The well-built clinical question: a key to evidence-based decisions. ACP Journal Club, 123(3):A12–13. find
  35. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. 2022. Self-critiquing models for assisting human evaluators. arXiv preprint arXiv:2206.05802.
  36. Liyan Tang, Zhaoyi Sun, Betina Idnay, Jordan G Nestor, Ali Soroush, Pierre A Elias, Ziyang Xu, Ying Ding, Greg Durrett, Justin F Rousseau, et al. 2023. Evaluating large language models on medical evidence summarization. NPJ Digital Medicine, 6(1):158. find
  37. Amy Y Tsou and Jonathan R Treadwell. 2016. Quality and clarity in systematic review abstracts: an empirical study. Research synthesis methods, 7(4):447–458. find
  38. Dave Van Veen, Cara Van Uden, Louis Blankemeier, Jean-Benoit Delbrouck, Asad Aali, Christian Bluethgen, Anuj Pareek, Malgorzata Polacin, Eduardo Pontes Reis, Anna Seehofnerová, et al. 2024. Adapted large language models can outperform medical experts in clinical text summarization. Nature Medicine,…
  39. Yuriy Vasilev, Irina Raznitsyna, Anastasia Pamova, Tikhon Burtsev, Tatiana Bobrovskaya, Pavel Kosov, Anton Vladzymyrskyy, Olga Omelyan-skaya, and Kirill Arzamasov. 2025. Evaluating medical text summaries using automatic evaluation metrics and LLM-as-a-judge approach: A pilot study. Diagnostics, 16(1… find
  40. Matthew J Page, David Moher, Patrick M Bossuyt, Isabelle Boutron, Tammy C Hoffmann, Cynthia D Mulrow, Larissa Shamseer, Jennifer M Tetzlaff, Elie A Akl, Sue E Brennan, et al. 2021b. find
  41. PRISMA 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ, 372. find
  42. Yuriy Vasilev, Irina Raznitsyna, Anastasia Pamova, Tikhon Burtsev, Tatiana Bobrovskaya, Pavel Kosov, Anton Vladzymyrskyy, Olga Omelyan-skaya, and Kirill Arzamasov. 2026. Evaluating medical text summaries using automatic evaluation metrics and llm-as-a-judge approach: A pilot study. Diagnostics, 16(1… find
  43. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  44. Qianqian Xie, Zheheng Luo, Benyou Wang, and Sophia Ananiadou. 2023. A survey for biomedical text summarization: From pre-trained to large language models. arXiv preprint arXiv:2304.08763. — [arXiv:2304.08763](https://arxiv.org/abs/2304.08763) find
  45. Libo Qin, Qiguang Chen, Xiachong Feng, Yang Wu, Yongheng Zhang, Yinghui Li, Min Li, Wanxiang Che, and Philip S Yu. 2024. Large language models meet NLP: A survey. arXiv preprint arXiv:2405.12819. — [arXiv:2405.12819](https://arxiv.org/abs/2405.12819) find
  46. Yiqing Xie, Sheng Zhang, Hao Cheng, Pengfei Liu, Zelalem Gero, Cliff Wong, Tristan Naumann, Hoifung Poon, and Carolyn Rose. 2024. DocLens: Multi-aspect fine-grained evaluation for medical text generation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol… find
  47. Fangyi Yu. 2025. When AIs judge AIs: The rise of agent-as-a-judge evaluation for LLMs. arXiv preprint arXiv:2508.02994. — [arXiv:2508.02994](https://arxiv.org/abs/2508.02994) find
  48. Gongbo Zhang, Qiao Jin, Yiliang Zhou, Song Wang, Betina Idnay, Yiming Luo, Elizabeth Park, Jordan G Nestor, Matthew E Spotnitz, Ali Soroush, et al. 2024. Closing the gap between open source and commercial large language models for medical evidence summarization. NPJ Digital Medicine, 7(1):239. find
  49. Haopeng Zhang, Philip S Yu, and Jiawei Zhang. 2025. A systematic survey of text summarization: From statistical methods to large language models. ACM Computing Surveys, 57(11):1–41. find
  50. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. BERTScore: Evaluating text generation with BERT. arXiv preprint arXiv:1904.09675. find
  51. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and Chatbot Arena. In Proceedings of the 37th International Conference on Neu…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/thompson-2026-evaluating-professional-acceptability-llm-generated/