Elamur Research⌕
☰
home / papers / 2026 / Understanding LLMs' summarization capabilities: an analysis of biomedical abstract and lay summary generation

Understanding LLMs' summarization capabilities: an analysis of biomedical abstract and lay summary generation

Paper · 2026 · Batuhan Nursal and Cassie S. Mitchell

Summary

The paper asks how well small, deployable LLMs (at most 8B parameters) can write two kinds of biomedical summaries without fine-tuning: technical abstracts for researchers and plain-language lay summaries for non-experts. The authors prompt instruction-tuned Llama and Qwen models zero-shot on the eLife and PLOS corpora, using GPT-4o as a larger reference point. Models write both long and short versions from full articles with section headers removed. Outputs are scored for relevance, readability and factuality using standard metrics. The study also adds two new analyses. The first labels each generated sentence by the article section it draws from. The second measures "desirable" content by checking how many article-derived questions a summary can answer. Finally, Gemini 2.0 Flash and human volunteers rank the summaries. Overall, small LLMs convey roughly as much useful information as human-written references. Judges often prefer the model outputs and frequently cannot tell them apart from human text. This suggests inexpensive models are a realistic option for scalable scientific and public-facing summarization. The code is released on GitHub.

Key points

  • Data and models: eLife (4,828 articles) and PLOS (27,525 articles) were used with their original splits. The models were Llama 3.1-8B-Instruct, Llama 3.2-3B-Instruct, Qwen 2.5-3B-Instruct and a 7B Qwen 2.5 model. GPT-4o was run on the eLife test set only, because of cost.
  • Metrics: Relevance was measured with ROUGE and BERTScore, readability with FRE, CLI and DCRS, and factuality with BARTScore and AlignScore. Llama and Qwen scored higher on ROUGE, while GPT-4o led on BERTScore.
  • Sentence origins: Compared with the gold references, LLM summaries contain about twice as many Discussion-derived sentences and about half as many from the Introduction. Longer outputs include more Methods content.
  • Answerability: On eLife, GPT-4o's long summaries answered more questions than the gold references did. On PLOS, the gold references did better, with LLM outputs weakest on Results and Methods questions.
  • Judging (40 articles, 80 undergraduates): An LLM output was ranked first in 65% (LLM judge) and 85% (human judges) of abstract cases, and in 73% and 83% of lay summary cases. Humans labeled 81% of LLM-generated abstracts and 56% of LLM-generated lay summaries as human-written.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Georgia Institute of TechnologyAtlanta · United States133 works · 24,497 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSummarization · 232 works

Cite

Cite as Batuhan Nursal and Cassie S. Mitchell. 2026. Understanding LLMs' summarization capabilities: an analysis of biomedical abstract and lay summary generation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.554

@inproceedings{nursal2026understanding,
  title = {{Understanding LLMs' summarization capabilities: an analysis of biomedical abstract and lay summary generation}},
  author = {Batuhan Nursal and Cassie S. Mitchell},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.554},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.554}
}

References

10 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Irfan Al-Hussaini, Davi Nakajima An, Albert J. Lee, Sarah Bi, and Cassie S. Mitchell. 2022. Ccs explorer: Relevance prediction, extractive summarization, and named entity recognition from clinical cohort studies. In 2022 IEEE International Conference on Big Data (Big Data), page 5173–5181. IEEE. find
  2. Prasanth Bathala, Christophe Ye, Batuhan Nursal, Shubham Lohiya, David Kartchner, and Cassie S. Mitchell. 2025. BioEL: A comprehensive python package for biomedical entity linking. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 1709–1721, Albuquerque, New Mexico. Ass… find
  3. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer.
  4. Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, Jianye Hou, and Benyou Wang. 2024. Huatuogpt-o1, towards medical complex reasoning with llms. find
  5. Marcio Fonseca and Shay Cohen. 2024. Can large language model summarizers adapt to diverse scientific communication goals? In Findings of the Association for Computational Linguistics: ACL 2024, pages 8599–8618, Bangkok, Thailand. Association for Computational Linguistics. find
  6. Tomas Goldsack, Carolina Scarton, and Chenghua Lin. 2025. Leveraging large language models for zero-shot lay summarisation in biomedicine and beyond. find
  7. Tomas Goldsack, Carolina Scarton, Matthew Shardlow, and Chenghua Lin. 2024. Overview of the BioLay-Summ 2024 shared task on the lay summarization of biomedical research articles. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, pages 122–131, Bangkok, Thailand. Associat… find
  8. Tomas Goldsack, Zhihao Zhang, Chenghua Lin, and Carolina Scarton. 2022. Making science simple: Corpora for the lay summarisation of scientific literature. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10589–10604, Abu Dhabi, United Arab Emirates. As…
  9. Tomas Goldsack, Zhihao Zhang, Chen Tang, Carolina Scarton, and Chenghua Lin. 2023. Enhancing biomedical lay summarisation with external knowledge graphs. find
  10. Rita González-Márquez, Luca Schmidt, Benjamin M. Schmidt, Philipp Berens, and Dmitry Kobak. 2024. The landscape of biomedical research. Patterns, 5(6):100968. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nursal-2026-understanding-llms-summarization-capabilities-analysis/