Elamur Research⌕
☰
home / papers / 2026 / Identifying Imaging Follow-Up in Radiology Reports: A Comparative Analysis of Traditional ML and LLM Approaches

Identifying Imaging Follow-Up in Radiology Reports: A Comparative Analysis of Traditional ML and LLM Approaches

Summary

Radiology reports often recommend follow-up imaging, but these recommendations frequently go unacted upon, and hospitals rarely track whether the recommended exam was ever done. Progress on automating this tracking has been held back by a shortage of suitable annotated data. The authors built such a corpus from a large de-identified clinical database. They chose CT and MRI "index" reports that recommend follow-up for a new lesion, then gathered every later report for the same patient as "candidate" reports. Medical students marked the earliest candidate that actually re-examines the flagged finding, with a radiologist resolving hard cases.

On this benchmark the authors compared:

  • classical classifiers (LR, SVM),
  • Longformer,
  • a fully fine-tuned Llama3-8B-Instruct,
  • prompted GPT-4o and GPT-OSS-20B, each in a plain setup and in an optimized one that restricts the input to metadata, recommendation sentences and nearby context.

The optimized generative models scored best, coming close to human agreement. The simple linear models remained competitive, which makes them practical low-cost options for clinical deployment.

Key points

  • Corpus size and sources: 6,393 reports from 586 patients, made up of 586 index reports and 5,807 candidates (about 9.9 per index). The source pool held 7 million reports from 959,382 patients covering 2007–2020.
  • Index report selection: Only CT and MRI index reports describing new lesions were included, spanning six body sites: the thyroid, pancreas, adrenal glands, liver, kidneys and lungs. Selection used an SVM recommendation-sentence classifier (0.87 F1) plus a BERT-based entity extractor.
  • Annotation: Inter-annotator agreement was 0.846 F1. 239 cases were double-annotated and 347 single-annotated. A follow-up report was found for 417 patients (71.3%).
  • Results: F1 scores were 0.832 for GPT-4o (Advanced), 0.828 for GPT-OSS-20B (Advanced), 0.776 for LR and 0.775 for SVM. 95% confidence intervals were obtained by bootstrapping.
  • Code: Available at github.com/uw-bionlp/ReportMatcher.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Language models & pretraining

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5eve2chrhdy5

@inproceedings{park2026identifying,
  title = {{Identifying Imaging Follow-Up in Radiology Reports: A Comparative Analysis of Traditional ML and LLM Approaches}},
  author = {Park, Namu and Ramachandran, Giridhar Kaushik and Lybarger, Kevin and Xia, Fei and Uzuner, Özlem and Gunn, Martin and Yetisgen, Meliha},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/5eve2chrhdy5},
  url = {https://doi.org/10.63317/5eve2chrhdy5}
}

References

28 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925. find our card
  2. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. find
  3. Leo Breiman. 2001. Random forests. Machine learning, 45:5–32. find
  4. Joanne L. Callen et al. 2012. Failure to follow-up test results for ambulatory patients: a systematic review. J Gen Intern Med, 27(10):1334–1348. find
  5. Emmanuel Carrodeguas, Ronilda Lacson, Whitney Swanson, and Ramin Khorasani. 2019. Use of machine learning to identify follow-up recommendations in radiology reports. Journal of the American College of Radiology, 16(3):336–343. find
  6. Sandeep Dalal, Vadiraj Hombal, Wei-Hung Weng, Gabe Mankovich, Thusitha Mabotuwana, Christopher S. Hall, Joseph Fuller, Bruce E. Lehnert, and Martin L. Gunn. 2020. Determining Follow-Up Imaging Study Using Radiology Reports. J Digit Imaging, 33(1):121–130. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, page… find
  8. Pierre Geurts, Damien Ernst, and Louis Wehenkel. 2006. Extremely randomized trees. Machine learning, 63:3–42. find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. find
  10. Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. 1998. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18–28. find
  11. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. find our card
  12. Nadja Kadom, Arjun K Venkatesh, Samantha A Shugarman, Judy H Burleson, Christopher L Moore, and David Seidenwurm. 2022. Novel quality measure set: closing the completion loop on radiology follow-up recommendations for non-critical actionable incidental findings. Journal of the American College of Ra… find
  13. Michal E. Kulon. 2016. Lost to Follow-Up : Automated Detection of Patients Who Missed Follow-Ups Which Were Recommended on Radiology Reports. find
  14. John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. 1(2):3. find
  15. Paul A. Larson et al. 2014. Actionable findings and the role of IT support: report of the ACR Actionable Reporting Work Group. J Am Coll Radiol, 11(6):552–558. find
  16. Wilson Lau, Kevin Lybarger, Martin L Gunn, and Meliha Yetisgen. 2022. Event-based clinical finding extraction from radiology reports with pre-trained language model. Journal of Digital Imaging, pages 1–14. find
  17. Wilson Lau, Thomas H Payne, Ozlem Uzuner, and Meliha Yetisgen. 2020. Extraction and analysis of clinically important follow-up recommendations in a large radiology dataset. AMIA Summits on Translational Science Proceedings, 2020:335. find
  18. Kahyun Lee, Nicholas J Dobbins, Bridget McInnes, Meliha Yetisgen, and Özlem Uzuner. 2021. Transferability of neural network clinical deidentification systems. Journal of the American Medical Informatics Association, 28(12):2661–2669. find
  19. Thusitha Mabotuwana, Christopher S Hall, Vadiraj Hombal, et al. 2019. Automated tracking of follow-up imaging recommendations. American Journal of Roentgenology, 212(6):1287–1294. find
  20. Thusitha Mabotuwana, Christopher S Hall, Joel Tieder, and Martin L. Gunn. 2018. Improving Quality of Follow-Up Imaging Recommendations in Radiology. AMIA Annu Symp Proc, 2017:1196–1204. find
  21. Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. 2025. Medvlm-r1: Incentivizing medical reasoning capability of vision-language models (vlms) via reinforcement learning. arXiv preprint arXiv:2502.19634. — [arXiv:2502.19634](https://arxiv.org/abs/2502.19634) find
  22. Namu Park, Kevin Lybarger, Giridhar Kaushik Ramachandran, Spencer Lewis, Aashka Damani, Ozlem Uzuner, Martin Gunn, and Meliha Yetisgen. 2024. A novel corpus of annotated medical imaging reports and information extraction results using bert-based language models. arXiv preprint arXiv:2403.18975. find
  23. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE transactions on Signal Processing, 45(11):2673–2681. find
  24. Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, et al. 2025. Medgemma technical report. arXiv preprint arXiv:2507.05201. find
  25. Thomas Sounack, Joshua Davis, Brigitte Durieux, Antoine Chaffin, Tom J Pollard, Eric Lehman, Alistair EW Johnson, Matthew McDermott, Tristan Naumann, and Charlotta Lindvall. 2025. Bioclinical modernbert: A state-of-the-art long-context encoder for biomedical and clinical nlp. arXiv preprint arXiv:25… — [arXiv:2506.10896](https://arxiv.org/abs/2506.10896) find
  26. Karen Sparck Jones. 1972. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1):11– 21. find
  27. Yan Xu, Junichi Tsujii, and Eric I-Chao Chang. 2012. Named entity recognition of follow-up and time information in 20 000 radiology reports. Journal of the American Medical Informatics Association, 19(5):792–799. find
  28. Meliha Yetisgen-Yildiz, Martin L Gunn, Fei Xia, and Thomas H Payne. 2011. Automatic identification of critical follow-up recommendation sentences in radiology reports. In AMIA Annual Symposium Proceedings, volume 2011, page 1593. American Medical Informatics Association. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/park-2026-identifying-imaging-follow-up-radiology/