📄 Structured Radiology Intelligence: Extracting Structured Data from MRI Reports Using LLMs

Summary

The paper tackles the conversion of free-text doctor notes on MRI reports into a standardized JSON record, so the data can feed hospital information systems, clinical decision support and large-scale analysis. The authors, from IIIT Hyderabad and CMC Vellore, gathered 630 de-identified notes from an academic health center. Clinicians defined 55 variables, and the examples suggest a rectal cancer focus: tumour measurements, CRM/MRF involvement, EMVI, TNM stage and MR TRG.

Annotations were built step by step. LLM drafts were checked by the authors and then by doctors in a custom web tool, and prompts were revised whenever clinicians disagreed with the output. Experiments ran in three stages: baseline, intermediate and fine-tuned. They compared zero-, one- and few-shot settings with one-question, multi-question and all-question prompts across LLaMA, Qwen and DeepSeek models. LLaMA 3.1 8B Instruct ranked best in the later stages, and after fine-tuning it reached a composite score of 87.36%.

The work matters because labeled resources pairing unstructured radiology notes with structured outputs are scarce. However, the dataset is not released for privacy reasons; only the model is shared.

Key points

• The data comes from 630 MRI doctor notes and is organized into dataset versions V1–V5:
  • 60 doctor-validated reports serve as the gold test set.
  • 65 reports supply prompt examples.
  • 300 validated reports are used for fine-tuning.
• With the basic one-question prompt, LLaMA 3.1 8B Instruct scored 61.16% against the validated data. Typical errors included:
  • spurious "not mentioned" labels;
  • repeated numbers;
  • inconsistent yes/no answers;
  • hallucinations.
• Under that baseline prompt, the seven models scored as follows:
  • Qwen3 4B Instruct-2507: 64.49% (best)
  • Qwen2.5 7B Instruct: 63.84%
  • DeepSeek R1 Distill Llama 8B: 63.01%
  • DeepSeek R1 Distill Qwen 7B: 53.94%
  • Llama 3.2 3B Instruct: 35.56%
  • MMed Llama 3 8B: 34.86%
• Enhanced prompts define how to extract each variable, including implicitly stated information. They use labels such as "Not Mentioned" and "Not Relevant" and group related variables together.
• Fine-tuned LLaMA 3.1 8B Instruct achieved the top composite score of 87.36%. The abstract reports an 18.42% improvement but does not state which stage it is measured against.