Elamur Research⌕
☰
home / papers / 2026 / Polimi at CRF Filling 2026: Prompt-Based Information Extraction from Italian Clinical Notes

Polimi at CRF Filling 2026: Prompt-Based Information Extraction from Italian Clinical Notes

Paper · 2026 · Vittorio Torri and Francesca Ieva

Summary

The paper presents the Polimi team's system for the CRF Filling Shared Task 2026. The task is to fill a Case Report Form of 134 clinical items (symptoms, vital signs, history, tests and treatments) from short emergency-department notes. Labelled data are very scarce, and about 95% of item values are unknown. The team worked only on the Italian notes. They used open-weight LLMs that run locally, so patient text never leaves hospital infrastructure. Extraction is zero-shot and split into separate prompts for binary, categorical and measured items, whose JSON outputs are then merged. Each prompt is enriched with an abbreviation glossary mined from unlabelled notes. Hand-written deterministic rules then normalise values, filter unsupported positives and recover missed labels. Two extra stages were tested but dropped because they lowered overall F1: an LLM-based verifier and a verifier guided by synthetic examples. With Mistral Small 3.2 24B, the system reached 67.51% F1 on development data and 63% on the test set. This placed third overall and second on Italian, showing that locally hosted models can be competitive for clinical extraction in a language other than English.

Key points

  • Data: 10 training notes, 80 development notes, 200 test notes and 2,667 unlabelled notes. The organisers' 80 synthetic notes were not used because their labels did not match the main schema.
  • Models compared: LLaMA 3.1 8B, LLaMA 3.3 70B FP8, Qwen 2.5 72B AWQ, Mistral-Small-3.2-24B-Instruct and an ensemble of these. GPT-4.1 mini and GPT-5.4 served as closed-source reference points.
  • Development F1: the zero-shot baseline scored 52.84% with Llama 3.1 8B and 57.78% with Llama 3.3 70B FP8. The best configuration, built on Mistral, reached 67.51%.
  • Glossary construction:
    • A regex captured uppercase tokens of 2–6 letters, and the 1,000 most frequent were kept.
    • An LLM proposed a meaning and a confidence score for each.
    • Acronyms with confidence below 0.20 or a wrong interpretation (e.g., EON, CE, NAO, APR, GB) went into the prompts.
  • Implementation: inference ran with vLLM using JSON-constrained structured decoding at temperature 0 on a single H100 or A100/V100 GPU. Code is published on GitHub (vittot/CRF-Filling-SharedTask-CL4Health26).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Politecnico di MilanoMilan · Italy5 works · 258 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 1Research subjects and languages the work is about.Information extraction · 167 works

Cite

Cite as Vittorio Torri and Francesca Ieva. 2026. Polimi at CRF Filling 2026: Prompt-Based Information Extraction from Italian Clinical Notes. Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/5moj6trbujsb

@inproceedings{torri2026polimi,
  title = {{Polimi at CRF Filling 2026: Prompt-Based Information Extraction from Italian Clinical Notes}},
  author = {Vittorio Torri and Francesca Ieva},
  booktitle = {Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026},
  year = {2026},
  doi = {10.63317/5moj6trbujsb},
  url = {https://doi.org/10.63317/5moj6trbujsb}
}

References

19 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Pietro Ferrazzi, Soumitra Ghosh, Alberto Lavelli, and Bernardo Magnini. 2026. Overview of the crf 2026 shared task on clinical case report forms filling. In Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health), Palma, Mallorca (Spain). ELRA. find
  3. Raffaello Fornasiere, Nicolò Brunello, Vincenzo Scotti, and Mark Carman. 2024. Medical information extraction with large language models. In Proceedings of the 7th International Conference on Natural Language and Speech Processing (ICNLSP 2024), pages 456–466. find
  4. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  5. Danqing Hu, Bing Liu, Xiaofeng Zhu, Xudong Lu, and Nan Wu. 2024. Zero-shot information extraction from radiological reports using chatgpt. International Journal of Medical Informatics, 183:105321. find
  6. Albert Q Jiang, A Sablayrolles, A Mensch, C Bamford, D Singh Chaplot, Ddl Casas, F Bressand, G Lengyel, G Lample, L Saulnier, et al. 2023. Mistral 7b. arxiv. arXiv preprint arXiv:2310.06825, 10:3. find
  7. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th symposium on operating systems principles, pages 611–626.
  8. Saskia Locke, Anthony Bashall, Sarah Al-Adely, John Moore, Anthony Wilson, and Gareth B Kitchen. 2021. Natural language processing in medicine: a review. Trends in Anaesthesia and Critical Care, 38:4–9. find
  9. Xiao Luo, Fattah Muhammad Tahabi, Tressica Marc, Laura Ann Haunert, and Susan Storey. 2024. Zero-shot learning to extract assessment criteria and medical services from the preventive healthcare guidelines using large language models. Journal of the American Medical Informatics Association, 31(8):174… find
  10. Aishik Nagar, Viktor Schlegel, Thanh-Tung Nguyen, Hao Li, Yuping Wu, Kuluhan Binici, and Stefan Winkler. 2025. Llms are not zero-shot reasoners for biomedical information extraction. In The Sixth Workshop on Insights from Negative Results in NLP, pages 106–120. find
  11. Aurélie Névéol, Hercules Dalianis, Sumithra Velupillai, Guergana Savova, and Pierre Zweigenbaum. 2018. Clinical natural language processing in languages other than english: opportunities and challenges. Journal of biomedical semantics, 9(1):12. find
  12. Aaditya Singh, Adam Fry, Adam Perelman, Adam Tart, Adi Ganesh, Ahmed El-Kishky, Aidan McLaughlin, Aiden Low, AJ Ostrow, Akhila Ananthram, et al. 2025. Openai gpt-5 system card. arXiv preprint arXiv:2601.03267.
  13. Aneeta Sylolypavan, Derek Sleeman, Honghan Wu, and Malcolm Sim. 2023. The impact of inconsistent human annotations on ai driven clinical decision making. NPJ Digital Medicine, 6(1):26. find
  14. Yanshan Wang, Liwei Wang, Majid Rastegar-Mojarad, Sungrim Moon, Feichen Shen, Naveed Afzal, Sijia Liu, Yuqun Zeng, Saeed Mehrabi, Sunghwan Sohn, et al. 2018. Clinical information extraction applications: a literature review. Journal of biomedical informatics, 77:34–49. find
  15. Qiang Wei, Amy Franklin, Trevor Cohen, and Hua Xu. 2018. Clinical text annotation–what factors are associated with the cost of time? In AMIA Annual Symposium Proceedings, volume 2018, page 1552. find
  16. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388. 7.
  17. Language Resource References Pietro Ferrazzi, Mattia Franzin, Alberto Lavelli, and Bernardo Magnini. 2026. Small llms for medical nlp: a systematic analysis of few-shot, constraint decoding, fine-tuning and continual pre-training in italian. find
  18. Pietro Ferrazzi, Alberto Lavelli, and Bernardo Magnini. 2025. Converting annotated clinical cases into structured case report forms. In Proceedings of the 24th Workshop on Biomedical Language Processing, pages 307–318, Viena, Austria. Association for Computational Linguistics. find
  19. Gabriela Anna Kaczmarek, Pietro Ferrazzi, Lorenzo Porta, Vicky Rubini, and Bernardo Magnini. 2026. Toward automatic filling of case report forms: A case study on data from an italian emergency department. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/torri-2026-polimi-crf-filling-2026-prompt/