Elamur Research⌕
☰
home / papers / 2026 / Lakefront AI Ramblers at MEDIQA-SYNUR 2026: Hybrid Retrieval and LLM Verification for Open-Source Schema-Guided Clinical Information Extraction

Lakefront AI Ramblers at MEDIQA-SYNUR 2026: Hybrid Retrieval and LLM Verification for Open-Source Schema-Guided Clinical Information Extraction

Paper · 2026 · Michael T. Saban, Arsalan Yaghoubi, Behnaz Eslami et al.

Summary

The paper describes a system for the MEDIQA-SYNUR 2026 shared task. The task turns synthetic nursing dictations into structured JSON observations tied to a fixed clinical schema of 193 concepts. Scoring uses strict exact matching. The authors start from the official baseline, which segments transcripts, retrieves schema items and extracts with an LLM, and add three extensions:

  • Hybrid retrieval: a weighted blend of dense embedding scores and BM25 scores to pick candidate concepts for each segment.
  • Fine-tuning: LoRA-based supervised fine-tuning of open-weight Llama models, trained on transcript-level targets.
  • Post-processing: deterministic schema validation, optional multi-run voting, and a separate LLM verifier that keeps or drops each extracted observation.

Hybrid retrieval brought schema coverage close to complete. Verification gave the largest F1 gains, especially for smaller fine-tuned models. On the test set, the best open-source submission reached 0.711 F1, while GPT-4o reached 0.791. The work shows that open models can come reasonably close to closed ones for privacy-sensitive clinical extraction. It also shows that post-processing gains on development data do not reliably carry over to test data.

Key points

  • The dataset has 122 training, 101 development and 199 test transcripts. Transcripts are split at each "[Clinician]" tag.
  • Retrieval combines BAAI/bge-base-en-v1.5 with BM25 (α=0.6), keeping the top 60 schema items per segment. This gives recall@60 of 0.994 on the development set.
  • The extractors were GPT-4o, GPT-4o-mini, Llama-3-8B-Instruct and Llama-3.3-70B-Instruct. The 70B model was fine-tuned with 4-bit QLoRA, and all training ran on a single NVIDIA H200.
  • The best test submission was Llama-3.3-70B-Instruct-SFT with union voting and GPT-4o-mini verification, at 0.711 F1. After the competition, Llama-3-8B-Instruct-SFT trained on train + dev reached 0.723 F1.
  • On the development set, verification by Llama-3.3-70B raised Llama-3-8B-SFT to 0.776 F1. GPT-4o (about 0.84 F1 on dev) did not benefit from post-processing.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Loyola University ChicagoChicago · United States2 works · 1 citation

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksBiochemistry, Genetics and Molecular Biology · 263 worksMolecular Biology · 260 works

🧭 Topics 2Research subjects and languages the work is about.Information retrieval & search · 552 worksInformation extraction · 175 works

Cite

Cite as Michael T. Saban, Arsalan Yaghoubi, Behnaz Eslami, Samie Tootooni and Dmitriy Dligach. 2026. Lakefront AI Ramblers at MEDIQA-SYNUR 2026: Hybrid Retrieval and LLM Verification for Open-Source Schema-Guided Clinical Information Extraction. Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2jfca3zi8p4o

@inproceedings{saban2026lakefront,
  title = {{Lakefront AI Ramblers at MEDIQA-SYNUR 2026: Hybrid Retrieval and LLM Verification for Open-Source Schema-Guided Clinical Information Extraction}},
  author = {Michael T. Saban, Arsalan Yaghoubi, Behnaz Eslami, Samie Tootooni and Dmitriy Dligach},
  booktitle = {Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026},
  year = {2026},
  doi = {10.63317/2jfca3zi8p4o},
  url = {https://doi.org/10.63317/2jfca3zi8p4o}
}

References

24 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., & Anadkat, S. (2023). GPT-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Bernal, J., & Mazo, C. (2022). Transparency of artificial intelligence in healthcare: insights from professionals in computing and healthcare worldwide. Applied Sciences, 12(20), 10228. find
  3. Bruch, S., Gai, S., & Ingber, A. (2023). An analysis of fusion functions for hybrid retrieval. ACM Transactions on Information Systems, 42(1), 1-35. find
  4. Chen, X., Lakhotia, K., Oguz, B., Gupta, A., Lewis, P., Peshterliev, S., Mehdad, Y., Gupta, S., & Yih, W.-t. (2022). Salient phrase aware dense retrieval: can a dense retriever imitate a sparse one? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 250-262. find
  5. Corbeil, J.-P., Abacha, A. B., Michalopoulos, G., Swazinna, P., Del-Agua, M., Tremblay, J., Daniel, A. J., Bader, C., Cho, K., & Krishnan, P. (2025). Empowering healthcare practitioners with language models: Structuring speech transcripts in two real-world clinical applications. In Proceedings of th… find
  6. Demsash, A. W., Kassie, S. Y., Dubale, A. T., Chereka, A. A., Ngusie, H. S., Hunde, M. K., Emanu, M. D., Shibabaw, A. A., & Walle, A. D. (2023). Health professionals’ routine practice documentation and its associated factors in a resource-limited setting: a cross-sectional study. BMJ health & care i… find
  7. Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36, 10088-10115.
  8. Fu, S., Chen, D., He, H., Liu, S., Moon, S., Peterson, K. J., Shen, F., Wang, L., Wang, Y., & Wen, A. (2020). Clinical concept extraction: a methodology review. Journal of biomedical informatics, 109, 103526. find
  9. Gao, L., Dai, Z., & Callan, J. (2021). COIL: Revisit exact lexical match in information retrieval with contextualized inverted list. arXiv preprint arXiv:2104.07186.
  10. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., & Vaughan, A. (2024). The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  11. Hein, D., Christie, A., Holcomb, M., Xie, B., Jain, A., Vento, J., Rakheja, N., Shakur, A. H., Christley, S., & Cowell, L. G. (2025). Prompts to table: specification and iterative refinement for clinical information extraction with large language models. medRxiv. find
  12. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), pages.
  13. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., & Amodei, D. (2020). Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  14. Lin, S.-C., & Lin, J. (2023). A dense representation framework for lexical and semantic matching. ACM Transactions on Information Systems, 41(4), 1-29. find
  15. Mandikal, P., & Mooney, R. (2024). Sparse meets dense: A hybrid approach to enhance scientific document retrieval. arXiv preprint arXiv:2401.04055. — [arXiv:2401.04055](https://arxiv.org/abs/2401.04055) find
  16. Michalopoulos, G., Corbeil, J.-P., Bader, C., Bodenstab, N., & Ben Abacha, A. (2026). Overview of the MEDIQA-SYNUR 2026 Shared Task on Observation Extraction from Nurse Dictations. In Proceedings of the 8th Clinical Natural Language Processing Workshop, ClinicalNLP@LREC 2026, pages, Palma, Mallorca,…
  17. Riedemann, L., Labonne, M., & Gilbert, S. (2024). The path forward for large language models in medicine is open. npj Digital Medicine, 7(1), 339. find
  18. Robertson, S., & Zaragoza, H. (2009). The probabilistic relevance framework: BM25 and beyond. Foundations and Trends in Information Retrieval, 3(4), 333-389. find
  19. Sezgin, E., Hussain, S.-A., Rust, S., & Huang, Y. (2023). Extracting medical information from free-text and unstructured patient-generated health data using natural language processing methods: feasibility study with real-world data. JMIR Formative Research, 7, e43014. find
  20. Song, K., Tan, X., Qin, T., Lu, J., & Liu, T.-Y. (2020). MPNet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33, 16857-16867.
  21. Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., & Zhou, M. (2020). MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in neural information processing systems, 33, 5776-5788. find
  22. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., & Zhou, D. (2022). Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171.
  23. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., & Funtowicz, M. (2020). Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, p…
  24. Xiao, S., Liu, Z., Zhang, P., Muennighoff, N., Lian, D., & Nie, J.-Y. (2024). C-Pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641-649.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/saban-2026-lakefront-ai-ramblers-mediqa-synur/