Elamur Research⌕
☰
home / papers / 2026 / LTRC-IIIT at MEDIQA-SYNUR 2026: Benchmarking a Fully Local, Training-Free RAG Pipeline

LTRC-IIIT at MEDIQA-SYNUR 2026: Benchmarking a Fully Local, Training-Free RAG Pipeline

Paper · 2026 · Aashwin Vaish and Dipti Misra Sharma

Summary

The paper describes a submission to the MEDIQA-SYNUR 2026 shared task, which asks systems to fill EHR flowsheet fields automatically from transcribed nurse dictations. The authors test how far a pipeline can go with no cloud APIs and no task-specific training, using only open models on local hardware. Their two-stage system first narrows hundreds of schema rows to likely candidates. It does this with all-MiniLM-L6-v2 embedding search, augmented by several heuristics: separate vectors for each Enum option, positive and negative phrasings for Yes/No fields, a forced lookup among NUMERIC fields whenever digits appear, comma-based chunking, and automatic inclusion of dependent fields such as units. An 8-bit Llama-3.1-8B-Instruct, constrained by a GBNF grammar in llama.cpp, then outputs JSON, and near-miss Enum values are mapped to valid options by similarity. The pipeline reaches F1 0.57, well below GPT-4o zero-shot (about 0.88). The authors present it as a negative-result baseline whose value lies in privacy, adaptation through schema changes alone, and a clear error analysis of compact LLMs.

Key points

  • The retrieval stage alone reaches recall 0.87 but precision only 0.35 (F1 0.50), by design. The LLM is relied on to discard irrelevant candidate rows.
  • End-to-end, the system scores precision 0.60, recall 0.55 and F1 0.57. The text also mentions 0.64 precision, an internal inconsistency. The top leaderboard F1 is 0.84, and the organizers' GPT-4o zero-shot result is about 0.88.
  • Manual review of 204 missed items on the validation set found two main error types. One is difficulty linking specific mentions to abstract STRING fields, such as "Sepsis" to "Secondary Diagnosis". The other is a completion bias, where the model fills in "No" or "None" for fields the transcript never mentions.
  • Grammar-constrained decoding ensures syntactically valid JSON every time. Paraphrased Enum outputs are corrected via cosine similarity, and values scoring below 0.6 are dropped.
  • The system runs on two NVIDIA 2080 Ti GPUs (about 22GB VRAM) and has only been tested on English transcripts with US-style schemas. The authors stress that it should be used only with human-in-the-loop review.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓International Institute of Information Technology, HyderabadHyderabad · India38 works · 1,821 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMedicine · 241 worksHealth Informatics · 184 works

Cite

Cite as Aashwin Vaish and Dipti Misra Sharma. 2026. LTRC-IIIT at MEDIQA-SYNUR 2026: Benchmarking a Fully Local, Training-Free RAG Pipeline. Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3wbuccufqusz

@inproceedings{vaish2026ltrc,
  title = {{LTRC-IIIT at MEDIQA-SYNUR 2026: Benchmarking a Fully Local, Training-Free RAG Pipeline}},
  author = {Aashwin Vaish and Dipti Misra Sharma},
  booktitle = {Proceedings of the 8th Workshop on Clinical Natural Language Processing (Clinical NLP) @ LREC 2026},
  year = {2026},
  doi = {10.63317/3wbuccufqusz},
  url = {https://doi.org/10.63317/3wbuccufqusz}
}

References

7 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jeffrey Budd. 2023. Burnout related to electronic health record use in primary care. Journal of primary care & community health, 14:21501319231166921. find
  2. Jean-Philippe Corbeil, Asma Ben Abacha, George Michalopoulos, Phillip Swazinna, Miguel Del-Agua, Jerome Tremblay, Akila Daniel, Cari Bader, Kevin Cho, Pooja Krishnan, Nathan Bodenstab, Thomas Lin, Wenxuan Teng, Francois Beaulieu, and Paul Vozila. 2025. Empowering healthcare practitioners with langua… find
  3. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, and Zhiwei Zhao. 20… — The Llama 3 Herd of Models
  4. Cari Bader Nate Bodenstab Asma Ben Abacha George Michalopoulos, Jean-Philippe Corbeil. 2026. Overview of the mediqa-synur 2026 shared task on observation extraction from nurse dictations. In Proceedings of the 8th Clinical Natural Language Processing Workshop, ClinicalNLP@LREC 2026, Palma, Mallorca,…
  5. Charaka Vinayak Kumar, Ashok Urlana, Gopichand Kanumolu, Bala Mallikarjunarao Garlapati, and Pruthwik Mishra. 2025. No llm is free from bias: A comprehensive study of bias evaluation in large language models. find
  6. Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks.
  7. Renato Umeton, Anne Kwok, Rahul Maurya, Domenic Leco, Naomi Lenane, Jennifer Willcox, Dana-Farber Committee, and Jason Johnson. 2023. Gpt-4 in a cancer center: Institute-wide deployment challenges and lessons learned. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vaish-2026-ltrc-iiit-mediqa-synur-2026/