Elamur Research⌕
☰
home / papers / 2026 / sebis at ArchEHR-QA 2026: How Much Can You Do Locally? Evaluating Grounded EHR QA on a Single Notebook

sebis at ArchEHR-QA 2026: How Much Can You Do Locally? Evaluating Grounded EHR QA on a Single Notebook

Paper · 2026 · Ibrahim Ebrar Yurt, Fabian Tobias Karl, Tejaswi Choppa et al.

Summary

Hospitals often cannot send patient records to cloud LLM services because of privacy rules such as HIPAA and GDPR, and many lack the infrastructure to host large models. This paper asks how well grounded question answering over electronic health records can work when every step runs on one ordinary notebook. The TUM sebis team competed in each of the four subtasks that make up the 2026 ArchEHR-QA shared task: question interpretation, evidence identification, answer generation, and evidence alignment. Training, inference, and evaluation were all done locally, with no external APIs.

The team compared three families of methods:

  • fine-tuned BERT-style classifiers, trained partly on synthetic cases produced by a locally run Llama model;
  • off-the-shelf dense embedding and retrieval models;
  • small or quantized generative LMs.

Their submissions scored above the task average in two subtasks. Smaller models came close to much larger systems when they were well configured. Untuned embedding models outperformed fine-tuned cross-encoders for selecting and aligning evidence. The results suggest that privacy-preserving clinical QA on commodity hardware is already practical.

Key points

  • Question interpretation: patient questions are condensed into clinician-style questions of at most 15 words.
    • Models used: Qwen3-4B, Qwen2.5-14B, and gpt-oss-120b, with 3- and 5-shot prompts.
    • Qwen2.5-14B outputs that ran too long were revised by Qwen3-4B in a two-step setup.
    • Query repetition was also tested.
  • Synthetic data: a local Llama3.1-70B produced 10 variants for each of the 20 development cases.
    • This yielded 200 cases with 1,818 sentences, each labelled essential, supplementary, or not-relevant.
    • Generation followed a two-stage generate-then-repair pipeline with fixed quality thresholds.
  • Evidence classifier: a BERT-style cross-encoder with two heads (3-way and binary), following HYDRA.
    • Real and synthetic data were balanced 1:1.
    • The configuration was chosen by case-level K-fold cross-validation.
  • Retrieval baselines: Qwen3-Embedding-8B and MedCPT-Cross-Encoder score each note sentence against the clinician question.
    • Decision thresholds were tuned on strict micro F1.
  • Output limits and code: answers are capped at 75 words and must cite sentences from the note; code is at github.com/ibrahimey/ArchEHR-QA-2026.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technical University of MunichMunich · Germany62 works · 48,970 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksMedicine · 275 worksHealth Informatics · 214 worksHealth Professions · 75 worksHealth Information Management · 56 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 487 works

Cite

Cite as Ibrahim Ebrar Yurt, Fabian Tobias Karl, Tejaswi Choppa and Florian Matthes. 2026. sebis at ArchEHR-QA 2026: How Much Can You Do Locally? Evaluating Grounded EHR QA on a Single Notebook. Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3mbfiuod7bfc

@inproceedings{yurt2026sebis,
  title = {{sebis at ArchEHR-QA 2026: How Much Can You Do Locally? Evaluating Grounded EHR QA on a Single Notebook}},
  author = {Ibrahim Ebrar Yurt, Fabian Tobias Karl, Tejaswi Choppa and Florian Matthes},
  booktitle = {Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health) @ LREC 2026},
  year = {2026},
  doi = {10.63317/3mbfiuod7bfc},
  url = {https://doi.org/10.63317/3mbfiuod7bfc}
}

References

34 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jayetri Bardhan, Kirk Roberts, and Daisy Zhe Wang. 2024. Question answering for electronic health records: scoping review of datasets and models. Journal of medical Internet research, 26:e53636. find
  2. Luka Blašković, Nikola Tanković, Ivan Lorencin, and Sandi Baressi Šegota. 2025. Robust clinical querying with local llms: Lexical challenges in nl2sql and retrieval-augmented qa on ehrs. Big Data and Cognitive Computing, 9(10):256. find
  3. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879, Vancouver, Canada. Association for Computational Lin…
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  5. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. find
  6. The llama 3 herd of models. arXiv:2407.21783. arXiv preprint Maxime Griot, Jean Vanderdonckt, and Demet Yuksel. 2025. Implementation of large language models in electronic health records. PLOS Digital Health, 4(12):1–18.
  7. Awni Hannun, Jagrit Digani, Angelos Katharopoulos, and Ronan Collobert. 2023. MLX: Efficient and flexible machine learning on apple silicon. find
  8. Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. Clinicalbert: Modeling clinical notes and predicting hospital readmission. CoRR, abs/1904.05342.
  9. Qiao Jin, Won Kim, Qingyu Chen, Donald C Comeau, Lana Yeganova, W John Wilbur, and Zhiyong Lu. 2023. Medcpt: Contrastive pre-trained transformers with large-scale pubmed search logs for zero-shot biomedical information retrieval. Bioinformatics, 39(11):btad651. find
  10. Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016. Mimic-iii, a freely accessible critical care database. Scientific data, 3(1):1–9.
  11. Jitendra Jonnagaddala and Zoie Shui-Yee Wong. 2025. Privacy preserving strategies for electronic health records in the era of large language models. npj Digital Medicine, 8(1):34. find
  12. Fabian Karl and Ansgar Scherp. 2023. Transformers are short-text classifiers. In Machine Learning and Knowledge Extraction - 7th IFIP TC 5, TC 12, WG 8.4, WG 8.9, WG 12.9 International Cross-Domain Conference, CD-MAKE 2023, Benevento, Italy, August 29 - September 1, 2023, Proceedings, volume 14065 o… find
  13. Fabian Karl and Ansgar Scherp. 2025. HYDRA: A multi-head encoder-only architecture for hierarchical text classification. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 9292–9303. Association for Computat… find
  14. Sunjun Kweon, Jiyoun Kim, Heeyoung Kwak, Dongchul Cha, Hangyul Yoon, Kwang Hyun Kim, Seunghyun Won, and Edward Choi. 2024. EHRNoteQA: A Patient-Specific Question Answering Benchmark for Evaluating Large Language Models in Clinical Settings. PhysioNet. Version 1.0.0. find
  15. Vojtech Lanz and Pavel Pecina. 2024. Paragraph retrieval for enhanced question answering in clinical documents. In Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, pages 580–590, Bangkok, Thailand. Association for Computational Linguistics. find
  16. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. Biobert: a pre-trained biomedical language representation model for biomedical text mining. CoRR, abs/1901.08746.
  17. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2025. Prompt repetition improves non-reasoning llms. arXiv preprint arXiv:2512.14982. find
  18. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  19. Femi Oke, Oyeneye Bolaji, and Michael Umakor. 2025. Building ai-ready infrastructure for us healthcare: a product management perspective. World J Adv Res Rev, 27(2):588–603. find
  20. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card.
  21. Anusri Pampari, Preethi Raghavan, Jennifer J. Liang, and Jian Peng. 2018. emrqa: A large corpus for question answering on electronic medical records. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 235…
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  23. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  24. Qwen Team. 2025. Qwen3 technical report. CoRR, abs/2505.09388.
  25. Qwen Team. 2026. Qwen3.5: Towards native multimodal agents. find
  26. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. CoRR, abs/1606.05250.
  27. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172–180.
  28. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. 2025. Toward expert-level medical question answering with large language models. Nature medicine, 31(3):943–950. find
  29. Sarvesh Soni and Dina Demner-Fushman. 2026a. A dataset for addressing patient’s information needs related to clinical course of hospitalization. Scientific Data. find
  30. Sarvesh Soni and Dina Demner-Fushman. 2026b. Overview of the archehr-qa 2026 shared task on grounded question answering from electronic health records. In Proceedings of the Third Workshop on Patient-Oriented Language Processing (CL4Health), Palma, Mallorca (Spain). ELRA. find
  31. Sarvesh Soni and Kirk Roberts. 2020. Evaluation of dataset selection for pre-training and fine-tuning transformer language models for clinical question answering. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 5532–5538, Marseille, France. European Language Resourc… find
  32. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415.
  33. Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin, and Meliha Yetisgen. 2023. Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Scientific data, 10(1):586. find
  34. Listing 2: Best-performing prompt template for Subtask 3: Answer Generation find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yurt-2026-sebis-archehr-qa-2026-how/