Elamur Research⌕
☰
home / papers / 2025 / MedGapGab at AraHealthQA: Modular LLM Assignment for Gaps and Gabs in Arabic Medical Question Answering

MedGapGab at AraHealthQA: Modular LLM Assignment for Gaps and Gabs in Arabic Medical Question Answering

Paper · 2025 · Baraa Hikal

Summary

This paper describes the winning entry for AraHealthQA Track 2, Sub-task 2, a shared task on generative Arabic medical QA. The task mixes two kinds of input: fill-in-the-blank items that ask for a missing medical term ("gaps"), and free-form patient questions taken from a doctor forum that need explanatory, safety-aware replies ("gabs"). The author's system, MedGapGab, first uses a simple rule to decide which kind of question it is, based on whether a blank placeholder is present. It then pulls the four most similar development-set examples of the same kind using TF-IDF similarity and inserts them into a prompt template written for that kind. Finally, it sends the question to whichever LLM scored best on that kind during development: Gemini 2.5 Flash for gaps and DeepSeek V3 for gabs. On the blind test set, the system reached 87.26% BERTScore and took first place. The work shows that choosing models per sub-type, guided by per-type development scores, can outperform any single model in low-resource specialised domains.

Key points

  • Data: the MedArabiQ dataset has 700 development instances and 200 test instances, of which 100 belong to Sub-task 2. Sources are Arabic medical school exams and notes (gaps) and the AraMed forum (gabs). The text is mostly MSA with some dialect, and a grammatically corrected parallel version is also provided.
  • Four LLMs were compared on the development set: Gemini 2.5 Flash, Qwen 3, DeepSeek V3 and Claude 4.
    • Gemini 2.5 Flash scored best on gaps (88.73).
    • DeepSeek V3 scored best on gabs (87.68).
    • Claude 4 was weak on gabs (82.54) because its outputs were overly cautious.
  • On the development set, the modular setup averaged 88.21. The best single model (DeepSeek V3) averaged 86.91, which is 1.30 points lower. Swapping Claude in for Gemini on gaps cost 0.86 points.
  • Blind test BERTScore by configuration:
    • Modular Gemini + DeepSeek: 87.26.
    • DeepSeek V3 alone: 86.85.
    • Claude 4 + DeepSeek V3: 86.40.
  • Evaluation used the F1 variant of BERTScore computed with a multilingual checkpoint. The task also lists BLEU and ROUGE as metrics. Only the provided development data was used, with no external resources.

Interesting ideas

  • I1. Route by question type: A cheap surface rule (here, the presence of a blank) splits the input into question types, and each type then gets its own prompt and its own engine instead of one model for everything.
  • I2. Choose the teacher per subset: The best model is picked separately for each subset using development scores, rather than by a single overall leaderboard number. The paper shows that no one model wins every subset.
  • I3. Pull examples only from the same type: In-context examples are restricted to the same question type before ranking them by similarity. This keeps the few-shot context matched in format to the target question.
  • I4. Pair routes with a contrast baseline: Each combined route is reported alongside a matched alternative, such as swapping one component or using a single model, and the averaged gain is checked against that alternative. On the test set, one mixed pairing actually scored below the single-model baseline.
  • I5. Use type-specific answer contracts: Gap prompts demand a single term, while gab prompts ask for reasoning and safety framing. The expected output shape is fixed per type.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Readers could first classify a test item by format, such as a cloze, a word problem or an open question about a fable, and send it to a dedicated solver with its own constraints.
  • I2 (learn): LLM teacher tiers could be assigned per phenomenon or per text genre after measuring each teacher on a small gold slice, rather than trusting one teacher across all phenomena.
  • I3 (learn): When prompting an LLM teacher for silver frames or repairs, examples could be drawn from sentences with the same construction, for example by matching on UD tree shape instead of on words.
  • I4 (learn): Every pipeline-composition change could be checked against a swap-one-component control, so that a gain is credited to the right module.
  • I5 (back): Generation from the meaning graph could pick an output template per question type, such as a bare entity for slot questions or an explained sentence for why-questions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of GöttingenGöttingen · Germany20 works · 158 citations

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksBiochemistry, Genetics and Molecular Biology · 263 worksMolecular Biology · 260 worksMedicine · 275 worksHealth Informatics · 214 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksQuestion answering · 487 works

Cite

Cite as Baraa Hikal. 2025. MedGapGab at AraHealthQA: Modular LLM Assignment for Gaps and Gabs in Arabic Medical Question Answering. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.30

@inproceedings{hikal2025medgapgab,
  title = {{MedGapGab at AraHealthQA: Modular LLM Assignment for Gaps and Gabs in Arabic Medical Question Answering}},
  author = {Baraa Hikal},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.30},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.30}
}

References

18 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Official Blind Test Performance We submitted three configurations to the official AraHealthQA blind test set leaderboard. Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, and Farah E. Shamout. 2025a. MedArabiQ: Benchmarking Large Language Models on Arabic Medical Tasks. arXiv preprint arXiv:2505.03427. and Generative Artificial Intelligence for Medical Education: Potential Impact and Opportunity. Academic Medicine, 99(1):22– 27. ↗
  2. Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, and Farah E Shamout. 2025b. Medarabiq: Benchmarking large language models on arabic medical tasks. arXiv eprints, pages arXiv–2505.
  3. Yan Cai, Linlin Wang, Ye Wang, Gerard de Melo, Ya Zhang, Yanfeng Wang, and Liang He. 2023. MedBench: A Large-Scale Chinese Benchmark for Evaluating Medical Large Language Models. arXiv preprint arXiv:2312.12806. ↗
  4. Ashwag Alasmari, Sarah Alhumoud, and Waad Alshammari. 2024. AraMed: Arabic medical question answering using pretrained transformer language models. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OS-ACT) with Shared Tasks on Arabic LLMs Hallucination and Dialect to MSA Machine Translation @ LREC-COLING 2024, pages 50–56, Torino, Italia. ELRA and ICCL.
  5. Kareem Darwish, Nizar Habash, Mourad Abbas, Hend Al-Khalifa, Huseein T. Al-Natsheh, Samhaa R. El-Beltagy, Houda Bouamor, Karim Bouzoubaa, Violetta Cavalli-Sforza, Wassim El-Hajj, Mustafa Jarrar, and Hamdy Mubarak. 2021. A panoramic survey of natural language processing in the arab world. arXiv preprint arXiv:2011.12631. ↗
  6. Hassan Alhuzali, Ashwag Alasmari, and Hamad Alsaleh. 2024. MentalQA: An Annotated Arabic Corpus for Questions and Answers of Mental Healthcare. IEEE Access, 12:101155–101165. ↗
  7. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR). ↗
  8. Hassan Alhuzali, Ashwag Alasmari, and 1 others. 2025a. Overview of the AraHealthQA 2025 Shared Task: Comprehensive Arabic Health Question Answering. In Proceedings of the ArabicNLP 2025 Workshop (Shared Task Overview, to appear). find
  9. Hassan Alhuzali, Farah Shamout, Muhammad Abdul-Mageed, Chaimae Abouzahir, Mouath Abu-Daoud, Ashwag Alasmari, Walid Al-Eisawi, Renad Al-Monef, Ali Alqahtani, Lama Ayash, and 1 others. 2025b. Arahealthqa 2025 shared task description paper. arXiv preprint arXiv:2508.20047. ↗
  10. Mohammed M. Altuwaijri. 2011. Empowering patients and health professionals in the arab world: The king abdullah bin abdulaziz arabic health encyclopedia on the web. Yearbook of Medical Informatics, pages 125–129. ↗
  11. Christy K. Boscardin, Brian Gin, Polo B. Golde, and Karen E. Hauer. 2024. find
  12. ChatGPT Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What Disease Does This Patient Have? A Large-Scale Open-Domain Question Answering Dataset from Medical Exams. In Proceedings of the 35th AAAI Conference on Artificial Intelligence (AAAI 2021). Dataset available as MedQA (arXiv:2009.13081, 2020). ↗
  13. Ahmad A. Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, and Mariette Awad. 2024. AraSTEM: A Native Arabic Multiple Choice Question Benchmark for Evaluating LLMs’ Knowledge in STEM Subjects. arXiv preprint arXiv:2501.00559. ↗
  14. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmed, and et al. 2023. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774. ↗
  15. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. MedM-CQA: A Large-Scale Multi-Subject Multi-Choice Dataset for Medical Domain Question Answering. In Proceedings of the ACM Conference on Health, Inference, and Learning (CHIL), pages 248–260. find
  16. Malik Sallam, Israa M. Alasfoor, Shahad W. Khalid, Rand I. Al-Mulla, Amwaj Al-Farajat, Maad M. Mijwil, Reem Zahrawi, Mohammed Sallam, Jan Egger, and Ahmad S. Al-Adwan. 2025. Chinese generative ai models (deepseek and qwen) rival chatgpt-4 in ophthalmology queries with excellent performance in arabic and english. Narra J, 5(1):e2371. ↗
  17. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, Mike Schaekermann, Amy Wang, Mohamed Amin, Sami Lachgar, Philip Mansfield, Sushant Prakash, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, and 12 others. 2023. Towards Expert-Level Medical Question Answering with Large Language Models (Med-PaLM 2). Nature Medicine, 29(7):1455– 1463.
  18. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In Proceedings of the 8th International Conference on Learning Representations (ICLR). Prompt Library ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hikal-2025-medgapgab-arahealthqa-modular-llm-assignment/