📄 Egyhealth at General Arabic Health QA (MedArabiQ): An Enhanced RAG Framework with Large-Scale Arabic Q&A Medical Data

Summary

This shared-task paper (AraHealthQA 2025, Track 2 on general medical QA) targets Arabic medical question answering. Two obstacles make the task hard: good Arabic data is scarce, and the language has heavy morphology, dialect variety and inconsistent spelling. The team merged two web-scraped Arabic health Q&A collections, AHD and MAQA, into a pool of roughly one million question–answer pairs. They cleaned the pool, normalized the orthography and aligned the two category schemes. The pool then serves as the knowledge store for a retrieval-augmented generation system. The pipeline has several stages:

• a Gemma model repairs typos in the incoming question;
• a fine-tuned AraBERT model predicts the likely medical specialties;
• retrieval runs only inside the top predicted specialties;
• a hybrid dense/lexical score reranks the hits;
• llama-3.3-70b-versatile writes the final answer.

Against a naïve RAG baseline, BERTScore rose from about 0.83 to 0.86. The gains in the retrieval metrics were small but consistent. The work shows that a large, noisy, scraped corpus can still serve as practical grounding for a low-resource medical chatbot if it is cleaned and searched in a structured way.

Key points

• Data:
  • AHD contributes 808,472 Q&A pairs in 45 categories.
  • MAQA contributes 273,174 pairs in 20 categories.
  • The 45 AHD categories were remapped onto MAQA's 20.
  • The task's development set, MedArabiQ, has 400 samples.
• Preprocessing:
  • Removed scraped junk such as links, "read more" phrases and image filenames.
  • Dropped entries written fully in a non-Arabic language.
  • Merged Alef variants into bare Alef, replaced final Yaa with its dotless form, and deleted Tatweel.
• Specialty classifier and embeddings:
  • The AraBERT specialty classifier reaches accuracy 0.7407 and F1 0.7379 over 20 classes.
  • Search is limited to the top 5 predicted classes, which roughly halves the search space.
  • Embeddings come from bert-base-arabic-camelbert-mix.
• Retrieval:
  • The system fetches 10 candidates and reranks them to 5 with a score of 0.7·cosine + 0.3·BM25.
  • On a 440-question test set, the full pipeline scores MRR 0.245 and Recall@5 0.259.
  • Basic retrieval scores MRR 0.188 and Recall@5 0.227.
  • Removing the classifier changes the scores only marginally (MRR 0.235).
• Generation:
  • Answers come from llama-3.3-70b-versatile at temperature 0.2.
  • On 100 mixed-format test questions, BERTScore is 0.8287 for naïve RAG and 0.8620 for the advanced RAG.

Interesting ideas

• I1. Because patient questions often span several specialties, the system filters on the top five predicted classes rather than trusting only the single best class, which keeps recall while still pruning.
• I2. Reranking mixes dense cosine similarity with BM25 under fixed, inspectable weights, so both lexical overlap and semantic closeness count and the contribution of each is easy to read.
• I3. The retrieval test set is built by having an LLM rephrase questions already in the index, balanced at 22 per category, so the probe checks robustness to wording rather than exact matching.
• I4. The ablation shows the classifier adds little accuracy but halves the search space, so the authors judge the component on efficiency as well as quality.
• I5. Two corpora with different taxonomies (45 vs 20 categories) were merged by mapping each fine label to its nearest coarse one before indexing.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (world): When resolving a word against compiled knowledge (WordNet senses, verb classes), Mova could keep the top few candidate domains as an explainable shortlist rather than committing to one label too early.
• I2 (answer): Fable and school-test readers could rank candidate supporting sentences by a transparent mix of lexical overlap and frame/graph similarity, logging each component as part of the explanation.
• I3 (learn): LLM-paraphrased variants of known items could test whether parses and frames stay stable under rewording, with unchanged originals and unrelated questions as controls.
• I4 (learn): Evaluation reports could record cost (search space, steps, time) beside accuracy for each module, so cheap pruning stages are not discarded just because their quality gain is small.
• I5 (world): A similar explicit crosswalk table could align FrameNet, VerbNet and Wikidata class inventories into one ontology layer, keeping the mapping auditable.