Elamur Research⌕
☰
home / papers / 2025 / mucAI at AraHealthQA 2025: Explain–Retrieve–Verify (ERV) Workflow for Multi-Label Arabic Health QA Classification

mucAI at AraHealthQA 2025: Explain–Retrieve–Verify (ERV) Workflow for Multi-Label Arabic Health QA Classification

Paper · 2025 · Ahmed Abdou

Summary

The paper addresses multi-label classification for Track 1 of AraHealthQA 2025 (MentalQA). The task has two parts: tagging Arabic mental-health questions with seven question types, and tagging doctors' answers with three communication strategies. The data is the MentalQA dataset, 500 question–answer pairs from Altibbi.com, split 300/50/150. The author proposes ERV (Explain–Retrieve–Verify), a training-free workflow built in DSPy on top of GPT-4. It runs in three steps:

  1. The LLM reasons step by step and proposes labels, a rationale and a self-confidence score.
  2. The system fetches the most similar training items using a multilingual Sentence-BERT model, and an LLM infers likely labels from their gold tags.
  3. A verifier LLM weighs both sources against the full taxonomy and outputs the final labels, a confidence score and a short note explaining how conflicts were settled.

For questions, ERV beats fine-tuned MARBERT and AraBERT (weighted F1 0.61 vs 0.56). For answers it trails them (0.73 vs 0.76). A hybrid that hands only the Emotional Support label to ERV reaches 0.80. The work shows that retrieval-backed LLM reasoning can recover rare classes that small-data fine-tuning ignores, and it leaves a human-readable decision trail.

Key points

  • Dataset: MentalQA, built from Altibbi.com posts from 2020–2021.

    • Question labels: seven categories, namely Diagnosis (A), Treatment (B), Anatomy & Physiology (C), Epidemiology (D), Healthy Lifestyle (E), Provider Choice (F) and Other (Z).
    • Answer labels: 1 Information, 2 Direct Guidance, 3 Emotional Support.
    • Metrics: weighted F1 and instance-based Jaccard.

  • Setup:

    • ERV uses GPT-4, embeddings from paraphrase-multilingual-mpnet-base-v2, and cosine kNN over cached training embeddings.
    • Baselines are MARBERT and AraBERT-v02, fine-tuned with lr 2×10⁻⁵, batch 16, 5 epochs and weight decay 0.01.

  • Question results (test): ERV scores 0.61 F1 / 0.53 Jaccard; both baselines score 0.56 / 0.51.

  • Answer results (test):

    SystemF1Jaccard
    MARBERT0.760.73
    AraBERT0.740.68
    ERV0.730.61
    MARBERT+ERV hybrid0.800.72

  • Per-label findings:

    • MARBERT scores zero F1 on C, F and Emotional Support. ERV reaches 0.20, 0.15 and 0.44 on these.
    • ERV leans toward recall, for example 0.99 recall on Direct Guidance.
    • The main cost is three sequential LLM calls per item.

Interesting ideas

  • I1. Two independent evidence channels (the model's own reasoning and the gold labels of nearest training cases) are merged by a verifier that must write down how it resolved each disagreement, so the decision stays auditable.
  • I2. The hybrid routes labels rather than averaging models: the strong classifier keeps the majority labels, and a different system takes over only the label where the main one scores zero.
  • I3. Nearest labelled neighbours act as case-based priors that rescue minority classes in tiny datasets, and the author suggests interpolating a fine-tuned model with kNN at inference.
  • I4. A per-class precision/recall breakdown exposed labels with zero F1 that the weighted averages hid, which changed the system design.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): role or speech-act assignment could run a rule-based reading and a nearest-annotated-example reading separately, with a reconciler logging why one reading won.
  • I2 (answer): question types on which the main reader systematically fails could be routed to a dedicated fallback reader, keeping each path's reasoning visible.
  • I3 (tree): the perceptron parser could be blended with kNN over treebank contexts to improve rare deprels where training counts are too small.
  • I4 (learn): evaluation reports should always list per-label scores and flag any zero-F1 class before trusting an aggregate number.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 worksPsychology · 1,649 worksSocial Psychology · 330 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 3,066 worksQuestion answering · 487 works

Cite

Cite as Ahmed Abdou. 2025. mucAI at AraHealthQA 2025: Explain–Retrieve–Verify (ERV) Workflow for Multi-Label Arabic Health QA Classification. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.32

@inproceedings{abdou2025mucai,
  title = {{mucAI at AraHealthQA 2025: Explain–Retrieve–Verify (ERV) Workflow for Multi-Label Arabic Health QA Classification}},
  author = {Ahmed Abdou},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.32},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.32}
}

References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Abdul-Mageed, AbdelRahim Elmadany, and El Moatez Billah Nagoudi. 2020. Arbert & marbert: Deep bidirectional transformers for arabic. arXiv preprint arXiv:2101.01785. ↗
  2. Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, and Farah E Shamout. 2025. Medarabiq: Benchmarking large language models on arabic medical tasks. arXiv e-prints, pages arXiv–2505.
  3. Hassan Alhuzali, Ashwag Alasmari, and Hamad Al-saleh. 2024. Mentalqa: An annotated arabic corpus for questions and answers of mental healthcare. IEEE Access. ↗
  4. Hassan Alhuzali, Farah Shamout, Muhammad Abdul-Mageed, Chaimae Abouzahir, Mouath Abu-Daoud, Ashwag Alasmari, Walid Al-Eisawi, Renad Al-Monef, Ali Alqahtani, Lama Ayash, Nizar Habash, and Leen Kharouf. 2025. Arahealthqa 2025 shared task description paper. In Proceedings of ArabicNLP 2025. find
  5. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. ↗
  6. Akshay Swaminathan, Iván López, Rafael Antonio Garcia Mar, Tyler Heist, Tom McClintock, Kaitlin Caoili, Madeline Grace, Matthew Rubashkin, Michael N Boggs, Jonathan H Chen, and 1 others. 2023. Natural language processing system for rapid detection and intervention of mental health crisis chat messages. NPJ digital medicine, 6(1):213. find
  7. Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Keloth, and 1 others. 2025. Medical foundation large language models for comprehensive text analysis and beyond. npj Digital Medicine, 8(1):141. find
  8. Kailai Yang, Shaoxiong Ji, Tianlin Zhang, Qianqian Xie, Ziyan Kuang, and Sophia Ananiadou. 2023. Towards interpretable mental health analysis with large language models. arXiv preprint arXiv:2304.03347. ↗
  9. Monica Zolezzi, Maha Alamri, Shahd Shaar, and Daniel Rainkie. 2018. Stigma associated with mental illness and its treatment in the arab culture: a systematic review. International Journal of Social Psychiatry, 64(6):597–609. ↗
  10. A Thushari Atapattu, Mahen Herath, Charitha Elvitigala, Piyanjali de Zoysa, Kasun Gunawardana, Menasha Thilakaratne, Kasun De Zoysa, and Katrina Falkner. 2022. Emoment: An emotion annotated mental health corpus from two south asian countries. arXiv preprint arXiv:2208.08486. ↗
  11. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  12. Jaya Chaturvedi, Sumithra Velupillai, Robert Stewart, and Angus Roberts. 2023. Identifying mentions of pain in mental health records text: a natural language processing approach. arXiv preprint arXiv:2304.01240. ↗
  13. Soumitra Ghosh, Asif Ekbal, and Pushpak Bhattacharyya. 2020. Cease, a corpus of emotion annotated suicide notes in english. In Proceedings of the twelfth language resources and evaluation conference, pages 1618–1626.
  14. Imane Guellil, Houda Saâdane, Faical Azouaou, Billel Gueni, and Damien Nouvel. 2021. Arabic natural language processing: An overview. Journal of King Saud University-Computer and Information Sciences, 33(5):497–507.
  15. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abdou-2025-mucai-arahealthqa-2025-explain-retrieve/