Elamur Research⌕
☰
home / papers / 2025 / AraHealthQA 2025: The First Shared Task on Arabic Health Question Answering

AraHealthQA 2025: The First Shared Task on Arabic Health Question Answering

Paper · 2025 · Hassan Alhuzali, Walid Al-Eisawi, Muhammad Abdul-Mageed et al.

Summary

AraHealthQA 2025 is a shared task held with ArabicNLP 2025 at EMNLP 2025. It addresses the shortage of Arabic medical question-answering resources and standardized ways to compare systems. It has two tracks:

  • MentalQA uses 500 real patient questions paired with expert doctor answers on mental health. It has three subtasks: classifying the question type, classifying the answer strategy and generating answers.
  • MedArabiQ covers general medicine, such as internal medicine, pediatrics and medical education. It has a multiple-choice subtask and a fill-in-the-blank/open-ended subtask. The full original 700-question benchmark served as development data, and a freshly written, unseen set served for testing.

The organizers describe the data, splits, metrics, baselines and participating systems. In the multiple-choice subtask, the strongest reported system used an Arabic chain-of-thought prompt with Gemini Pro 2.5 and reached 0.76 accuracy. Across both tracks, teams did better on structured classification than on open generation. Automatic similarity metrics were judged too weak to measure whether answers were appropriate. The task gives Arabic health NLP its first common benchmark and points to the need for expert human evaluation.

Key points

  • The MentalQA subtasks use a 300 / 50 / 150 split (train / dev / blind test), and the same split applies to answer generation.
  • MentalQA uses multi-label classification:
    • The question-type labels span seven categories: treatment, diagnosis, epidemiology, anatomy & physiology, choice of provider, healthy lifestyle, and a catch-all "other."
    • Answer strategies are information provision, direct guidance and emotional support.
  • In MedArabiQ, the 700-question dev set was the whole original benchmark (Daoud et al., 2025). The test questions were similar in kind but new, and fine-tuning on task data was restricted.
  • The best multiple-choice system, by AlDahoul and Zaki, reported these accuracies:
    • Gemini Pro 2.5: 0.76.
    • Gemini Flash 2.5 and GPT-o3: 0.74 each.
    • The team also applied majority voting across models.
  • Zero-shot prompting sometimes beat few-shot prompting. Track 2 scores showed little spread across teams. BERTScore captured only surface overlap in open answers.

Interesting ideas

  • I1. Annotating each answer with the communicative strategy it uses (informing, directing, comforting), as a multi-label layer beside the question type, separates what is asked from how the reply responds.
  • I2. Releasing the entire original benchmark as dev data and testing on newly written look-alike questions checks whether systems truly generalize or only fit known items.
  • I3. A detailed Arabic zero-shot prompt beat a plain English one. It asked the model to reason step by step, explain the relevant concepts, rule out wrong options and rely on trusted facts.
  • I4. Few-shot examples sometimes hurt, apparently because unrelated examples pushed models toward hallucination. Example selection therefore matters more than example count.
  • I5. Embedding-based scores like BERTScore missed whether open answers were correct and appropriate, which motivated human-in-the-loop judging.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (frames): Speech-act analysis in prag could label answer sentences with such strategies, so readers can explain why a reply counts as advice versus information.
  • I2 (learn): Evaluation could keep a pool of freshly authored test items that mirror each known test set, as a guard against silent overfitting.
  • I3 (answer): School-test readers could produce an explicit elimination trace that gives a reason for rejecting each distractor, making multiple-choice answers explainable.
  • I4 (learn): LLM teacher prompts should use only closely matched examples, with a no-example control run to detect harm from demonstrations.
  • I5 (answer): Open-ended QA could be scored by matching the meaning graph of the answer against the reference instead of by vector similarity, with human spot checks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of British ColumbiaVancouver · Canada98 works · 34,805 citations🎓New York University Abu DhabiAbu Dhabi · United Arab Emirates122 works · 4,602 citations🎓Umm al-Qura UniversityMakkah al Mukarramah · Saudi Arabia21 works · 274 citations🎓King Khalid UniversityAbhā · Saudi Arabia4 works · 16 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksQuestion answering · 441 works

Cite

Cite as Hassan Alhuzali, Walid Al-Eisawi, Muhammad Abdul-Mageed, Chaimae Abouzahir, Mouath Abu-Daoud, Ashwag Alasmari, Renad Al-Monef, Ali Alqahtani, Lama Ayash, Leen Kharouf, Farah E. Shamout and Nizar Habash. 2025. AraHealthQA 2025: The First Shared Task on Arabic Health Question Answering. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.18

@inproceedings{alhuzali2025arahealthqa,
  title = {{AraHealthQA 2025: The First Shared Task on Arabic Health Question Answering}},
  author = {Hassan Alhuzali, Walid Al-Eisawi, Muhammad Abdul-Mageed, Chaimae Abouzahir, Mouath Abu-Daoud, Ashwag Alasmari, Renad Al-Monef, Ali Alqahtani, Lama Ayash, Leen Kharouf, Farah E. Shamout and Nizar Habash},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.18},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.18}
}

References

44 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdelaziz Amr AbdelAziz, Mohamed Ahmed Youssef, Mamdouh Mohamed Koritam, Marwa ELDeeb, and Ensaf Hussein. 2025. Retaug at arabic mental health question answering: A multi-task approach with advanced retrieval-augmented generation. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 1.
  2. Ahmed Abdou. 2025. mucai at arahealthqa 2025: Explain-retrieve-verify (erv) workflow for multi-label arabic health qa classification. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 1.
  3. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗
  4. Ali Al-Laith and Mamdouh Alenezi. 2021. Monitoring people’s emotions and symptoms from arabic tweets during the COVID-19 pandemic. Information (Basel), 12(2):86. ↗
  5. N Al-Musallam and M Al-Abdullatif. 2022. Depression detection through identifying depressive arabic tweets from saudi arabia: Machine learning approach. In 2022 Fifth National Conference of Saudi Computers Colleges (NCCC), pages 11–18. find
  6. Ashwag Alasmari. 2025. A scoping review of arabic natural language processing for mental health. In Healthcare, volume 13-9, page 963. MDPI. ↗
  7. Ashwag Alasmari, Sarah Alhumoud, and Waad Alshammari. 2024. Aramed: Arabic medical question answering using pretrained transformer language models. In Proceedings of the 6th Workshop on Open-Source Arabic Corpora and Processing Tools (OS-ACT), pages 50–56.
  8. Ashwag Alasmari, Luke Kudryashov, Shweta Yadav, Heera Lee, and Dina Demner-Fushman. 2023. Chq-socioemo: Identifying social and emotional support needs in consumer-health questions. Scientific Data, 10(1):329. find
  9. Nouar AlDahoul and Yasir Zaki. 2025. Nyuad at arahealthqa shared task: Benchmarking the medical understanding and reasoning of large language models in arabic healthcare tasks. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 2.
  10. Shahad Hathal Aldhafer and Mourad Yakhlef. 2022. Depression detection in arabic tweets using deep learning. In 2022 6th International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE). IEEE. find
  11. Hassan Alhuzali and Ashwag Alasmari. 2025. Pre-trained language models for mental health: An empirical study on arabic q&a classification. Healthcare, 13(9). ↗
  12. Hassan Alhuzali, Ashwag Alasmari, and Hamad Al-saleh. 2024. Mentalqa: An annotated arabic corpus for questions and answers of mental healthcare. IEEE Access, 12:101155–101165. ↗
  13. Hossam Amer, Rawan Tarek Taha, Gannat Ibrahim, and Ensaf Hussein Mohammed. 2025. Egyhealth at general arabic health qa (medarabiq): An enhanced rag framework with large scale arabic q&a medical data. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 2.
  14. Thushari Atapattu, Mahen Herath, Charitha Elvitigala, Piyanjali de Zoysa, Kasun Gunawardana, Menasha Thilakaratne, Kasun de Zoysa, and Katrina Falkner. 2022. Emoment: An emotion annotated mental health corpus from two south asian countries. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6991–7001. Mabrouka Bessghaier, Shimaa Ibrahim, Md. ↗
  15. Rafiul Biswas, and Wajdi Zaghouani. 2025. Marsadlab at arahealthqa: Hybrid contextual-lexical fusion with arabert for question and answer categorization. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 1.
  16. Sajib Bhattacharjee, Ratnajit Dhar, Kawsar Ahmed, and Mohammed Moshiul Hoque. 2025. Binary_bunch at arahealthqa track 1: Advancing multi-label question and answer categorization with data augmentation and fine-tuned transformer model. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 1. find
  17. Jaya Chaturvedi, Sumithra Velupillai, Robert Stewart, and Angus Roberts. 2023. Identifying mentions of pain in mental health records text: a natural language processing approach. arXiv preprint arXiv:2304.01240. ↗
  18. Adiba Fairooz Chowdhury and MD Sagor Chowdhury. 2025. Quasar at arahealthqa track 1: Leveraging zero-shot large language models for question and answer categorization in arabic mental health. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 1.
  19. Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, and Farah E. Shamout. 2025. Medarabiq: Benchmarking large language models on arabic medical tasks. Preprint, arXiv:2505.03427. ↗
  20. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. ↗
  21. Fatimah Emad Elden and Mumina Abukar. 2025. Sakinah-ai at mentalqa: A comparative study of few-shot, optimized, and ensemble methods for arabic mental health question classification. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 1.
  22. Fatima Emad Eldin and Mumina Abukar. 2025. Medlingua at medarabiq2025: Zero- and few-shot prompting of large language models for arabic medical qa. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 2.
  23. Nejood A. Bin Eshaq. 2025. Mindllm at arahealthqa 2025 track 1: Leveraging large language models for mental health question answering. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 1.
  24. Yanjun Gao, Dmitriy Dligach, Timothy Miller, John Caskey, Brihat Sharma, Matthew M. Churpek, and Majid Afshar. 2023. Dr.bench: Diagnostic reasoning benchmark for clinical natural language processing. Journal of Biomedical Informatics, 138:104286. ↗
  25. Muskan Garg, Manas Gaur, Raxit Goswami, and Sunghwan Sohn. 2023. Lost: A mental health dataset of low self-esteem in reddit posts. In 2023 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 3854–3859. IEEE. ↗
  26. Muskan Garg, Chandni Saxena, Sriparna Saha, Veena Krishnan, Ruchi Joshi, and Vijay Mago. 2022. Cams: An annotated corpus for causal analysis of mental health issues in social media posts. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 6387–6396. find
  27. Gemini Team: Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, and Damien Vincent et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ↗
  28. Soumitra Ghosh, Asif Ekbal, and Pushpak Bhattacharyya. 2020. Cease, a corpus of emotion annotated suicide notes in english. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1618–1626.
  29. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, and Akhil Mathur et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783. ↗
  30. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300. ↗
  31. Baraa Hikal. 2025. Medgapgab at arahealthqa: Modular llm assignment for gaps and gabs in arabic medical question answering. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 2.
  32. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Preprint, arXiv:2009.13081. ↗
  33. Muhammad Khubayeeb Kabir, Maisha Islam, Anika Nahian Binte Kabir, Adiba Haque, and Md Khalilur Rhaman. 2022. Detection of depression severity using bengali social media posts on mental health: Study using natural language processing techniques. JMIR Form. Res., 6(9):e36118. ↗
  34. AbdulRahman A. Morsy, Saad Mankarious, and Ayah Zirikly. 2025. Sindbad at arahealthqa track 1: Leveraging large language models for mental health q&a. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 1.
  35. Ahmad Mustapha, Hadi Al-Khansa, Hadi Al-Mubasher, Aya Mourad, Ranam Hamoud, Hasan El-Husseini, Marwah Al-Sakkaf, and Mariette Awad. 2024. Arastem: A native arabic multiple choice question benchmark for evaluating llms knowledge in stem subjects. Preprint, arXiv:2501.00559. ↗
  36. Aryan Rastogi, Qian Liu, and Erik Cambria. 2022. Stress detection from social media articles: New dataset benchmark and analytical study. In 2022 International Joint Conference on Neural Networks (IJCNN). IEEE. find
  37. Caroline Sabty, Mohamad Rasmy, Mohamed Eyad Badran, Nourhan Sakr, and Alia El Bolock. 2025. Fahmni at arahealthqa track 1: Multi-agent retrieval-augmented generation and multi-label classification for arabic mental health q&a. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 1.
  38. Guangyao Shen, Jia Jia, Liqiang Nie, Fuli Feng, Cunjun Zhang, Tianrui Hu, Tat-Seng Chua, and Wenwu Zhu. 2017. Depression detection via harvesting social media: A multimodal dictionary learning solution. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, pages 3838– 3844, California. International Joint Conferences on Artificial Intelligence Organization. find
  39. Hao Sun, Zhenru Lin, Chujie Zheng, Siyang Liu, and Minlie Huang. 2021. PsyQA: A chinese dataset for generating long counseling text for mental health support. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Stroudsburg, PA, USA. Association for Computational Linguistics. find
  40. Mohamed Tarek, Saif Ahmed, and Mohamed Basem. 2025. Msa at arahealthqa 2025 shared task: Enhancing llm performance for arabic clinical question answering through prompt engineering and ensemble learning. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 2. ↗
  41. Elsbeth Turcan and Kathleen Mckeown. 2019. Dreaddit: A reddit dataset for stress analysis in social media. In Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019), pages 97–107. ↗
  42. Zhen Xu, Sergio Escalera, Adrien Pavão, Magali Richard, Wei-Wei Tu, Quanming Yao, Huan Zhao, and Isabelle Guyon. 2022. Codabench: Flexible, easy-to-use, and reproducible meta-benchmark platform. Patterns, 3(7):100543. ↗
  43. Mohamed Zaytoon, Ahmed Salem, Ahmed Sakr, and Hossam Elkordi. 2025. Araminds at arahealthqa 2025: A retrieval-augmented generation system for fine-grained classification and answer generation of arabic mental health q&a. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), AraHealthQA Shared Task Track 1.
  44. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Preprint, arXiv:1904.09675. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alhuzali-2025-arahealthqa-2025-first-shared-task/