Elamur Research⌕
☰
home / papers / 2025 / Fahmni at AraHealthQA Track 1: Multi-Agent Retrieval-Augmented Generation and Multi-Label Classification for Arabic Mental Health Q&A

Fahmni at AraHealthQA Track 1: Multi-Agent Retrieval-Augmented Generation and Multi-Label Classification for Arabic Mental Health Q&A

Paper · 2025 · Caroline Sabty, Mohamad Rasmy, Mohamed Eyad Badran et al.

Summary

Arabic mental-health NLP has very few resources. The AraHealthQA 2025 Track 1 shared task asks systems to do three things on the small MentalQA corpus: tag patient questions by type (seven classes), tag doctor replies by strategy (three classes), and write answers. The authors build FAHMNI, one modular system that covers all three subtasks.

For the two multi-label tagging problems, they compare Qwen3 and SILMA under several regimes:

  • zero-shot and few-shot prompting,
  • a shallow head over frozen representations,
  • PEFT,
  • instruction tuning.

All of these are measured against a fine-tuned MARBERT.

For generation, a decision agent chooses per query between two evidence sources. One is a curated library of reference texts. The other is a team of Gemini-2.0-Flash agents that search reputable health websites and summarise what they find for the answering LLM.

MARBERT still wins classification, and the best LLM setups come close. The RAG generator scores 0.652 BERTScore with very rare contradictions, but it covers only about 60% of gold key points. Grounded generation works, but recall is still the weak spot.

Key points

  • In MentalQA, questions carry labels from a set of seven types, while replies are tagged with a strategy drawn from a set of three: Information, Direct Guidance and Emotional Support. Fleiss' κ is 0.61 and 0.96 respectively. The 350 training posts were split 300/50, and 150 posts form the test set.

  • Subtask 1:

    • Fine-tuned MARBERT reaches 0.541 weighted-F1 and 0.494 Jaccard.
    • Qwen3+PEFT reaches 0.507/0.434.
    • Instruction-tuned few-shot Qwen gets 0.533 F1 but only 7.33% exact matches, because it tends to add extra labels.

  • Subtask 2: MARBERT reaches 0.805/0.727, and SILMA+PEFT is the best LLM at 0.753/0.670.

  • Subtask 3 (SILMA vs Qwen3), where both use Qwen3-Embedding-4B for local retrieval:

    MetricSILMAQwen3
    BERTScore0.6520.645
    Hallucination0.060.04
    Completeness0.5670.600

  • The curated knowledge base draws on:

    • DSM-5-TR for diagnosis,
    • OpenStax Anatomy & Physiology,
    • the CDC's epidemiology text,
    • MedlinePlus for provider choice and general guidance.

  • PEFT runs sometimes output no label at all, because every score fell under the 0.5 threshold (7 empty outputs per model in Subtask 1).

Interesting ideas

  • I1. Multi-label outputs are scored as exact, partial or wrong instead of a single F1, which shows whether a model is mostly hitting subsets or missing whole cases.
  • I2. A router agent first checks whether curated sources can answer the query and only falls back to broader external search when coverage is thin, keeping high-precision knowledge as the default.
  • I3. Generated answers are judged against extracted gold key points, separating missing content (omissions) from content that contradicts the gold answer (hallucination); this matters because BERTScore alone penalised harmless extra detail.
  • I4. The required answer style is predicted first, and the generator is then conditioned on that style, making style a decided input rather than an accident of decoding.
  • I5. Empty outputs were traced to a fixed decision threshold, a simple failure that is easy to diagnose once outputs are counted per run.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Score predicted role sets and frame sets the same way, so a parser change that turns exact hits into partial ones is caught even when F1 stays flat.
  • I2 (world): Look up compiled WordNet/Wikidata knowledge first and consult a weaker external source only on a coverage miss, tagging such facts so explanations can show their origin.
  • I3 (answer): Break fable and tale QA references into key facts and report coverage and contradiction separately, which fits Mova's explainable answers better than surface similarity.
  • I4 (back): Choose the speech act or register (inform, advise, comfort) explicitly from the meaning graph before realising text, so the stylistic choice is itself explainable.
  • I5 (frames): Monitor how often multi-label speech-act or role decisions come back empty under a fixed threshold, and fall back to the top-scoring label with a low-confidence flag.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓American University in CairoCairo · Egypt2 works · 50 citations🎓German International UniversityCairo · Egypt4 works · 13 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksInformation retrieval & search · 473 works

Cite

Cite as Caroline Sabty, Mohamad Rasmy, Mohamed Eyad Badran, Nourhan Sakr and Alia El Bolock. 2025. Fahmni at AraHealthQA Track 1: Multi-Agent Retrieval-Augmented Generation and Multi-Label Classification for Arabic Mental Health Q&A. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.29

@inproceedings{sabty2025fahmni,
  title = {{Fahmni at AraHealthQA Track 1: Multi-Agent Retrieval-Augmented Generation and Multi-Label Classification for Arabic Mental Health Q\&A}},
  author = {Caroline Sabty, Mohamad Rasmy, Mohamed Eyad Badran, Nourhan Sakr and Alia {El Bolock}},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.29},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.29}
}

References

13 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hassan Alhuzali and Ashwag Alasmari. 2025. Pre-trained language models for mental health: An empirical study on arabic qa classification. Healthcare, 13(9). find
  2. Hassan Alhuzali, Ashwag Alasmari, and Hamad Al-saleh. 2024. Mentalqa: An annotated arabic corpus for questions and answers of mental healthcare. IEEE Access, 12:101155–101165. ↗
  3. Hassan Alhuzali, Farah Shamout, Muhammad Abdul-Mageed, Chaimae Abouzahir, Mouath Abu-Daoud, Ashwag Alasmari, Walid Al-Eisawi, Renad Al-Monef, Ali Alqahtani, Lama Ayash, Nizar Habash, and Leen Kharouf. 2025. Arahealthqa 2025 shared task description paper. In Proceedings of ArabicNLP 2025. find
  4. American Psychiatric Association. 2022. Diagnostic and Statistical Manual of Mental Disorders: DSM-5-TR. American Psychiatric Association Publishing. find
  5. Orlando Ayala and Patrice Bechard. 2024. Reducing hallucination in structured outputs via retrieval-augmented generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 6: Industry Track), pages 228–238, Mexico City, Mexico. Association for Computational Linguistics. ↗
  6. J Gordon Betts, Kelly A Young, James A Wise, Eddie Johnson, Brandon Poe, Dean H Kruse, Oksana Korol, Jody E Johnson, Mark Womble, and Peter DeSaix. 2024. Anatomy and physiology 2e. find
  7. Third Edition. 2006. Principles of epidemiology. find
  8. Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, David Stap, Jay Gala, Wissam Siblini, Dominik Krzemiński, Genta Indra Winata, Saba Sturua, Saiteja Utpala, Mathieu Ciancone, Marion Schaeffer, Gabriel Sequeira, Diganta Misra, Shreeya Dhakal, Jonathan Rystrøm, Roman Solomatin, and 67 others. 2025. Mmteb: Massive multilingual text embedding benchmark. arXiv preprint arXiv:2502.13595. ↗
  9. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459– 9474.
  10. Karim Ouda. 2025. ABBL: An advanced benchmark and leaderboard for comprehensive evaluation of arabic language models. Accessed: 2025-07-19.  Ï @ á« PA¾ ¯ @ð ,  ®  JJË@ ú ¯ Z@ YK @ ð HñÖ Qª ƒ  @ AÒ» . èAJ mÌ '@ ú ¯ éJ  . «P ÐY«ð  ,  ® JË@ . A KAJ k @ I.ªJËAK. find
  11. Mohamad Rasmy, Caroline Sabty, Nourhan Sakr, and Alia El Bolock. 2024. Enhanced cognitive distortions detection and classification through data augmentation techniques. In Pacific Rim International Conference on Artificial Intelligence, pages 134–145, Singapore. Springer Nature Singapore. Answer strategy: Direct Guidance SILMA-AI. 2025. Silma kashif 2b instruct v1.0. https://huggingface.co/silma-ai/ SILMA-Kashif-2B-Instruct-v1.0. , éJ  ‚ ®  K  @QÓ @ I  .  J.£ ù ªk. @P : h. . éÓ P B@ PðAm. ' ú ¯ ¼ Y«A‚ ƒð SILMA9BInstruct. 2024. Silma 9b instruct v1.0. https://huggingface.co/silma-ai/ SILMA-9B-Instruct-v1.0. ↗
  12. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. ↗
  13. Kunlun Zhu, Yifan Luo, Dingling Xu, Yukun Yan, Zhenghao Liu, Shi Yu, Ruobing Wang, Shuo Wang, Yishan Li, Nan Zhang, Xu Han, Zhiyuan Liu, and Maosong Sun. 2025. Rageval: Scenario specific rag evaluation dataset generation framework. Preprint, arXiv:2408.01262. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sabty-2025-fahmni-arahealthqa-track-1-multi/