Elamur Research⌕
☰
home / papers / 2025 / Quasar at AraHealthQA Track 1 : Leveraging Zero-Shot Large Language Models for Question and Answer Categorization in Arabic Mental Health

Quasar at AraHealthQA Track 1 : Leveraging Zero-Shot Large Language Models for Question and Answer Categorization in Arabic Mental Health

Paper · 2025 · Adiba Fairooz Chowdhury and Md Sagor Chowdhury

Summary

The paper describes a team's entry to AraHealthQA 2025 Track 1, a shared task on Arabic mental-health Q&A. The task has two multi-label subtasks. The first sorts patient questions into seven topical types, such as Diagnosis, Treatment and Provider Choices. The second labels answers by the strategy they use: Information, Direct Guidance or Emotional Support. The data come from the MentalQA dataset: 500 annotated posts, 350 used as labelled development data and 150 as a blind test set. The authors skip fine-tuning entirely. They run six models zero-shot with Arabic prompts that list the categories and ask the model to reason before answering. A regex stage then pulls labels out of the free-form replies. Qwen3-14B, run with its thinking mode enabled, was the strongest model and placed 4th in both subtasks. Answer-strategy labelling turned out much easier than question typing. The work suggests that off-the-shelf multilingual LLMs can be a reasonable starting point for small, culturally specific classification problems. It also shows that fine-grained intent distinctions remain hard for them.

Key points

  • Six models were compared: Qwen3-14B, Qwen2.5-7B, Qwen2-7B, Llama3.1-8B, DeepSeek R1-7B (a distilled Qwen) and AraBERTv2. The larger models were loaded with 4-bit NF4 quantization.
  • Blind test results for Qwen3-14B were as follows:
    • Question typing: weighted F1 0.52, Jaccard 0.41.
    • Answer strategy: weighted F1 0.76, Jaccard 0.66.
    • It ranked 4th in both subtasks.
  • On the development data, Qwen3-14B reached 0.507 question F1. Llama3.1-8B reached only 0.315, which falls below even a random-label baseline (0.326). The weighted-random baseline scored 0.386 for questions and 0.587 for answers.
  • Within the Qwen family, scores improved with each newer and larger version. For answers, the encoder-only AraBERTv2 had the lowest F1 (0.466), but its Jaccard (0.563) was not the worst.
  • The most frequent confusions were:
    • Treatment vs Diagnosis (111 cases).
    • Diagnosis vs Healthy Lifestyle (89 cases).
    • Information vs Direct Guidance, on the answer side.
  • Practical problems with model outputs included messy formatting, answers mixing Arabic and English label names, and occasional refusals on sensitive content.

Interesting ideas

  • I1. The authors compare every model against three trivial label generators: uniform random, majority class and frequency-weighted random. This reveals when a "working" model is actually no better than chance, as happened with Llama on questions.
  • I2. A tolerant, multi-pattern extractor maps whatever an LLM returns onto the fixed label set, accepting both label letters and label names in either language. This keeps differently behaving models comparable on the same evaluation.
  • I3. The answer labels (information vs direct guidance vs emotional support) amount to a coarse typology of what a reply is doing. The main confusion lies between giving information and giving instructions.
  • I4. For multi-label scoring, the paper reports per-sample Jaccard alongside class-weighted F1. One metric rewards getting each item's whole label set right, while the other rewards getting frequent classes right.
  • I5. Refusal to answer is recorded as a distinct failure mode rather than silently counted as a wrong label.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Every evaluation of a new parser or teacher tier could ship with matching dumb baselines as negative controls, so that a gain over chance is shown rather than assumed.
  • I2 (learn): LLM teacher tiers producing silver annotations need the same kind of normalizing layer, and the cases it fails to parse should be logged as their own error class.
  • I3 (frames): The prag speech-act layer could adopt this three-way split for responses and add explicit tests on the informing-versus-directing boundary, where the paper's models stumbled most.
  • I4 (answer): Questions with several acceptable answer facets could be scored both ways, so that partial credit and per-category bias are visible separately.
  • I5 (learn): When LLM teachers decline or hedge, those items should be flagged and routed to other witnesses instead of polluting the silver data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chittagong University of Engineering & TechnologyChittagong · Bangladesh25 works · 29 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksSocial Psychology · 306 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 works

Cite

Cite as Adiba Fairooz Chowdhury and Md Sagor Chowdhury. 2025. Quasar at AraHealthQA Track 1 : Leveraging Zero-Shot Large Language Models for Question and Answer Categorization in Arabic Mental Health. Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.arabicnlp-sharedtasks.23

@inproceedings{chowdhury2025quasar,
  title = {{Quasar at AraHealthQA Track 1 : Leveraging Zero-Shot Large Language Models for Question and Answer Categorization in Arabic Mental Health}},
  author = {Adiba Fairooz Chowdhury and Md Sagor Chowdhury},
  booktitle = {Proceedings of The Third Arabic Natural Language Processing Conference: Shared Tasks},
  year = {2025},
  doi = {10.18653/v1/2025.arabicnlp-sharedtasks.23},
  url = {https://doi.org/10.18653/v1/2025.arabicnlp-sharedtasks.23}
}

References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amani Abdulsalam, Abdullah Alhothali, and Saeed Al-Ghamdi. 2024. Detecting suicidality in arabic tweets using machine learning and deep learning techniques. Arabian Journal for Science and Engineering, 49:12729–12742. ↗
  2. Mouath Abu Daoud, Chaimae Abouzahir, Leen Kharouf, Walid Al-Eisawi, Nizar Habash, and Farah E Shamout. 2025. Medarabiq: Benchmarking large language models on arabic medical tasks. arXiv e-prints, pages arXiv–2505.
  3. Afnan Al-Laith and Mansour Alenezi. 2021. Monitoring people’s emotions and symptoms from arabic tweets during the covid-19 pandemic. Information, 12(2):86. ↗
  4. Nouf Al-Musallam and Majed Al-Abdullatif. 2022. Depression detection through identifying depressive arabic tweets from saudi arabia: Machine learning approach. In 2022 Fifth National Conference of Saudi Computer Colleges (NCCC), pages 11–18. IEEE. find
  5. Ashwag Alasmari. 2025. A scoping review of arabic natural language processing for mental health. In Healthcare, volume 13-9, page 963. MDPI. ↗
  6. Saeed H. Aldhafer and Mahdi Yakhlef. 2022. Depression detection in arabic tweets using deep learning. In 2022 6th International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE), pages 1–6. IEEE. find
  7. Hassan Alhuzali and Ashwag Alasmari. 2025. Pre-trained language models for mental health: An empirical study on arabic qa classification. Healthcare, 13(9). find
  8. Hassan Alhuzali, Ashwag Alasmari, and Hamad Al-saleh. 2024. Mentalqa: An annotated arabic corpus for questions and answers of mental healthcare. IEEE Access, 12:101155–101165. ↗
  9. Hassan Alhuzali, Farah Shamout, Muhammad Abdul-Mageed, Chaimae Abouzahir, Mouath Abu-Daoud, Ashwag Alasmari, Walid Al-Eisawi, Renad Al-Monef, Ali Alqahtani, Lama Ayash, Nizar Habash, and Leen Kharouf. 2025. Arahealthqa 2025 shared task description paper. In Proceedings of ArabicNLP 2025. find
  10. Thilina Atapattu, Madurika Herath, Chiran Elvitigala, Prashan de Zoysa, Kasun Gunawardana, Madurika Thilakaratne, Kasun de Zoysa, and Katrina Falkner. 2022. Emoment: An emotion annotated mental health corpus from two south asian countries. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6991–7001. ↗
  11. Nour Bensalah, Hiba Ayad, Ali Adib, and Abdelouahid Ibn El Farouk. 2024. Mindwave app: Leveraging ai for mental health support in english and arabic. In 2024 IEEE 12th International Symposium on Signal, Image, Video and Communications (ISIVC), pages 1–6. IEEE. find
  12. Mahmoud El-Ramly, Hadeer Abu-Elyazid, Yomna Mo’men, Ghadah Alshaer, Nermine Adib, Khaled A. Eldeen, and Manar El-Shazly. 2021. Cairodep: Detecting depression in arabic posts using bert transformers. In 2021 Tenth International Conference on Intelligent Computing and Information Systems (ICICIS), pages 207–212. IEEE. find
  13. Imane Guellil, Hichem Saâdane, Faycal Azouaou, Bachir Gueni, and Damien Nouvel. 2021. Arabic natural language processing: An overview. Journal of King Saud University-Computer and Information Sciences, 33:497–507.
  14. Kuan He, Ruixiang Mao, Qian Lin, Yuxuan Ruan, Xiaodong Lan, Minlie Feng, and Erik Cambria. 2023. A survey of large language models for healthcare: From data, technology, and applications to accountability and ethics. arXiv e-prints, pages arXiv–2310.05694.
  15. Md Kamrul Kabir, Md Islam, A N M Baki Kabir, Anwarul Haque, and Md Kamrul Rhaman. 2022. Detection of depression severity using bengali social media posts on mental health: Study using natural language processing techniques. JMIR Formative Research, 6:e36118. ↗
  16. Jiaming Liu, Dan Li, Hao Cao, Tao Ren, Zhen Liao, and Jian Wu. 2023. Chatcounselor: A large language model for mental health support. arXiv e-prints, pages arXiv–2309.15461. find
  17. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. ↗
  18. Khaled Shaalan, Sajid Siddiqui, Moustafa Alkhatib, and Ahmed Abdel Monem. 2019. Challenges in arabic natural language processing. In Computational Linguistics, Speech and Image Processing for Arabic Language, pages 59–83. World Scientific. ↗
  19. Aniket Sharma, I.W. Lin, Adam S. Miner, David C. Atkins, and Tim Althoff. 2023. Human–ai collaboration enables more empathic conversations in text-based peer-to-peer mental health support. Nature Machine Intelligence, 5:46–57. ↗
  20. Laura Brocki, George C. Dyer, Agnieszka Gładka, and N.C. Chung. 2023. Deep learning mental health dialogue system. In 2023 IEEE International Conference on Big Data and Smart Computing (BigComp), pages 395–398. IEEE. find
  21. Huan Sun, Zheng Lin, Chuxu Zheng, Shasha Liu, and Minlie Huang. 2021. Psyqa: A chinese dataset for generating long counseling text for mental health support. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1489–1503. find
  22. Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2021. 8-bit optimizers via block-wise quantization. Preprint, arXiv:2110.02861. ICLR 2022 Spotlight Version. ↗
  23. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32. ↗
  24. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, pages 4171–4186.
  25. Nourhan Zahran, Amr E Fouda, Rana J Hanafy, and Mostafa E Fouda. 2025. A comprehensive evaluation of large language models on mental illnesses in arabic context. arXiv preprint arXiv:2501.06859. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chowdhury-2025-quasar-arahealthqa-track-1-leveraging/