📄 MindLLM at AraHealthQA 2025 Track 1: Leveraging Large Language Models for Mental Health Question Answering

Summary

This paper is a shared-task entry for AraHealthQA 2025, Sub-task 3. The task is to generate answers in Arabic to real user questions about mental health. The data is the MentalQA dataset: 500 question–answer pairs whose answers were written by experts, split into 350 train/dev and 150 hidden test items.

The author did no fine-tuning and used no external data. Instead, four off-the-shelf LLMs were compared: GPT-4o, Gemini, Allam and Qwen. Several prompting schemes were also tested: zero-shot, 3- and 5-shot, chain-of-thought, self-consistency and ensemble refinement. Prompt variants were first compared with LLaMA-3-70B-Instruct as a reference model, using BERTScore F1 against the expert answers.

The plainest option worked best. A 3-shot prompt that tells the model to reply in Arabic beat both zero-shot and the more elaborate strategies. With GPT-4o, this setup scored 0.670 BERTScore F1 on the hidden test set and took 2nd place.

The work shows that careful, controlled prompt design can be competitive in a sensitive low-resource domain. The author also reports that GPT-4o avoided making diagnoses or prescribing treatments.

Key points

• Data: MentalQA, 500 Arabic samples (350 train/dev, 150 hidden test). Each sample is labelled with:

  • a question type: diagnosis, treatment, epidemiology or healthy lifestyle;
  • an answer strategy: informational, direct guidance or emotional support.

• Strategy comparison with LLaMA-3-70B (BERTScore F1):

  Strategy	F1
  zero-shot	0.61
  3-shot	0.66
  3-shot + self-consistency	0.66
  5-shot	0.64
  chain-of-thought	0.615
  3-shot + ensemble refinement	0.61

• Model comparison (3-shot, train set, F1): GPT-4o 0.6551, Allam 0.6316, Gemini 0.6210, Qwen 0.6131.

• Decoding settings, shared by all models: temperature 0.1, top-p 0.9, max 1024 tokens, fixed seeds.

• Evaluation: multilingual bert_score v0.3.11.

• Qualitative contrast between models: Gemini was empathetic but less precise, Allam leaned on technical vocabulary, and Qwen gave generic replies.

Interesting ideas

• I1. The author drew 10 exemplar pairs from the training split and reused the same set across every prompting strategy, so differences between strategies reflect the prompt rather than the choice of examples.
• I2. The exemplars were chosen so that every question type and every answer strategy appears at least once, a stratified selection rather than a random one.
• I3. Prompt strategies were ranked on a separate open model before the final generator was chosen, which separates "which prompt works" from "which model works".
• I4. Adding more shots or chain-of-thought lowered the reference-similarity score, and the 5-shot outputs grew redundant, so extra complexity was not free under this metric.
• I5. The dataset labels each answer with a communicative strategy (informational, guidance, emotional support), treated as a dimension separate from the question topic.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

• I1 (learn): When comparing LLM teacher-tier prompts, hold a single fixed exemplar set constant across variants so that silver-data quality changes can be credited to the prompt itself.
• I2 (answer): Few-shot or reference sets for school tests and fable QA could be stratified by question kind (who, why, how many, moral) so that no answer pattern is left unrepresented.
• I3 (learn): Prompt templates for teacher tiers could first be tuned on a cheaper local model, with only the winning template then run on the expensive teacher.
• I4 (answer): This is a caution that surface-similarity metrics can reward or punish style rather than correctness, so Mova's QA evaluation should also check answers against the world model and negative controls.
• I5 (frames): The speech-act layer in prag could carry a similar "answer strategy" tag, giving generation an explicit and explainable choice of how to respond, not only what to say.