Elamur Research⌕
☰
home / papers / 2026 / HerHealthEval: Evaluating Multilingual and Register-Sensitive Understanding of Women's Health Communication

HerHealthEval: Evaluating Multilingual and Register-Sensitive Understanding of Women's Health Communication

Paper · 2026 · Hassan Saeed Hassan Albattra, Mazen Mohammed Bahgat, Rahatara Ferdousi et al.

Summary

Most evaluations of LLMs in healthcare judge the quality of the generated answer and take for granted that the user's concern was understood. HerHealthEval tests that understanding directly for women's health: menstruation, PCOS/hormonal issues and fertility. Each case is fixed and rewritten in six forms: canonical, clinical, layperson, indirect/hedged, emotionally concerned, and a deliberately under-specified version that drops key details to see whether the model asks for clarification. The cases are matched across English, French and Modern Standard Arabic, with native-speaker validation. Models are scored separately on concern category, risk level, clarification, parse compliance and cross-form consistency.

The authors compare Qwen3.5-9B-Instruct with QLoRA-adapted variants. Joint multilingual adaptation made nearly all French and Arabic urgent cases look routine. It also almost eliminated clarification-seeking, while consistency scores looked better than before. The cause was risk labels produced by an English-keyed heuristic applied to translated text. Deriving labels once from the English source repaired much of the triage failure. The broader lesson is that aggregate metrics can hide safety-relevant collapse, and that adaptation labels must not vary with language.

Key points

  • Each language has 540 matched scored items. Leakage between adaptation and evaluation data is controlled by exact-match, source-provenance and paraphrase-family exclusion.
  • Zero-shot Model 1 had clarification recall of 0.856 (EN), 0.811 (FR) and 0.889 (AR). Multilingual QLoRA Model 4 dropped this to 0.000, 0.000 and 0.011.
  • Model 4 raised under-triage in French and Arabic to 0.994 (from 0.632 and 0.638), so about 173 of 174 see-doctor cases were labelled routine. English stayed nearly flat (0.718 → 0.711).
  • Model 4's risk consistency rose misleadingly (FR 0.544 → 0.978, AR 0.456 → 0.967), and French–Arabic agreement reached 0.991, because both languages collapsed onto "routine".
  • The risk-corrected re-adaptation M3-ML-RC (Model 5) cut under-triage to 0.590 (EN), 0.572 (FR) and 0.558 (AR), with McNemar p ≈ 10⁻²¹–10⁻²³. Clarification recall stayed at 0.000 everywhere, and the authors call their inferential claims exploratory.

Interesting ideas

  • I1. Keep the case fixed and vary only its wording, so any change in prediction must come from the wording itself; add one meaning-removing variant scored separately as a "should ask" probe.
  • I2. High agreement can come from a model collapsing onto one class, so consistency should be reported next to per-class error rates such as under-triage.
  • I3. Compute a label once on the source and copy it to every aligned translation, instead of re-running a language-specific heuristic on translated text.
  • I4. Contamination control should go beyond exact matches and also exclude items that share a source or belong to the same paraphrase family as test data.
  • I5. Scoring interpretation, urgency and need-for-clarification as separate targets shows failures that a single answer-quality score would hide.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Fable and school-test QA sets could include register and indirect paraphrases of the same question that must get identical answers, plus a stripped version where the correct output is "not enough information".
  • I2 (learn): When witness parsers or teacher tiers agree, check that agreement against a trivial majority-label baseline as a negative control before trusting the silver data.
  • I3 (back): When translating through the meta-world, attach frames and annotations to the source meaning graph and carry them over, rather than re-deriving them from the target-language surface.
  • I4 (learn): LLM-generated or self-play training text should be filtered by provenance and paraphrase cluster against evaluation fables and tests, so scores are not inflated.
  • I5 (frames): Hedged or emotional phrasings could be checked to yield the same speech act and implicature frame as the direct phrasing, scored separately from what the answerer later produces.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Queen's UniversityKingston · Canada8 works · 1,610 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksMedicine · 80 worksHealth Informatics · 53 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 467 works

Cite

Cite as Hassan Saeed Hassan Albattra, Mazen Mohammed Bahgat, Rahatara Ferdousi, Hana Essam Sayed Ahmed Amrya and Mariam Mousa. 2026. HerHealthEval: Evaluating Multilingual and Register-Sensitive Understanding of Women's Health Communication. arXiv:2609.20684.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2609.20684 · arXiv 2609.20684

@misc{albattra2026herhealtheval,
  title = {{HerHealthEval: Evaluating Multilingual and Register-Sensitive Understanding of Women's Health Communication}},
  author = {Hassan Saeed Hassan Albattra, Mazen Mohammed Bahgat, Rahatara Ferdousi, Hana Essam Sayed Ahmed Amrya and Mariam Mousa},
  year = {2026},
  doi = {10.48550/arxiv.2609.20684},
  eprint = {2609.20684},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2609.20684}
}

References

16 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adhikary, P.K., Motiyani, I., Oke, G., Joshi, M., Pathak, K., Singh, S.M., Chakraborty, T.: Menstrual health education using a specialized large language model in India: Development and evaluation study of MenstLLaMA. Journal of Medical Internet Research 27, e71977 (2025). DOI 10.2196/71977 ↗
  2. Chervenak, J., Lieman, H., Blanco-Breindel, M., Jindal, S.: The promise and peril of using a large language model to obtain clinical information: ChatGPT performs strongly as a fertility counseling tool with limitations. Fertility and Sterility 120(3), 575–583 (2023). DOI 10.1016/j.fertnstert.2023.05.151 ↗
  3. de Vries, P.L.M., Baud, D., Baggio, S., Ceulemans, M., Favre, G., Gerbier, E., et al.: Enhancing perinatal health patient information through ChatGPT—an accuracy study. PEC Innovation 6, 100381 (2025) find
  4. Deva, R., Ramani, D., Divate, T., Jalota, S., Ismail, A.: “Kya Family Planning after Marriage Hoti Hai?”: Integrating cultural sensitivity in an LLM chatbot for reproductive health. In: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (2025). DOI 10.1145/3706598.3713362 ↗
  5. Graça, S., Dallaway, A., Alloh, F., Randeva, H.S., Kite, C., Kyrou, I.: Assessing ChatGPT vs. evidence-based online responses for polycystic ovary syndrome self-management and education: An international cross-sectional blinded survey of healthcare professionals. Frontiers in Digital Health 7, 1700018 (2026). DOI 10.3389/fdgth.2025.1700018 ↗
  6. Hershcovich, D., Frank, S., Lent, H., de Lhoneux, M., Abdou, M., Brandl, S., Bugliarello, E., Cabello Piqueras, L., Chalkidis, I., Cui, R., Fierro, C., Margatina, K., Rust, P., Søgaard, A.: Challenges and strategies in cross-cultural NLP. In: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pp. 6997–7013 (2022). DOI 10.18653/v1/2022.acl-long.482 ↗
  7. Jin, Y., Chandra, M., Verma, G., Hu, Y., De Choudhury, M., Kumar, S.: Better to ask in english: Cross-lingual evaluation of large language models for healthcare queries. In: Proceedings of the ACM Web Conference 2024 (2024). DOI 10.1145/3589334.3645643 ↗
  8. Li, Y.: ChatDoctor [dataset and code repository]. GitHub (2023). URL https://github.com/Kent0n-Li/ChatDoctor ↗
  9. Li, Y., Li, Z., Zhang, K., Dan, R., Jiang, S., Zhang, Y.: Chat-Doctor: A medical chat model fine-tuned on a large language model Meta-AI (LLaMA) using medical domain knowledge. Cureus 15(6), e40895 (2023). DOI 10.7759/cureus.40895 ↗
  10. Malikeh1375: medical-question-answering-datasets [data set]. Hugging Face (2023). URL https://huggingface.co/datasets/Malikeh1375/medical-question-answering-datasets ↗
  11. Maurya, S., Saboo, P., Kumar, G.: WHBench: Evaluating frontier LLMs with expert-in-the-loop validation on women’s health topics (2026). URL https://arxiv.org/abs/2604.00024 ↗
  12. Mughal, S.F., Aamir, S., Samad, A., Zehra, U., Syed, A.A.: Mai: A transformer based domain specific chatbot for menstrual health. ACM Journal on Responsible Computing 2(1), 1–32 (2025). DOI 10.1145/3711711 ↗
  13. Recker, F., Neubauer, R., Wittek, A., Scholten, N.: Large language models and women’s health: a digital companion for informed decision-making. Archives of Gynecology and Obstetrics 312, 663–670 (2025). DOI 10.1007/s00404-025-08065-9 ↗
  14. Schlicht, I.B., Zhao, Z., Sayin, B., Flek, L., Rosso, P.: Do LLMs provide consistent answers to health-related questions across languages? (2025). URL https://arxiv.org/abs/2501.14719 ↗
  15. Testoni, A., Fernández, R.: Asking the right question at the right time: Human and model uncertainty guidance to ask clarification questions. In: Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics, pp. 258–275 (2024). DOI 10.18653/v1/2024. eacl-long.16 ↗
  16. Zhang, M.J., Choi, E.: Clarify when necessary: Resolving ambiguity through interaction with LMs. In: Findings of the Association for Computational Linguistics: NAACL 2025, pp. 5541–5558 (2025). DOI 10.18653/v1/2025.findings-naacl.306 ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/albattra-2026-herhealtheval-evaluating-multilingual-register-sensitive/