Elamur Research⌕
☰
home / papers / 2025 / AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset

AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset

Paper · 2025 · Charles Nimo, Tobi Olatunji, Abraham Toluwase Owodunni et al.

Summary

LLMs score well on medical exam benchmarks, but those benchmarks come mostly from Western sources. It is unclear whether the models handle African clinical contexts, regional disease patterns and local varieties of English. To test this, the authors built AfriMed-QA, an English medical QA benchmark covering the whole African continent. It holds 15,275 items of three kinds:

  • expert-written multiple-choice questions (MCQs) with rationales;
  • open-ended short-answer questions (SAQs);
  • crowdsourced consumer health queries (CQs), prompted by a list of conditions common in African communities.

Contributors used a crowdsourcing platform built by Intron Health and were screened by clinicians. The authors ran 30 LLMs on the benchmark and assessed them in two ways: automatic metrics, and blinded human ratings for correctness, harm, omissions, hallucinations and bias.

Main findings:

  • Accuracy varies widely by specialty and by country.
  • MCQ scores fall below those on MedQA (USMLE).
  • Biomedical models do worse than general-purpose ones.
  • Small edge-friendly models often fail to reach a passing score.
  • Lay raters consistently preferred LLM answers over clinicians' answers.

The resource matters because it gives a regionally grounded way to test medical AI before deployment in low-resource health systems.

Key points

  • Dataset composition: 4,269 expert items (3,910 MCQ, 359 SAQ) and 11,006 crowdsourced items (129 MCQ, 877 SAQ, 10,000 CQ). Contributors numbered 621, from 60+ medical schools in 16 countries, covering 32 specialties. Over 60% of expert MCQs come from West Africa.
  • Models evaluated: 30 LLMs (13 proprietary, 17 open), ranging from 3B to over 540B parameters. They include GPT-4, Claude 3, Gemini, MedLM, Phi-3, Meditron, OpenBioLLM and MedLlama.
  • Human evaluation: 379 raters (58 clinicians, 321 non-clinicians) produced 37,435 ratings. Lay raters scored only relevance, helpfulness and local context, not correctness.
  • Open-ended metrics: automatic scores separated models poorly. BERTScore clustered between 0.86 and 0.89, while QuestEval (0.19–0.51) and ROUGE-Lsum spread scores more widely. Analysis of open-ended answers is therefore left to future clinician review.
  • Explanations and errors: asking models to explain lowered MCQ accuracy, because the chosen option was hard to extract from the explanation. Small models such as Llama-3-8b and JSL-Medllama-8b produced the most hallucinations and harmful answers.
  • Availability: the data is on Hugging Face as intronhealth/afrimedqa_v2 under the CC-BY-NC-SA 4.0 license.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations🎓Brown UniversityProvidence · United States76 works · 25,952 citations🎓Georgia Institute of TechnologyAtlanta · United States133 works · 24,497 citations🎓The Ohio State UniversityColumbus · United States82 works · 18,502 citations🎓McGill UniversityMontreal · Canada87 works · 11,027 citations🎓University of MinnesotaMinneapolis · United States28 works · 2,184 citations

Also on the byline: University of Cape Coast · Kenyatta University.

 Fields, topics and concepts

📚 Shelves 8The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksBiochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksQuestion answering · 441 works

Cite

Cite as Charles Nimo, Tobi Olatunji, Abraham Toluwase Owodunni, Tassallah Abdullahi, Emmanuel Ayodele, Mardhiyah Sanni, Ezinwanne C. Aka, Folafunmi Omofoye, Foutse Yuehgoh, Timothy Faniran, Bonaventure F. P. Dossou, Moshood O. Yekini, Jonas Kemp, Katherine A Heller, Jude Chidubem Omeke, Chidi Asuzu Md, Naome A Etori, Aïmérou Ndiaye, Ifeoma Okoh, Evans Doe Ocansey, Wendy Kinara, Michael L. Best, Irfan Essa, Stephen Edward Moore, Chris Fourie and Mercy Nyamewaa Asiedu. 2025. AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.96

@inproceedings{nimo2025afrimed,
  title = {{AfriMed-QA: A Pan-African, Multi-Specialty, Medical Question-Answering Benchmark Dataset}},
  author = {Charles Nimo, Tobi Olatunji, Abraham Toluwase Owodunni, Tassallah Abdullahi, Emmanuel Ayodele, Mardhiyah Sanni, Ezinwanne C. Aka, Folafunmi Omofoye, Foutse Yuehgoh, Timothy Faniran, Bonaventure F. P. Dossou, Moshood O. Yekini, Jonas Kemp, Katherine A Heller, Jude Chidubem Omeke, Chidi Asuzu Md, Naome A Etori, Aïmérou Ndiaye, Ifeoma Okoh, Evans Doe Ocansey, Wendy Kinara, Michael L. Best, Irfan Essa, Stephen Edward Moore, Chris Fourie and Mercy Nyamewaa Asiedu},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.96},
  url = {https://doi.org/10.18653/v1/2025.acl-long.96}
}

References

45 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Asma Ben Abacha, Yassine Mrabet, Mark E. Sharp, Travis R. Goodwin, Sonya E. Shooshan, and Dina Demner-Fushman. 2019. Bridging the gap between consumers’ medication questions and trusted answers. Studies in health technology and informatics, 264:25–29. find
  2. Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.
  3. Shah. 2024. Medalign: A clinician-generated dataset for instruction following with electronic medical records. Proceedings of the AAAI Conference on Artificial Intelligence, 38(20):22021–22030. find
  4. Agasthya Gangavarapu. 2023. Llms: A promising new tool for improving healthcare in low-resource nations. In 2023 IEEE Global Humanitarian Technology Conference (GHTC), pages 252–255. IEEE. find
  5. Michael Hanna and Ondřej Bojar. 2021. A fine-grained analysis of bertscore. In Proceedings of the Sixth Conference on Machine Translation, pages 507–517.
  6. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. ArXiv, abs/2009.03300.
  7. Tassallah Abdullahi, Ritambhara Singh, Carsten Eickhoff, et al. 2024. Learning to make rare and complex diagnoses with generative ai assistance: Qualitative study of popular large language models. JMIR Medical Education, 10(1):e51391. find
  8. Niclas Hertzberg and Anna Lokrantz. 2024. MedQA-SWE - a clinical question & answer dataset for Swedish. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 11178–11186, Torino, Italia. ELRA and ICCL. find
  9. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  10. Israt Jahan, Md Tahmid Rahman Laskar, Chun Peng, and Jimmy Xiangji Huang. 2024. A comprehensive evaluation of large language models on benchmark biomedical text processing tasks. Computers in Biology and Medicine, 171:108189. find
  11. Anthropic. 2023. Introducing claude. Accessed: 2024-05-31. find
  12. Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020. Evaluation of text generation: A survey. arXiv preprint arXiv:2006.14799. find
  13. Hanjie Chen, Zhouxiang Fang, Yash Singla, and Mark Dredze. 2024. Benchmarking large language models on answering and explaining challenging medical questions. arXiv preprint arXiv:2402.18060.
  14. Zeming Chen, Alejandro Hernández Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, et al. 2023. Meditron-70b: Scaling medical pretraining for large language models. arXiv preprint arXiv:2311.16079.
  15. Alexander V Eriksen, Sören Möller, and Jesper Ryg. 2023. Use of gpt-4 to diagnose complex clinical cases. find
  16. Scott L. Fleming, Alejandro Lozano, William J. Haberkorn, Jenelle A. Jindal, Eduardo Reis, Rahul Thapa, Louis Blankemeier, Julian Z. Genkins, Ethan Steinberg, Ashwin Nayak, Birju Patel, Chia-Chun Chiang, Alison Callahan, Zepeng Huo, Sergios Gatidis, Scott Adams, Oluseyi Fayanju, Shreya J. Shah, Thom… find
  17. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2020. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. ArXiv, abs/2009.13081.
  18. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  19. Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
  20. Anastasia Krithara, Anastasios Nentidis, Konstantinos Bougiatiotis, and Georgios Paliouras. 2023. Bioasq-qa: A manually curated corpus for biomedical question answering. Scientific Data, 10:170. find
  21. Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al. 2023. Performance of chatgpt on usmle: potential for ai-assisted medical education using large language models. PLoS digital h…
  22. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  23. Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, LEI ZHU, and Michael Lingzhi Li. 2023. Benchmarking large language models on cmexam - a comprehensive chinese medical exam dataset. In Advances in Neural Information Processing Systems,… find
  24. Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, Lei Zhu, et al. 2024a. Benchmarking large language models on cmexam-a comprehensive chinese medical exam dataset. Advances in Neural Information Processing Systems, 36. find
  25. Michael Xieyang Liu, Frederick Liu, Alexander J Fiannaca, Terry Koo, Lucas Dixon, Michael Terry, and Carrie J Cai. 2024b. " we need structured output": Towards user-centered constraints on large language model output. In Extended Abstracts of the CHI Conference on Human Factors in Computing Systems,… find
  26. Qianying Liu, Sicong Jiang, Yizhong Wang, and Sujian Li. 2020. LiveQA: A question answering dataset over sports live. In Proceedings of the 19th Chinese National Conference on Computational Linguistics, pages 1057–1067, Haikou, China. Chinese Information Processing Society of China. find
  27. Yong Liu, Shenggen Ju, and Junfeng Wang. 2024c. Exploring the potential of chatgpt in medical dialogue summarization: a study on consistency with human preferences. BMC Medical Informatics and Decision Making, 24(1):75. find
  28. C’ecile Log’e, Emily L. Ross, David Yaw Amoah Dadey, Saahil Jain, Adriel Saporta, Andrew Y. Ng, and Pranav Rajpurkar. 2021. Q-pain: A question answering dataset to measure social bias in pain management. ArXiv, abs/2108.01764. — [arXiv:2108.01764](https://arxiv.org/abs/2108.01764) find
  29. Tobi Olatunji, Tejumade Afonja, Aditya Yadavalli, Chris Chinenye Emezue, Sahib Singh, Bonaventure FP Dossou, Joanne Osuchukwu, Salomey Osei, Atnafu Lambebo Tonja, Naome Etori, et al. 2023. Afrispeech-200: Pan-african accented speech dataset for clinical and general domain asr. Transactions of the As…
  30. Ankit Pal, Pasquale Minervini, Andreas Geert Motzfeldt, Aryo Pradipta Gema, and Beatrice Alex. 2024. open-lifescienceai/open_medical_llm_leaderboard. https://huggingface.co/spaces/ openlifescienceai/open_medical_llm_ leaderboard. ↗
  31. Ankit Pal and Malaikannan Sankarasubbu. 2024. Openbiollms: Advancing open-source large language models for healthcare and life sciences. https://huggingface.co/aaditya/ OpenBioLLM-Llama3-70B. Hugging Face repository. ↗
  32. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022a. Medmcqa : A large-scale multi-subject multi-choice dataset for medical domain question answering. In ACM Conference on Health, Inference, and Learning.
  33. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022b. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Proceedings of the Conference on Health, Inference, and Learning, volume 174 of Proceedings of Machine Learning Research, pages 2…
  34. Stephen R Pfohl, Heather Cole-Lewis, Rory Sayres, Darlene Neal, Mercy Asiedu, Awa Dieng, Nenad Tomasev, Qazi Mamunur Rashid, Shekoofeh Azizi, Negar Rostamzadeh, et al. 2024. A toolbox for surfacing health equity harms and biases in large language models. arXiv preprint arXiv:2403.12025. — [arXiv:2403.12025](https://arxiv.org/abs/2403.12025) find
  35. Sandeep Reddy. 2023. Evaluating large language models for use in healthcare: A framework for translational value assessment. Informatics in Medicine Unlocked, 41:101304. find
  36. Thomas Scialom, Paul-Alexis Dray, Patrick Gallinari, Sylvain Lamprier, Benjamin Piwowarski, Jacopo Staiano, and Alex Wang. 2021. Questeval: Summarization asks for fact-based evaluation. Preprint, arXiv:2103.12693.
  37. K. Singhal, Shekoofeh Azizi, Tao Tu, Said Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Kumar Tanwani, Heather J. Cole-Lewis, Stephen J. Pfohl, P A Payne, Martin G. Seneviratne, Paul Gamble, Chris Kelly, Nathaneal Scharli, Aakanksha Chowdhery, P. A. Mansfield, Blaise Agüera y Arcas, Dale… — Large Language Models Encode Clinical Knowledge
  38. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. 2022b. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138.
  39. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. 2023. Towards expert-level medical question answering with large language models. arXiv preprint arXiv:2305.09617.
  40. Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  41. David Thulke, Yingbo Gao, Petrus Pelser, Rein Brune, Rricha Jalota, Floris Fok, Michael Ramos, Ian van Wyk, Abdallah Nasir, Hayden Goldstein, et al. 2024. Climategpt: Towards ai synthesizing interdisciplinary research on climate change. arXiv preprint arXiv:2401.09646. find
  42. Tao Tu, Anil Palepu, Mike Schaekermann, Khaled Saab, Jan Freyberg, Ryutaro Tanno, Amy Wang, Brenna Li, Mohamed Amin, Nenad Tomasev, et al. 2024. Towards conversational diagnostic ai. arXiv preprint arXiv:2401.05654. — [arXiv:2401.05654](https://arxiv.org/abs/2401.05654) find
  43. Logesh Kumar Umapathi, Ankit Pal, and Malaikannan Sankarasubbu. 2023. Med-halt: Medical domain hallucination test for large language models. arXiv preprint arXiv:2307.15343.
  44. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  45. Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023a. Pmc-llama: Further fine-tuning llama on medical papers. arXiv preprint arXiv:2304.14454.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nimo-2025-afrimed-qa-pan-african-multi/