Elamur Research⌕
☰
home / papers / 2025 / Scaling Arabic Medical Chatbots Using Synthetic Data: Enhancing Generative AI with Synthetic Patient Records

Scaling Arabic Medical Chatbots Using Synthetic Data: Enhancing Generative AI with Synthetic Patient Records

Summary

Arabic medical chatbots have little training data. An earlier collection of 20,000 real patient–doctor exchanges, gathered from social media and medical forums, was too small for fine-tuned LLMs to generalize well. The authors grow this corpus five-fold to 100,000 records. They prompt ChatGPT-4o and Gemini 2.5 Pro to write 80,000 new question–answer pairs that follow the structure and language of the real data. The synthetic pairs are filtered for semantic relevance, checked by hand, and merged with the original records. Five LLMs, including Mistral-7B, LLaMA-2-7B and AraGPT2, are then fine-tuned again. Evaluation combines BERTScore precision, recall and F1 with expert judgments of coherence and informativeness. The augmented models are more fluent and accurate, overfit less, and repeat themselves less often. An ablation shows that the choice of generator matters: data from ChatGPT-4o beat data from Gemini for every model. The work suggests that LLM-generated records are a workable, privacy-friendly way to scale medical NLP in low-resource languages.

Key points

  • The seed corpus has 20,000 real Arabic patient–doctor interactions from social media. These are informal, dialect-rich and noisy.
  • ChatGPT-4o and Gemini 2.5 Pro produced 80,000 synthetic QA pairs, for a total of 100,000 training records.
  • The quality pipeline has two steps: semantic filtering of generated samples, then manual validation, before fine-tuning.
  • Five LLMs were fine-tuned, including Mistral-7B, LLaMA-2-7B and AraGPT2. They were scored with BERTScore (P/R/F1) and expert qualitative review.
  • In the ablation, data volume and settings were held equal. ChatGPT-4o data gave higher F1 and fewer hallucinations than Gemini data across all models.

Interesting ideas

  • I1. The generator ablation holds the amount of synthetic data and the training setup fixed and changes only the source model, so it shows how much the source itself matters. — For Mova (learn): Train on equal-sized silver sets from each LLM teacher tier with identical settings. This would measure which teacher actually improves the parser or reader, instead of just assuming it.
  • I2. Synthetic examples are tied to the shape and register of a real seed corpus, so augmentation stays close to the true input distribution instead of drifting into generic text. — For Mova (answer): Generate extra maths word problems and fable QA modelled on real school-test items. This would widen coverage while keeping the style the readers will actually face.
  • I3. Generated data is not trusted blindly: an automatic semantic filter is followed by a human check before anything enters training. — For Mova (world): Use the world-of-the-text model as the automatic filter. Synthetic stories whose events, holdings or states contradict each other would be rejected before any human looks at them.
  • I4. An embedding-based overlap metric is combined with expert judgment, and hallucinations are tracked as a separate outcome. This catches fluent but fabricated outputs that similarity scores reward. — For Mova (back): Score graph → text generation both for meaning overlap and for content not supported by the graph. Invented facts would then show up as errors.
  • I5. The synthetic data is meant to preserve dialectal and informal phrasing, giving robustness to noisy user input rather than only clean text. — For Mova (tokens): Augment tagger and lemmatizer training with controlled informal spellings and colloquial variants. This would harden the first pipeline stage against messy real-world English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2509.10108 · arXiv 2509.10108

@misc{allam2025scaling,
  title = {{Scaling Arabic Medical Chatbots Using Synthetic Data: Enhancing Generative AI with Synthetic Patient Records}},
  author = {Allam, Abdulrahman and Ahmed, Seif and Hamdi, Ali and Shaban, Khaled},
  year = {2025},
  doi = {10.48550/arxiv.2509.10108},
  eprint = {2509.10108},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2509.10108}
}

References

23 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. E. Detmer, E. B. Steen, and R. S. Dick, The computer-based patient record: an essential technology for health care. National Academy Press, 1997. find
  2. T. C. Brown, “The concept of value in resource allocation,” Land economics, vol. 60, no. 3, pp. 231–246, 1984. find
  3. M. Rashid and M. Sharma, “Ai-assisted diagnosis and treatment planning—a discussion of how ai can assist healthcare professionals in making more accurate diagnoses and treatment plans for diseases,” in AI in Disease Detection: Advancements and Applications, 2025, pp. 313– 336. find
  4. N. Kühl, M. Schemmer, M. Goutier, and G. Satzger, “Artificial intelligence and machine learning,” Electronic Markets, vol. 32, no. 4, pp. 2235–2244, 2022. find
  5. M. L. PRIYANKA, B. T. R. NAYAKA, B. S. RAKSHITHA, B. VAMSHI, and B. M. K. YADAV, “Hospital management system with chatbot,” International Journal of Mechanical Engineering Research and Technology, vol. 16, no. 2, pp. 144–154, 2024. find
  6. Z. A. Nazi and W. Peng, “Large language models in healthcare and medical domain: A review,” Informatics, vol. 11, no. 3, p. 57, 2024. find
  7. C. Peng, X. Yang, A. Chen, K. E. Smith, N. PourNejatian, A. B. Costa, C. Martin, M. G. Flores, Y. Zhang, T. Magoc et al., “A study of generative large language model for medical research and healthcare,” NPJ digital medicine, vol. 6, no. 1, p. 210, 2023. find
  8. M. Abdelhay, A. Mohammed, and H. A. Hefny, “Deep learning for arabic healthcare: Medicalbot,” Social Network Analysis and Mining, vol. 13, no. 1, p. 71, 2023. find
  9. D. Al-Ghadhban and N. Al-Twairesh, “Nabiha: an arabic dialect chatbot,” International Journal of Advanced Computer Science and Applications, vol. 11, no. 3, 2020. find
  10. A. Figueira and B. Vaz, “Survey on synthetic data generation, evaluation methods and gans,” Mathematics, vol. 10, no. 15, p. 2733, 2022. find
  11. Y. Lu, M. Shen, H. Wang, X. Wang, C. van Rechem, T. Fu, and W. Wei, “Machine learning for synthetic data generation: a review,” arXiv preprint arXiv:2302.04062, 2023. ↗
  12. H. Murtaza, M. Ahmed, N. F. Khan, G. Murtaza, S. Zafar, and A. Bano, “Synthetic data generation: State of the art in health care domain,” Computer Science Review, vol. 48, p. 100546, 2023. find
  13. J. Dahmen and D. Cook, “Synsys: A synthetic data generation system for healthcare applications,” Sensors, vol. 19, no. 5, p. 1181, 2019. find
  14. F. K. Dankar and M. Ibrahim, “Fake it till you make it: Guidelines for effective synthetic data generation,” Applied Sciences, vol. 11, no. 5, p. 2158, 2021. find
  15. P. Zhang and M. N. Kamel Boulos, “Generative ai in medicine and healthcare: promises, opportunities and challenges,” Future Internet, vol. 15, no. 9, p. 286, 2023. find
  16. A. Rekik, M. Chebbi, and N. Boughanmi, “A medical chatbot for tunisian dialect using a rule-based and machine learning approach,” Journal of Medical Informatics, vol. 12, no. 3, pp. 45–56, 2023. find
  17. J. Anwar, P. Nadi, and N. Seddik, “Towards building a chatbot-based first aid service in arabic language,” Journal of Advanced Research in Applied Sciences and Engineering Technology, vol. 45, no. 2, pp. 1–10, 2025. find
  18. R. G. Gamage, K. U. Senadeera, N. S. Bandara, K. Y. Abeywardena, D. D. Diyamullage, and N. Amarasena, “Pharmago: An online pharmaceutical ordering platform,” in 2021 3rd International Conference on Advancements in Computing (ICAC), 2021, pp. 364–370. find
  19. O. H. Abdellaif, A. Nader, and A. Hamdi, “Lmrpa: Large language model-driven efficient robotic process automation for ocr,” arXiv preprint arXiv:2412.18063, 2024. ↗
  20. A. Hamdi, A. A. Mazrou, and M. Shaltout, “Llm-sem: A sentiment-based student engagement metric using llms for e-learning platforms,” arXiv preprint arXiv:2412.13765, 2024. ↗
  21. M. Aldea, P. Rolland, S. Simon, A. Poplu, M. Wartelle, B. Vignal, J.-C. Louis, F. Lion, A. Borie, D. Planchard et al., “Using ai to automatically process data from unstructured health records of patients with lung cancer,” Cancer Research, vol. 84, no. 6 Supplement, pp. 3569–3569, 2024. find
  22. A. Hamdi, H. Kassab, M. Bahaa, and M. Mohamed, “Riro: Reshaping inputs, refining outputs unlocking the potential of large language models in data-scarce contexts,” arXiv preprint arXiv:2412.15254, 2024. ↗
  23. O. H. Abdellaif, A. N. Hassan, and A. Hamdi, “Erpa: Efficient rpa model integrating ocr and llms for intelligent document processing,” in 2024 International Mobile, Intelligent, and Ubiquitous Computing Conference (MIUCC). IEEE, 2024, pp. 295–300. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/allam-2025-scaling-arabic-medical-chatbots-using/