Elamur Research⌕
☰
home / papers / 2023 / Can ChatGPT be Your Personal Medical Assistant?

Can ChatGPT be Your Personal Medical Assistant?

Paper · 2023 · Md. Rafiul Biswas, Ashhadul Islam, Zubair Shah et al.

Summary

The paper asks whether a fine-tuned commercial LLM can act as a medical question-answering assistant for Arabic speakers. Arabic healthcare chatbots are scarce, and Arabic dialects vary widely, which makes them hard to build. The authors used MAQA, a public collection of about 430K Arabic medical Q&A pairs scraped from websites and covering 20 specialties. They took a random subset from the largest category, gynecology, and converted it into chat-format JSONL with system, user and assistant roles. On this data they fine-tuned gpt-3.5-turbo through the OpenAI API.

The fine-tuned model was assessed in two ways:

  • Automatically: perplexity, coherence, cosine similarity to the reference answer, and token counts.
  • By humans: native Arabic speakers with medical background rated answers for relevance, accuracy, precision, logic and originality.

Answers from the tuned model were disease-specific, while the base model gave generic replies. Perplexity was moderate, and lexical similarity to the references was low. The authors still view the approach as promising. They also flag cost, token limits, and the poor explanatory quality of the reference answers as real obstacles.

Key points

  • Dataset: MAQA has 430K+ questions in 20 medical categories, stored in SQLite with six fields (q_body, a_body, their counts, category, category_id). q_body was used as the user turn and a_body as the assistant turn.
  • Training subset: from the gynecology category (category 15, 103,683 pairs), 4,000 pairs were sampled for training and 1,000 for validation.
  • Training: gpt-3.5-turbo was fine-tuned for three epochs, ending with a training loss of 0.3355.
  • Automatic scores: average perplexity was 13.96 and average cosine similarity between generated and original answers was 0.10.
  • Stated limitations:
    • The reference answers often lack explanation.
    • The training sample is small.
    • API costs grow with token volume, and there is a per-cycle token cap.
  • Future work: the authors propose hospital-sourced dialogue data and retrieval + generation ensembles.

Interesting ideas

  • I1. Run the same test conversations through both the base model and the tuned model, and treat the base output as a reference point; here it revealed that the untuned model answered generically rather than specifically.
  • I2. The tuned answers matched the references poorly on cosine similarity (0.10), yet the authors' overall judgement of the tuned model was favourable, which suggests surface word overlap is a weak yardstick for generated answers.
  • I3. Human judges scored each answer along several separate axes (relevance, accuracy, precision, logic, originality) rather than giving one overall grade.
  • I4. The authors note that training targets with thin explanations cap what the tuned model can learn, so teacher quality limits student quality.
  • I5. Restricting fine-tuning to a single large, homogeneous category keeps the experiment controlled and cheap before scaling to all classes.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Any new repair rule or teacher-tier change could be evaluated against the untouched pipeline on identical inputs, serving as a built-in negative control.
  • I2 (answer): Score answers by comparing frames or world-state facts against the gold answer's meaning graph instead of word vectors, so paraphrases are not penalised.
  • I3 (back): Generated text from the graph → text path could be rated on distinct axes such as factual fidelity to the graph versus fluency, making failures easier to localise.
  • I4 (learn): When selecting LLM teacher tiers or silver data, filter or down-weight examples whose targets lack justification, rather than only maximising volume.
  • I5 (learn): Pilot new training regimes on one well-populated domain, such as fables only, to measure the effect cleanly before mixing genres.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hamad bin Khalifa UniversityDoha · Qatar71 works · 12,942 citations

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksMedicine · 80 worksHealth Informatics · 53 works

🧭 Topics 1Research subjects and languages the work is about.Semitic languages · 1,484 works

Cite

Cite as Md. Rafiul Biswas, Ashhadul Islam, Zubair Shah, Wajdi Zaghouani and Samir Brahim Belhaouari. 2023. Can ChatGPT be Your Personal Medical Assistant?. arXiv:2312.12006 · The International Symposium on Foundation and Large Language Models (FLLM2023) https://fllm-conference.org/2023/.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2312.12006 · arXiv 2312.12006

@misc{biswas2023can,
  title = {{Can ChatGPT be Your Personal Medical Assistant?}},
  author = {Md. Rafiul Biswas, Ashhadul Islam, Zubair Shah, Wajdi Zaghouani and Samir Brahim Belhaouari},
  year = {2023},
  doi = {10.48550/arxiv.2312.12006},
  eprint = {2312.12006},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2312.12006}
}

References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. B. J. Cahn, “CHATBOT : Architecture , Design , & Development,” 2017. find
  2. A. Ahmed, N. Ali, M. Alzubaidi, W. Zaghouani, A. Abd-alrazaq, and M. Househ, “Arabic chatbot technologies: A scoping review,” Comput. Methods Programs Biomed. Updat., vol. 2, p. 100057, 2022, doi: 10.1016/j.cmpbup.2022.100057. ↗
  3. L. T. Car et al., “Conversational agents in health care: Scoping review and conceptual analysis,” J. Med. Internet Res., vol. 22, no. 8, 2020, doi: 10.2196/17158. ↗
  4. Y. K. Dwivedi et al., “‘So what if ChatGPT wrote it?’ Multidisciplinary perspectives on opportunities, challenges and implications of generative conversational AI for research, practice and policy,” Int. J. Inf. Manage., vol. 71, no. March, 2023, doi: 10.1016/j.ijinfomgt.2023.102642. ↗
  5. Q. Lu, B. Qiu, L. Ding, L. Xie, and D. Tao, “Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models: A Case Study on ChatGPT,” 2023, [Online]. Available: http://arxiv.org/abs/2303.13809. ↗
  6. Y. Bang et al., “A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity,” 2023, [Online]. Available: http://arxiv.org/abs/2302.04023. ↗
  7. T. Wu et al., “A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development,” IEEE/CAA J. Autom. Sin., vol. 10, no. 5, pp. 1122–1136, 2023, doi: 10.1109/JAS.2023.123618. ↗
  8. “Fine-tuning.” https://platform.openai.com/docs/guides/fine-tuning (accessed Sep. 09, 2023). ↗
  9. A. Lecler, L. Duron, and P. Soyer, “Revolutionizing radiology with GPT-based models: Current applications, future possibilities and limitations of ChatGPT,” Diagn. Interv. Imaging, vol. 104, no. 6, pp. 269–274, 2023, doi: 10.1016/j.diii.2023.02.003. ↗
  10. S. S. Biswas, “Role of Chat GPT in Public Health,” Ann. Biomed. Eng., vol. 51, no. 5, pp. 868–869, 2023, doi: 10.1007/s10439-023-03172-7. ↗
  11. A. M. Mohammed Abdelhay, “MAQA: Medical Arabic Q&A Dataset,” 2022, doi: 10.7910/DVN/Y2JBEZ. ↗
  12. T. F. Heston and C. Khun, “Prompt Engineering in Medical Education,” Int. Med. Educ., vol. 2, no. 3, pp. 198–205, 2023, doi: 10.3390/ime2030019. ↗
  13. T. Z. Zhao, E. Wallace, S. Feng, D. Klein, and S. Singh, “Calibrate Before Use: Improving Few-Shot Performance of Language Models,” Proc. Mach. Learn. Res., vol. 139, pp. 12697–12706, 2021. ↗
  14. A. Y. Sun et al., “Does fine-tuning GPT-3 with the OpenAI API leak personally-identifiable information?,” 2023, [Online]. Available: http://arxiv.org/abs/2307.16382. ↗
  15. F. Jelinek, R. L. Mercer, L. R. Bahl, and J. K. Baker, “Perplexity—a measure of the difficulty of speech recognition tasks,” J. Acoust. Soc. Am., vol. 62, no. S1, pp. S63–S63, 1977, doi: 10.1121/1.2016299. ↗
  16. N. Dehak, R. Dehak, J. Glass, D. Reynolds, and P. Kenny, “Cosine similarity scoring without score normalization techniques,” Odyssey 2010 Speak. Lang. Recognit. Work., pp. 71–75, 2010. find
  17. D. Newman, J. H. Lau, K. Grieser, and T. Baldwin, “Automatic evaluation of topic coherence,” NAACL HLT 2010 - Hum. Lang. Technol. 2010 Annu. Conf. North Am. Chapter Assoc. Comput. Linguist. Proc. Main Conf., no. June, pp. 100–108, 2010.
  18. T. Susnjak, “ChatGPT: The End of Online Exam Integrity?,” pp. 1–21, 2022, [Online]. Available: http://arxiv.org/abs/2212.09292. ↗
  19. V. S. Spelmen and R. Porkodi, “A Review on Handling Imbalanced Data,” Proc. 2018 Int. Conf. Curr. Trends Towar. Converging Technol. ICCTCT 2018, pp. 1–11, 2018, doi: 10.1109/ICCTCT.2018.8551020. ↗
  20. R. C. Kinga et al., “Appropriateness of ChatGPT in answering heart failure related questions Ryan,” 2023. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/biswas-2023-chatgpt-personal-medical-assistant/