Elamur Research⌕
☰
home / papers / 2024 / Synthetic Arabic Medical Dialogues Using Advanced Multi-Agent LLM Techniques

Synthetic Arabic Medical Dialogues Using Advanced Multi-Agent LLM Techniques

Paper · 2024 · Mariam ALMutairi, Lulwah AlKulaib, Melike Aktas et al.

Summary

Arabic medical NLP lacks open doctor–patient conversation data. Real consultations are private and seldom recorded, and machine translation of English corpora tends to produce Modern Standard Arabic, losing regional dialect. The authors take the English clinical notes of ACI-bench and use a multi-agent LLM pipeline to produce synthetic consultations in the Saudi Najdi dialect. The generation step relies on Claude-3-Opus, and GPT-4 is also part of the system. Prompts ask for a long, layman-friendly, culturally sensitive conversation that walks through every part of the note, from complaint and history through exams and results to the plan. Output quality is checked in three ways: overlap metrics (ROUGE, BERTScore), LLM-based scoring with Claude-3-Opus and GPT-4-Turbo, and review by domain experts. Model-based ratings came out close to the maximum. Human reviewers judged the dialogues faithful and well structured, but they also found factual slips and terminology that patients would struggle with. The resulting corpus is released openly, giving Arabic medical dialogue research a shared resource.

Key points

  • Source data is ACI-bench (Yim et al., 2023):
    • 207 dialogue–note pairs, the largest public set for model-assisted clinical note generation;
    • notes are split into subjective, objective_exam, objective_results and assessment_and_plan;
    • the English notes serve as the seed for generation.
  • The Generation Agent uses Claude-3-Opus-20240229.
    • It was chosen over GPT-4 because it handled the Najdi dialect better in a small pilot comparison.
    • The target length is roughly 50 turns and about 3000 words per dialogue.
  • LLM-based scores:
    • Claude-3-Opus: 4.87 overall, 4.71 for medical accuracy/completeness, 4.94 for communication/rapport;
    • GPT-4-Turbo: top marks across the board, with 4.99 in both of those categories.
  • Human experts found two main problems:
    • content contradictions, such as a dialogue placing pain in the right knee when the note says both knees;
    • overly technical terms in three generations.
  • The dataset is published on Hugging Face as Mars203020/arabic_medical_dialogue. Proposed future work is a cross-lingual similarity measure suited to long texts and Arabic medical terms.

Interesting ideas

  • I1. The generator is fed a sectioned, structured record and told to cover each section in turn, so the output is grounded in the source rather than invented freely.
  • I2. The model for a particular language variety was picked through a small pilot comparison rather than by reputation.
  • I3. Automatic and model-based scores sat near the top of the scale, yet human experts still found concrete errors, which shows that ceiling scores hide faults.
  • I4. The most telling error type was a contradiction of one specific fact (one knee versus both), not a matter of style.
  • I5. Reviewers flagged wording too technical for the intended listener, which treats audience register as part of output quality.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Generation from the world of a text could walk the entities, states and events in a fixed order, giving a coverage checklist that makes each sentence traceable to a graph node.
  • I2 (learn): Before assigning a teacher tier to a genre (fables, word problems, school tests), run a cheap pilot on a handful of items and keep the decision and its evidence on record.
  • I3 (learn): Evaluation should include negative controls, such as deliberately corrupted outputs, to confirm a judge can actually tell bad from good before its scores are trusted.
  • I4 (world): Parse generated or paraphrased text back into a world model and diff its states against the source world, so that mismatches such as quantity or laterality are flagged automatically and explainably.
  • I5 (back): The text generator could carry a register parameter that swaps specialist lemmas for plainer WordNet neighbours when the target reader is a layperson or a child.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Virginia TechBlacksburg · United States36 works · 43,036 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksMedicine · 241 worksHealth Informatics · 184 works

🧭 Topics 2Research subjects and languages the work is about.Semitic languages · 2,908 worksDialogue systems · 624 works

Cite

Cite as Mariam ALMutairi, Lulwah AlKulaib, Melike Aktas, Sara Alsalamah and Chang-Tien Lu. 2024. Synthetic Arabic Medical Dialogues Using Advanced Multi-Agent LLM Techniques. Proceedings of the Second Arabic Natural Language Processing Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.arabicnlp-1.2

@inproceedings{almutairi2024synthetic,
  title = {{Synthetic Arabic Medical Dialogues Using Advanced Multi-Agent LLM Techniques}},
  author = {Mariam ALMutairi, Lulwah AlKulaib, Melike Aktas, Sara Alsalamah and Chang-Tien Lu},
  booktitle = {Proceedings of the Second Arabic Natural Language Processing Conference},
  year = {2024},
  doi = {10.18653/v1/2024.arabicnlp-1.2},
  url = {https://doi.org/10.18653/v1/2024.arabicnlp-1.2}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abrar Al Oudah, Kholoud Al Bassam, Heba Kurdi, and Shiroq Al-Megren. 2019. Wajeez: An extractive automatic arabic text summarisation system. In Social Computing and Social Media. Design, Human Behavior and Analytics: 11th International Conference, SCSM 2019, Held as Part of the 21st HCI International Conference, HCII 2019, Orlando, FL, USA, July 26-31, 2019, Proceedings, Part I 21, pages 3–14. Springer. find
  2. Jaafar Hammoud, Aleksandra Vatian, Natalia Dobrenko, Nikolai Vedernikov, Anatoly Shalyto, and Natalia Gusarova. 2021. New arabic medical dataset for diseases classification. In Intelligent Data Engineering and Automated Learning–IDEAL 2021: 22nd International Conference, IDEAL 2021, Manchester, UK, November 25–27, 2021, Proceedings 22, pages 196– 203. Springer. Tahmid Hasan, Abhik Bhattacharjee, Md.
  3. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang, M. Sohel Rahman, and Rifat Shahriyar. 2021. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4693–4703, Online. Association for Computational Linguistics. find
  4. Abbas Raza Ali, Muhammad Ajmal Siddiqui, Rema Algunaibet, and Hasan Raza Ali. 2023. A large and diverse arabic corpus for language modeling. Procedia Computer Science, 225:12–21. Mohammed Alsuhaibani. Fine-tuned pegasus: Exploring the performance of the transformer-based model on a diverse text summarization dataset.
  5. Rihab A HASANAIN, Kirsten Vallmuur, and Michele Clark. 2015. Electronic medical record systems in saudi arabia: knowledge and preferences of healthcare professionals. Journal of Health Informatics in Developing Countries, 9(1). Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, ukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, ukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Severin Husmann, Hugo Yèche, Gunnar Rätsch, and Rita Kuznetsova. 2022. On the importance of clinical notes in multi-modal learning for ehr data. arXiv preprint arXiv:2212.03044. ↗
  6. Jiawen Jiang, Haiyang Zhang, Chenxu Dai, Qingjuan Zhao, Hao Feng, Zhanlin Ji, and Ivan Ganchev. 2021. Enhancements of attention-based bidirectional lstm for hybrid automatic text summarization. IEEE Access, 9:123660–123671. ↗
  7. A Koubaa, A Ammar, L Ghouti, O Necar, and S Sibaee. 2024. Arabiangpt: Native arabic gpt-based large language model. ↗
  8. Kundan Krishna, Sopan Khosla, Jeffrey P Bigham, and Zachary C Lipton. 2020. Generating soap notes from doctor-patient conversations using modular summarization techniques. arXiv preprint arXiv:2005.01795. ↗
  9. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  10. George Michalopoulos, Kyle Williams, Gagandeep Singh, and Thomas Lin. 2022. Medicalsum: A guided clinical abstractive summarization model for generating medical reports from patient-doctor conversations. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4741– 4749. find
  11. Jinane Mounsef, Maheen Hasib, and Ali Raza. 2022. Building an arabic dialectal diagnostic dataset for healthcare. International Journal of Advanced Computer Science and Applications, 13(7). ↗
  12. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024. Gpt-4 technical report. Preprint, arXiv:2303.08774. ↗
  13. Jonathan Pilault, Raymond Li, Sandeep Subramanian, and Christopher Pal. 2020. On extractive and abstractive neural document summarization with transformer language models. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 9308–9319. ↗
  14. Thomas Schaaf, Longxiang Zhang, Alireza Bayeste-htashk, Mark Fuhs, Shahid Durrani, Susanne Burger, Monika Woszczyna, and Thomas Polzin. 2021. Are you dictating to me? detecting embedded dictations in doctor-patient conversations. In 2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 587–593. IEEE. find
  15. Natalie Schluter. 2017. The limits of automatic summarisation according to ROUGE. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 41–45, Valencia, Spain. Association for Computational Linguistics.
  16. Deeksha Varshney, Aizan Zafar, Niranshu Kumar Behera, and Asif Ekbal. 2023. Knowledge grounded medical dialogue generation using augmented graphs. Scientific Reports, 13(1):3310. ↗
  17. Chen Yan, Susannah Rose, Michael B Rothberg, Mary Beth Mercer, Kenneth Goodman, and Anita D Misra-Hebert. 2016. Physician, scribe, and patient perspectives on clinical scribes in primary care. Journal of general internal medicine, 31:990–995. ↗
  18. Wen-wai Yim, Yujuan Fu, Asma Ben Abacha, Neal Snider, Thomas Lin, and Meliha Yetisgen. 2023. Aci-bench: a novel ambient clinical intelligence dataset for benchmarking automatic visit note generation. Scientific Data, 10(1):586. find
  19. Sumaya Al Zarouni, Noor Majed Al Mheiri, Kalthoom Al Blooshi, Yousif Al Serkal, Neema Pre-man, Sadaf Ahsan Naqvi, and Yasir Khan. 2022. Impact of an electronic medical record-based automated screening program for critical congenital heart disease: Emirates health services, united arab emirates. BMC Medical Informatics and Decision Making, 22(1):165. ↗
  20. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Preprint, arXiv:1904.09675. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/almutairi-2024-synthetic-arabic-medical-dialogues-advanced/