Elamur Research⌕
☰
home / papers / 2024 / Advancing Complex Medical Communication in Arabic with Sporo AraSum: Surpassing Existing Large Language Models

Advancing Complex Medical Communication in Arabic with Sporo AraSum: Surpassing Existing Large Language Models

Summary

The paper tackles automatic summarization of doctor–patient conversations in Arabic. Rich morphology, flexible syntax, diglossia and scarce medical dialogue corpora make this hard for general Arabic models. Sporo Health compares its proprietary model, Sporo AraSum, against Inception AI's JAIS. Its architecture and training are not described in any detail.

For automatic scoring, the authors used GPT-4o to create 4,000 synthetic Arabic consultations, with GPT-4o reference summaries translated into Arabic. They scored outputs with item-level clinical content precision, recall and F1, plus ROUGE, BLEU and BERTScore.

For human judgment, three short clinical vignettes were summarized zero-shot by both models. Blinded Arabic-fluent evaluators rated the summaries using a PDQI-9 variant extended with three language-oriented attributes.

AraSum scored higher on every automatic metric, most strongly in recall and F1. The authors also report that JAIS sometimes produced truncated summaries.

The work matters as an early domain-specific evaluation template for clinical Arabic. However, it rests on synthetic data and a very small human study, and the per-attribute human scores did not survive extraction.

Key points

  • Data: 4,000 synthetic Arabic patient–physician dialogues were generated with GPT-4o. The reference summaries were also produced by GPT-4o and then translated into Arabic.
  • Clinical content scores (JAIS vs AraSum):
    • precision 0.364 vs 0.557
    • recall 0.160 vs 0.549
    • F1 0.220 vs 0.552
  • Overlap and similarity scores (JAIS vs AraSum):
    • ROUGE-1/2/L: 0.000 for JAIS vs 0.452/0.354/0.452 for AraSum
    • BLEU 0.012 vs 0.209
    • BERTScore F1 0.724 vs 0.807
  • Human evaluation set: three vignettes (soap-induced rash, palpitations with hypertension, tonsillitis). Clinician-written summaries, translated into Arabic, served as references. Both models were prompted zero-shot.
  • Rubric: Tierney et al.'s ten-item PDQI-9 variant for ambient AI notes, extended with three newly added items named Cultural Competence, Syntactic Proficiency, and Domain-Specific Linguistic Precision. That gives 13 attributes.

Interesting ideas

  • I1. Instead of n-gram overlap, the authors pull salient facts from each source into an inventory and count how many a summary covers (recall) versus how many of its claims are correct (precision); this measures content fidelity directly and works for any summarizer. — For Mova (back): The text → graph → text compressor could be scored by listing the facts in the source world and checking which ones survive in regenerated text, and which spurious ones appear.
  • I2. They take an established quality rubric and add only the language-specific axes it lacks, such as native-like syntax and domain-appropriate wording, so results stay comparable with prior work while covering new failure modes. — For Mova (learn): Mova's evaluation suites could keep a fixed core rubric and attach small per-task extensions, such as fluency of generated English or correct use of domain vocabulary.
  • I3. Human raters see both system outputs with identical formatting and no source labels, which removes style or brand cues from side-by-side judgments. — For Mova (learn): When comparing Mova's answers against an LLM baseline, normalize presentation and hide system identity before human or LLM-teacher grading.
  • I4. Where real annotated data is missing, the authors have an LLM synthesize realistic dialogues as an evaluation set, and they openly flag this as a validity limit. — For Mova (learn): Synthetic fables or word problems could fill coverage gaps, paired with a small real-data control set to detect when synthetic results diverge.
  • I5. The cultural-competence item recognizes that clinicians may refer to a disease indirectly (cancer as "the illness") or lack a direct term for a concept like palliative care. — For Mova (frames): A euphemism/indirect-reference layer in prag could link such vague mentions to their intended referent via implicature, and expose that link as an explainable step for coreference.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2411.13518 · arXiv 2411.13518

@misc{lee2024advancing,
  title = {{Advancing Complex Medical Communication in Arabic with Sporo AraSum: Surpassing Existing Large Language Models}},
  author = {Lee, Chanseo and Kumar, Sonu and Vogt, Kimon A. and Meraj, Sam and Vogt, Antonia},
  year = {2024},
  doi = {10.48550/arxiv.2411.13518},
  eprint = {2411.13518},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.13518}
}

References

14 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tibyan Corpus: Balanced and Comprehensive Error Coverage Corpus Using ChatGPT for Arabic Grammatical Error Correction. Arxiv.org. Published 2014. Accessed November 9, 2024. https://arxiv.org/html/2411.04588v1 ↗ our card
  2. Lee C, Vogt KA and Kumar S (2024) Prospects for AI clinical summarization to reduce the burden of patient chart review. Front. Digit. Health 6:1475092. doi: 10.3389/fdgth.2024.1475092 ↗
  3. Alexis Amid Neme, Eric Laporte. Pattern-and-root inflectional morphology: the Arabic broken plural. Language Sciences, 2013, 40, pp.221-250. ff10.1016/j.langsci.2013.06.002ff. ffhal-00831338f find our card
  4. Antoun W, Baly F, Hajj H. AraBERT: Transformer-based model for Arabic language understanding. In: Al-Khalifa H, Magdy W, Darwish K, Elsayed T, Mubarak H, eds. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection. Marseille, France: European Language Resource Association; 2020:9-15. Available from: https://aclanthology.org/2020.osact-1.2 ↗ our card
  5. 1.The JAIS Family. Core42; 2024. Accessed November 8, 2024. https://www.core42.ai/files/Whitepaper-Jais-30B.pdf ↗
  6. ALMutairi M, AlKulaib L, Aktas M, Alsalamah S, Lu CT. Synthetic Arabic medical dialogues using advanced multi-agent LLM techniques. In: Habash N, Bouamor H, Eskander R, Tomeh N, Abu Farha I, Abdelali A, Touileb S, Hamed I, Onaizan Y, Alhafni B, Antoun W, Khalifa S, Haddad H, Zitouni I, AlKhamissi B, Almatham R, Mrini K, eds. Proceedings of The Second Arabic Natural Language Processing Conference. Bangkok, Thailand: Association for Computational Linguistics; 2024:11-26. doi:10.18653/v1/2024.arabicnlp-1.2. Available from: https://aclanthology.org/2024.arabicnlp-1.2 ↗
  7. Hicks SA, Strümke I, Thambawita V, Hammou M, Riegler MA, Halvorsen P, Parasa S. On evaluation metrics for medical applications of artificial intelligence. Sci Rep. 2022 Apr 8;12(1):5979. doi: 10.1038/s41598-022-09954-8. PMID: 35395867; PMCID: PMC8993826. ↗
  8. Lin CY. ROUGE: A Package for Automatic Evaluation of Summaries.; 2004. https://aclanthology.org/W04-1013.pdf ↗
  9. Papineni K, Roukos S, Ward T, Zhu WJ. BLEU: a method for automatic evaluation of machine translation. In: Proceedings of the 40th Annual Meeting on Association for Computational Linguistics; July 7-12, 2002; Philadelphia, PA. Association for Computational Linguistics; 2002:311-318. find our card
  10. Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. BERTScore: Evaluating text generation with BERT. arXiv. Published online April 21, 2019. doi:10.48550/arXiv.1904.09675. ↗
  11. Stetson PD, Bakken S, Wrenn JO, Siegler EL. Assessing Electronic Note Quality Using the Physician Documentation Quality Instrument (PDQI-9). Appl Clin Inform. 2012;3(2):164-174. doi: 10.4338/aci-2011-11-ra-0070. PMID: 22577483; PMCID: PMC3347480. ↗
  12. Tierney AA, Gayre G, Hoberman B, et al. Ambient Artificial Intelligence Scribes to Alleviate the Burden of Clinical Documentation. NEJM Catalyst Innovations in Care Delivery. 2024;5(3). doi:https://doi.org/10.1056/cat.23.0404 ↗
  13. Lee, C. et al (2024). Improving Clinical Documentation with AI: A Comparative Study of Sporo AI Scribe and GPT-4o mini. arXiv. https://arxiv.org/abs/2410.15528 ↗
  14. Lee, C. et al (2024). Ambient AI scribing support: Comparing the performance of specialized AI agentic architecture to leading foundational models. arXiv. https://arxiv.org/abs/2411.06713 ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lee-2024-advancing-complex-medical-communication-arabic/