Elamur Research⌕
☰
home / papers / 2026 / Linguistic symptoms: retrieval-augmented generation and reasoning in LLMs under Portuguese–English variation in medical contexts

Linguistic symptoms: retrieval-augmented generation and reasoning in LLMs under Portuguese–English variation in medical contexts

Original (Portuguese): Sintomas Linguísticos: Geração Aumentada por Recuperação e Raciocínio em LLMs sob a Variação Português-Inglês em Contextos Médicos

Paper · 2026 · Guilherme Vianna de Moura, Gabriel Assis and Aline Paes

Summary

The paper asks whether general-purpose medical LLMs trained mostly on English stay reliable when questions arrive in Portuguese, and whether retrieval can close any gap. The authors tested two MedGemma sizes, 4B and 27B, on English and Portuguese versions of three medical datasets: medical-o1-reasoning, MedQA, and Global-MMLU. The English-only sets were machine-translated, with the translation model chosen through a back-translation comparison. For the smaller model, they also built a Portuguese RAG pipeline whose knowledge base was drawn from the translated data. They scored accuracy and also analysed the structure of the reasoning chains and answers, which the 27B variant writes out explicitly.

The 4B model performed worse whenever the input was Portuguese. The 27B model held similar accuracy and reasoning structure in both languages. Retrieval quality was judged good, but adding RAG did not reliably help the 4B model, which suggests it cannot make good use of the extra context. The findings show that cross-lingual robustness in clinical QA depends heavily on model scale, which matters for deploying medical LLMs in lower-resource languages.

Key points

  • Research questions. The study poses three questions:

    • how input language affects accuracy;
    • whether Portuguese RAG mitigates language bias;
    • whether language changes explicit reasoning.

  • Data.

    • medical-o1-reasoning has about 19.7k items with GPT-4o-verified answers.
    • MedQA has about 10.1k board-exam multiple-choice questions. It was reduced to about 5.3k by removing items with cosine similarity above 0.75 to medical-o1-reasoning.
    • Global-MMLU contributes about 1.5k medical questions that already exist in both languages.

  • Data roles. Global-MMLU and 500 held-out MedQA items were used for evaluation. The remaining data formed the RAG knowledge base.

  • Translation choice. GemmaX2-28-9B outperformed Llama-3.1 8B on every back-translation metric, for example:

    MetricGemmaX2-28-9BLlama-3.1 8B
    BLEU50.442.7
    xCOMET0.7720.498

    Its translations were used in all experiments.

  • Main result. The Portuguese penalty appears mainly in MedGemma 4B, and RAG does not consistently offset it. MedGemma 27B is largely robust across the two languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 3Research subjects and languages the work is about.English · 2,263 worksReasoning · 1,426 worksInformation retrieval & search · 473 works

Cite

Cite as Guilherme Vianna de Moura, Gabriel Assis and Aline Paes. 2026. Sintomas Linguísticos: Geração Aumentada por Recuperação e Raciocínio em LLMs sob a Variação Português-Inglês em Contextos Médicos. Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{moura2026sintomas,
  title = {{Sintomas Linguísticos: Geração Aumentada por Recuperação e Raciocínio em LLMs sob a Variação Português-Inglês em Contextos Médicos}},
  author = {Guilherme Vianna {de Moura}, Gabriel Assis and Aline Paes},
  booktitle = {Proceedings of the 17th International Conference on Computational Processing of Portuguese (PROPOR 2026) - Vol. 1},
  year = {2026},
  url = {https://aclanthology.org/2026.propor-1.46/}
}

References

40 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohamed Abo El-Enen, Sally Saad, e Taymoor Nazmy. 2025. A survey on retrieval-augmentation generation (rag) models for healthcare applications. Neural Computing and Applications, 37(33):28191–28267. find
  2. AI@Meta. 2024. Llama 3 model card. find
  3. Iñigo Alonso, Maite Oronoz, e Rodrigo Agerri. 2024. Medexpqa: Multilingual benchmarking of large language models for medical question answering. Artificial Intelligence in Medicine, 155:102938. find
  4. Muhammad Arslan, Hussam Ghanem, Saba Munawar, e Christophe Cruz. 2024. A survey on rag with llms. Procedia computer science, 246:3781–3790. find
  5. Eser Aygün, Anastasiya Belyaeva, Gheorghe Comanici, Marc Coram, Hao Cui, Jake Garrison, Renee Johnston Anton Kast, Cory Y. McLean, Peter Norgaard, Zahra Shamsi, David Smalling, James Thompson, Subhashini Venugopalan, Brian P. Williams, Chujun He, Sarah Martinson, Martyna Plomecka, Lai Wei, Yuchen Zh… — An AI system to help scientists write expert-level empirical software
  6. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, S. Buch, Dallas Card, Rodrigo Castellon, Niladri S. Chatterji, Annie S. Chen, Kathleen A. Creel, Jared Davis, Dora Demsz… — On the Opportunities and Risks of Foundation Models
  7. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, e Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with gpt-4. Preprint, arX…
  8. Tyler A Chang, Catherine Arnett, Zhuowen Tu, e Ben Bergen. 2024. When is multilinguality a curse? language modeling for 250 high-and low-resource languages. Em Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, páginas 4074–4096.
  9. Junying Chen, Zhenyang Cai, Ke Ji, Xidong Wang, Wanlong Liu, Rongsheng Wang, e Benyou Wang. 2025. Towards medical complex reasoning with llms through medical verifiable problems. Em Findings of the Association for Computational Linguistics: ACL 2025, páginas 14552–14573, Vienna, Austria. Association… find
  10. Menglong Cui, Pengzhi Gao, Wei Liu, Jian Luan, e Bin Wang. 2025. Multilingual machine translation with open large language models at practical scale: An empirical study. Em Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Hum… find
  11. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, e André F. T. Martins. 2024. xCOMET: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995.
  12. Jiedong Lang, Zhehao Guo, e Shuyu Huang. 2024. A comprehensive study on quantization techniques for large language models. find
  13. Em 2024 4th International Conference on Artificial Intelligence, Robotics, and Communication (ICAIRC), páginas 224–231. find
  14. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, e Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Em Proceedings of the 34th International…
  15. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. Em Text Summarization Branches Out, páginas 74–81, Barcelona, Spain. Association for Computational Linguistics.
  16. Jerry Liu. 2022. LlamaIndex. find
  17. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, e Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  18. Niklas Muennighoff, Nouamane Tazi, Loic Magne, e Nils Reimers. 2023. MTEB: Massive text embedding benchmark. Em Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, páginas 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistic…
  19. Wei Huang, Yangdong Liu, Haotong Qin, Ying Li, Shiming Zhang, Xianglong Liu, Michele Magno, e Xiaojuan Qi. 2024. Billm: pushing the limit of post-training quantization for llms. Em Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. find
  20. Letícia C. Navarro, Filipe Mutz, Thiago M. Paixão, Guilherme G. Zanetti, Claudine Badue, Alberto F. De Souza, e Thiago Oliveira-Santos. 2025. Ragpharma: A rag-based chatbot for medicine leaflets with a dual-dataset evaluation framework. Journal of the Brazilian Computer Society, 31(1):1137–1149. find
  21. Gautier Izacard e Edouard Grave. 2021. Leveraging passage retrieval with generative models for open domain question answering. Em Proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, páginas 874–880.
  22. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt-4o system card.…
  23. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, e Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  24. Won Tae Kim, Jaegwang Shin, In-Sang Yoo, Jae-Woo Lee, Hyun Jeong Jeon, Hyo-Sun Yoo, Yongwhan Kim, Jeong-Min Jo, ShinJi Hwang, Woo-Jeong Lee, Seung Park, e Yong-June Kim. 2024. Medication extraction and drug interaction chatbot: Generative pre-trained transformer-powered chatbot for drug-drug interac… find
  25. Kishore Papineni, Salim Roukos, Todd Ward, e Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. Em Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, página 311–318, USA. Association for Computational Linguistics.
  26. Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, e André F. T. Martins. 2023. Scaling up CometKiwi: Unbabel-IST 2023 submission for the quality estimation shared task. Em Proceedings of the Eighth Conference on Machine Translation, págin…
  27. Long Papers), páginas 18761–18799, Vienna, Austria. Association for Computational Linguistics. find
  28. Ricardo Rei, Craig Stewart, Ana C Farinha, e Alon Lavie. 2020. COMET: A neural framework for MT evaluation. Em Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), páginas 2685–2702, Online. Association for Computational Linguistics.
  29. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, e Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672.
  30. Sreedhar Kumar S, Syed Thouheed Ahmed, Afifa Salsabil Fathima, Nishabai M, e Sophia S. 2024. Medical chatbot assistance for primary clinical guidance using machine learning techniques. Procedia Computer Science, 233:279–287. 5th International Conference on Innovative Data Communication Technologies… find
  31. Elisa Terumi Rubel Schneider, João Vitor Andrioli de Souza, Yohan Bonescki Gumiel, Claudia Moro, e Emerson Cabrera Paraiso. 2021. A gpt-2 language model for biomedical texts in portuguese. find
  32. Em 2021 IEEE 34th International Symposium on Computer-Based Medical Systems (CBMS), páginas 474–479. find
  33. Rui Yang, Yilin Ning, Emilia Keppo, Mingxuan Liu, Chuan Hong, Danielle Bitterman, Jasmine Ong, Daniel Ting, e Nan Liu. 2025. Retrieval-augmented generation for generative artificial intelligence in health care. npj Health Systems, 2. find
  34. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, e Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Em International Conference on Learning Representations. find
  35. Elisa Terumi Rubel Schneider, Fernando Henrique Schneider, Emerson Cabrera Paraiso, Alceu Souza Britto Jr, e Rafael Menelau Oliveira Cruz. 2025. Medgemma-sum-pt: A lightweight model for portuguese clinical summarization. find
  36. Em CLEF 2025 Working Notes: Notebook for the BioASQ Lab at CLEF 2025, Madrid, Spain. CEUR Workshop Proceedings. Available under Creative Commons Attribution 4.0 International (CC BY 4.0). find
  37. Andrew Sellergren, Sahar Kazemzadeh, Tiam Jaroensri, Atilla Kiraly, Madeleine Traverse, Timo Kohlberger, Shawn Xu, Fayaz Jamil, Cían Hughes, Charles Lau, and 1 others. 2025. Medgemma technical report. arXiv preprint arXiv:2507.05201. find
  38. Rico Sennrich, Barry Haddow, e Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. Em Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), páginas 86– 96, Berlin, Germany. Association for Computational… find
  39. Guxue Shan, Xiaonan Chen, Chen Wang, Li Liu, Yuanjing Gu, Huiping Jiang, e Tingqi Shi. 2025. Comparing diagnostic accuracy of clinical professionals and large language models: Systematic review and meta-analysis. JMIR medical informatics, 13:e64963. find
  40. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, Raymond Ng, Shayne Longpre, Sebastian Ruder, Wei-Yin Ko, Antoine Bosselut, Alice Oh, Andre Martins, Leshem Chosh… — Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/moura-2026-sintomas-linguisticos-geracao-aumentada-por/