Elamur Research⌕
☰
home / papers / 2025 / Evaluation of Machine Translation Errors in German Plain Language Texts in the Domain of Health Information

Evaluation of Machine Translation Errors in German Plain Language Texts in the Domain of Health Information

Paper · 2025 · Sarah Ahrens, Silvana Deilen, Sergio Hernandez Garrido et al.

Summary

The paper asks whether machine translation can safely produce Plain German (Einfache Sprache) versions of health information, a need in Germany given widespread low health literacy. The authors draw on an existing corpus of 30 standard German texts from Apotheken Umschau. They compare two systems: SUMM AI's best Plain Language model, which was fine-tuned on domain texts and human gold translations, and ChatGPT-4o, driven by a two-step role-based prompt. Errors were annotated manually with an adapted MQM scheme that sorts them into four categories, covering accuracy, terminology, adherence to linguistic conventions, and suitability for the intended readers. Both systems made many errors in every category, but their profiles differed. ChatGPT produced far more accuracy errors, including mistranslations, ambiguity and hallucinations, while SUMM AI did better on linguistic conventions and also scored higher on SARI. ChatGPT handled audience appropriateness better. The authors conclude that task- and domain-specific fine-tuning beats general prompting. Neither output is safe for end users without human post-editing, which matters in a safety-critical domain.

Key points

  • The data are 30 Apotheken Umschau source texts reused from Deilen et al. (2024a,b). SUMM AI "model 2" was chosen because it had been the strongest of three SUMM AI variants on correctness, readability and SARI.
  • The ChatGPT-4o prompt has two steps:
    • first, ask the model to define Plain Language in accessible communication;
    • then, cast it as a Plain Language expert and give it the text topic and the target readers (people with reading difficulties).
  • The adapted MQM scheme covers terminology (e.g. inconsistent or wrong terms), accuracy, linguistic conventions and audience appropriateness. Adherence to Plain Language guidelines was deliberately left out.
  • Audience-appropriateness errors included two kinds of problems:
    • advice that was not in the source, such as telling readers to search the web;
    • informal or patronising address (ihr, passt auf!) where the formal register was needed.
  • Limitations named by the authors:
    • different annotator teams handled each system, so judgements may not be comparable;
    • error severity was not yet scored;
    • the study covers only one domain;
    • user testing was ruled out on ethical grounds.

Interesting ideas

  • I1. An error taxonomy built for interlingual translation (MQM) can be reused for same-language rewriting, keeping fidelity categories separate from register and audience fit.
  • I2. A surface metric (SARI) can favour a system while saying nothing about hallucinations or omitted content, so the authors pair automatic scores with a manual error audit.
  • I3. Content added beyond the source, and a shift in how the reader is addressed, are counted as errors even when the sentence itself is fluent.
  • I4. Using several terms for one concept within a single text counts as a terminology error.
  • I5. The authors plan to study annotator disagreements as a pointer to genuinely ambiguous cases, and they warn against splitting annotators by system.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Generated or compressed text from the text → graph → text path could be audited with these four categories, with accuracy errors traced back to the meaning graph.
  • I2 (learn): Every generation or simplification benchmark should report a content-error count next to its overlap metrics, so a model cannot look good on surface overlap alone.
  • I3 (back): The generator could check that every output proposition is supported by a source frame, and that formal versus informal register stays consistent.
  • I4 (world): Once entities are resolved to a single concept, the world model could enforce one surface form per entity when answering or regenerating text.
  • I5 (learn): Cases where witness parsers disagree could be collected as an ambiguity set, and each system variant should be judged by the same witnesses to avoid bias.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,087 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Sarah Ahrens, Silvana Deilen, Sergio Hernandez Garrido, Ekaterina Lapshinova-Koltunski and Christiane Maaß. 2025. Evaluation of Machine Translation Errors in German Plain Language Texts in the Domain of Health Information. Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Workshops.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{ahrens2025evaluation,
  title = {{Evaluation of Machine Translation Errors in German Plain Language Texts in the Domain of Health Information}},
  author = {Sarah Ahrens, Silvana Deilen, Sergio Hernandez Garrido, Ekaterina Lapshinova-Koltunski and Christiane Maaß},
  booktitle = {Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Workshops},
  year = {2025},
  url = {https://aclanthology.org/2025.konvens-2.13/}
}

References

22 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sarah Ahrens, Silvana Deilen, Ekaterina Lapshinova-Koltunski, Sergio Hernández Garrido, and Christiane Maaß. 2025. Evaluation of translations into plain german produced by humans and mt systems including chatgpt. SKASE Journal of Translation and Interpretation. In press. ↗
  2. Sarah Ahrens and Christiane Maaß. 2024. Der Einfluss von Vorwissen und sprachlichen Texteigen-schaften auf die Anwendung des Fragebogens in einer gynäkologischen Patientenaufklärung: eine leitfadengestützte Interviewstudie. Prävention und Gesundheitsförderung, 19:504–511. find
  3. Miriam Anschütz, Joshua Oehms, Thomas Wimmer, Bartłomiej Jezierski, and Georg Groh. 2023. Language models for german text simplification: Overcoming parallel data scarcity through style-specific pre-training. arXiv preprint arXiv:2305.12908. ↗
  4. The MQM Council. n.d. The MQM error typology. Last accessed on Feb 18th 2025. find
  5. Silvana Deilen, Sergio Hernández Garrido, Ekaterina Lapshinova-Koltunski, and Christiane Maaß. 2023. Using chatgpt as a cat tool in easy language translation. arXiv preprint arXiv:2308.11563. ↗
  6. Silvana Deilen, Ekaterina Lapshinova-Koltunski, Sergio Garrido, Julian Hörner, Christiane Maaß, Vanessa Theel, and Sophie Ziemer. 2024a. Evaluation of intralingual machine translation for health communication. In Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pages 469–479. find
  7. Silvana Deilen, Ekaterina Lapshinova-Koltunski, Sergio Hernández Garrido, Christiane Maaß, Julian Hörner, Vanessa Theel, and Sophie Ziemer. 2024b. Towards AI-supported Health Communication in Plain Language: Evaluating Intralingual Machine Translation of Medical Texts. In Proceedings of the First Workshop on Patient-Oriented Language Processing (CL4Health)@ LREC-COLING 2024, pages 44–53. find
  8. Silvia Hansen-Schirra, Walter Bisang, Arne Nagels, Silke Gutermuth, Julia Fuchs, Liv Borghardt, Silvana Deilen, Anne-Kathrin Gros, Laura Schiffl, and Johanna Sommer. 2020. Intralingual translation into Easy language–or how to reduce cognitive processing costs. Easy Language Research: Text and User Perspectives. Berlin: Frank & Timme, pages 197–225. find
  9. Janina Kröger and Christiane Maaß. 2024. Mangelnde verständlichkeit durch fachsprache in gesundheitsin-formationen zu chronischen erkrankungen – eine qualitative korpusanalyse. Prävention und Gesundheitsförderung, 19:497–503. find
  10. Arle Lommel, Hans Uszkoreit, and Aljoscha Burchardt. 2014. Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics. Tradumàtica, 12:0455–463. ↗
  11. Christiane Maaß. 2020. Easy language–Plain language– Easy language plus: Balancing comprehensibility and acceptability. Frank & Timme. ↗
  12. Christiane Maaß. 2024a. Hi ChatGPT, translate this text into Easy Language. Is the new Easy Language translator a machine? In Proceedings of the XLIV International VAKKI Symposium. find
  13. Christiane Maaß. 2024b. Intralingual Translation in Easy Language and Plain Language. In Routledge Handbook of Intralingual Translation, pages 234– 251. Routledge. find
  14. Silvia Rodríguez Vázquez, Abigail Kaplan, Pierrette Bouillon, Cornelia Griebel, and Razieh Azari. 2022. La traduction automatique des textes faciles à lire et à comprendre (falc): une étude comparative. Meta, 67(1):18–49. ↗
  15. Constanze Rossmann and Matthias R Hastall. 2019. Handbuch der Gesundheitskommunikation. Springer. find
  16. Doris Schaeffer, Eva-Maria Berens, Svea Gille, Lennert Griese, Klinger Julia, Steffen de Sombre, Dominique Vogt, and Klaus Hurrelmann. 2021. Gesundheitskompetenz der Bevölkerung in Deutschland vor und während der Corona Pandemie: Ergebnisse des HLS-GER 2. Universität Bielefeld, Interdisziplinäres Zentrum für Gesundheitskompetenzforschung. ↗
  17. Doris Schaeffer, Klaus Hurrelmann, Ullrich Bauer, and Kai Kolpatzik. 2018a. National Action Plan Health Literacy. Promoting health literacy in Germany. Kompart, Berlin. find
  18. Doris Schaeffer, Klaus Hurrelmann, Ulrich Bauer, Kai Kolpatzik, and Attila Altiner. 2018b. Nationaler Aktionsplan Gesundheitskompetenz: die Gesundheitskompetenz in Deutschland stärken: Informationen bewerten, finden, anwenden, verstehen. Hertie School of Governance. find
  19. Doris Schaeffer, Dominique Vogt, Eva-Maria Berens, and Klaus Hurrelmann. 2017. Gesundheitskompetenz der bevölkerung in deutschland: ergebnisbericht. ↗
  20. Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, and 4 others. 2022. Taxonomy of Risks posed by Language Models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, page 214–229, New York, NY, USA. Association for Computing Machinery. find
  21. Theresa Isabelle Wilhelm, Jonas Roos, and Robert Kaczmarczyk. 2023. Large language models for therapy recommendations across 3 clinical specialties: Comparative study. J Med Internet Res, 25:e49324. ↗
  22. Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401–415. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ahrens-2025-evaluation-machine-translation-errors-german/