Elamur Research⌕
☰
home / papers / 2026 / Evaluating LLM-based Text Simplification for German: Effects on Post-Editing Effort, Quality Ratings, and User Comprehension

Evaluating LLM-based Text Simplification for German: Effects on Post-Editing Effort, Quality Ratings, and User Comprehension

Paper · 2026 · Luisa Carrer, Andreas Säuberli, Martin Kappus et al.

Summary

The paper asks whether current LLM-based simplifiers for German are useful in realistic document-level Easy Language production. Earlier work found that neural systems gave post-editors no measurable speed-up. The authors evaluated two closed systems: the commercial capito.ai at CEFR level A2, and zero-shot GPT-4o (snapshot gpt-4o-2024-08-06). They used eight source documents taken from an Austrian provider's parallel corpus. Each document was rendered in five simplified versions: two raw machine outputs, a fully manual simplification, and post-edited versions of each machine output. Three stakeholder groups took part:

  • Post-editors, whose effort was measured.
  • Professional Easy Language experts, who rated quality.
  • End users, whose perceived and actual comprehension were tested.

Semi-automatic simplification raised productivity by roughly 30% compared with manual work. Expert ratings were broadly favourable, and GPT-4o outputs improved user comprehension. The groups differed noticeably in how they perceived and understood the texts. The study argues that German ATS has advanced and that a single evaluator type is not enough to judge whether it serves accessibility goals.

Key points

  • Systems and outputs: capito.ai (A2 level) and GPT-4o were tested. GPT-4o got a short zero-shot prompt asking for an Easy Language rendering with no commentary. The five target versions per text were capito, gpt-4o, human, capito+pe and gpt-4o+pe.
  • Source texts: eight texts of 100–300 words, covering news, administrative and tourism genres. They were chosen to avoid widely familiar topics and to need no extra context. Austrian wording and spelling were adjusted for Swiss readers (e.g., Straße > Strasse).
  • Productivity: post-editing machine output was about 30% faster than simplifying from scratch.
  • Text measures and readability: the two raw LLM outputs had similar length and vocabulary measures. The post-edited and manual versions were longer and lexically less varied, which suggests that humans added explanations (explicitation). Readability, measured with the Amstad Flesch-Index, varied less across versions.
  • Expert verdicts: experts often judged that even the human and post-edited texts were not yet true Easy Language. They cited subordinate clauses, passives, unexplained terms and dense layout.

Interesting ideas

  • I1. Judging the same outputs through three lenses (editor effort, expert rating, reader comprehension) reveals disagreements that a single metric would hide.
  • I2. Post-editing effort is split into time, edit actions and cognitive load, where cognitive load is approximated by clusters of short typing pauses. This gives a fine-grained cost of fixing machine output.
  • I3. Multiple-choice comprehension items include a "not in the text" distractor, which catches readers or systems that invent content.
  • I4. Comparing simple surface statistics across versions (length, unique words, type/token ratio) exposed that humans add explanations while LLMs mostly cut.
  • I5. The gap between how well readers think they understood a text and how well they actually did is treated as a finding in its own right.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (answer): Score answers both by gold-test accuracy and by an independent "would a reader actually get this" check, and flag cases where the two disagree.
  • I2 (learn): Measure how much repair LLM-teacher or silver parses need (edit counts, correction time) to rank teacher tiers by real cost rather than by raw accuracy.
  • I3 (answer): Add an explicit "not stated" option to fable and tale QA as a negative control, so that guessing from world knowledge is penalised.
  • I4 (back): Track the same statistics between input and text regenerated through the meaning graph, to detect whether the compressor silently drops or adds content.
  • I5 (learn): Log model confidence next to correctness on each evaluation set to spot overconfident stages in the pipeline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ludwig-Maximilians-Universität MünchenMunich · Germany107 works · 56,478 citations🎓University of ZurichZurich · Switzerland78 works · 12,260 citations🎓ZHAW Zurich University of Applied SciencesWinterthur · Switzerland6 works · 181 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,087 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Luisa Carrer, Andreas Säuberli, Martin Kappus, Lukas Fischer and Sarah Ebling. 2026. Evaluating LLM-based Text Simplification for German: Effects on Post-Editing Effort, Quality Ratings, and User Comprehension. Proceedings of the Fifteenth Language Resources and Evaluation Conference.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3688akbpcnjn

@inproceedings{carrer2026evaluating,
  title = {{Evaluating LLM-based Text Simplification for German: Effects on Post-Editing Effort, Quality Ratings, and User Comprehension}},
  author = {Luisa Carrer, Andreas Säuberli, Martin Kappus, Lukas Fischer and Sarah Ebling},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3688akbpcnjn},
  url = {https://doi.org/10.63317/3688akbpcnjn}
}

References

65 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sweta Agrawal and Marine Carpuat. 2024. Do text simplification systems preserve meaning? A human evaluation via reading comprehension. Transactions of the Association for Computational Linguistics, 12:432–448. find
  2. Sarah Ahrens, Silvana Deilen, Sergio Hernandez Garrido, Ekaterina Lapshinova-Koltunski, and Christiane Maaß. 2025. Evaluation of machine translation errors in German Plain Language texts in the domain of health information. In Proceedings of the 21st Conference on Natural Language Processing (KONVENS 2025): Workshops, pages 176–185, Hannover, Germany. HsH Applied Academics.
  3. Suha S. Al-Thanyyan and Aqil M. Azmi. 2021. Automated text simplification: A survey. ACM Comput. Surv., 54(2). find
  4. Oliver Alonzo, Jessica Trussell, Becca Dingman, and Matt Huenerfauth. 2021. Comparison of methods for evaluating complexity of simplified texts among deaf and hard-of-hearing adults at different literacy levels. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, Yokohama, Japan. Association for Computing Machinery. find
  5. Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2020. Data-driven sentence simplification: Survey and benchmark. Computational Linguistics, 46(1):135–187. ↗
  6. Fernando Alva-Manchego, Carolina Scarton, and Lucia Specia. 2021. The (un)suitability of automatic evaluation metrics for text simplification. Computational Linguistics, 47(4):861–889. ↗
  7. Sergi Alvarez-Vidal and Antoni Oliver. 2023. Assessing MT with measures of PE effort. Ampersand, 11:100125. ↗
  8. Fabio Alves. 2003. Triangulating Translation: Perspectives in process oriented research. John Benjamins Publishing Company, Amsterdam. find
  9. Toni Amstad. 1978. Wie verständlich sind unsere Zeitungen? Ph.D. thesis, University of Zurich, Zürich. find
  10. Heather Becker, Greg Roberts, Janet Morrison, and Julie Silver. 2004. Recruiting people with disabilities as research participants: Challenges and strategies to address them. Mental Retardation, 42(6):471–475. ↗
  11. Nadine Beks van Raaij, Daan Kolkman, and Ksenia Podoynitsyna. 2024. Clearer governmental communication: Text simplification with Chat-GPT evaluated by quantitative and qualitative research. In Proceedings of the Workshop on DeTermIt! Evaluating Text Difficulty in a Multilingual Context @ LREC-COLING 2024, pages 152–178, Torino, Italia. ELRA and ICCL. find
  12. Gilles Caporossi, Christophe Leblay, and Hakim Usoof. 2023. GenoGraphiX-Log version 2.0. User guide. Les Cahiers Du GERAD G-2020-68, pages 1–63. find
  13. Luisa Carrer, Andreas Säuberli, Martin Kappus, and Sarah Ebling. 2024. Towards holistic human evaluation of automatic text simplification. In Proceedings of the Fourth Workshop on Human Evaluation of NLP Systems (HumEval) @ LREC-COLING 2024, pages 71–80, Torino, Italia. ELRA and ICCL. ↗
  14. Liam Cripwell, Joël Legrand, and Claire Gardent. 2024. Evaluating document simplification: On the importance of separately assessing simplicity and meaning preservation. In Proceedings of the 3rd Workshop on Tools and Resources for People with REAding DIfficulties (READI) @ LREC-COLING 2024, pages 1–14, Torino, Italia. ELRA and ICCL. ↗
  15. Scott A Crossley, Hae Sung Yang, and Danielle S McNamara. 2014. What’s so simple about simplified texts? A computational and psycholinguistic investigation of text comprehension and text processing. Reading in a Foreign Language, 26(1):92–113. find
  16. Cristina Cumbreño and Nora Aranberri. 2021. What Do You Say? Comparison of Metrics for Post-editing Effort, page 57–79. Springer International Publishing. find
  17. Silvana Deilen, Sergio Hernández Garrido, Ekaterina Lapshinova-Koltunski, and Christiane Maaß. 2023. Using ChatGPT as a CAT tool in Easy Language translation. In Proceedings of the Second Workshop on Text Simplification, Accessibility and Readability, pages 1–10, Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria. ↗
  18. Silvana Deilen, Ekaterina Lapshinova-Koltunski, Sergio Garrido, Julian Hörner, Christiane Maaß, Vanessa Theel, and Sophie Ziemer. 2024. Evaluation of intralingual machine translation for health communication. In Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pages 469–479, Sheffield, UK. European Association for Machine Translation (EAMT). find
  19. Christa Dürscheid, Stephan Elspaß, and Arne Ziegler. 2015. Variantengrammatik des Standarddeutschen. Konzeption, methodische Fragen, Fallanalysen, pages 207–236. VR Unipress. find
  20. Sarah Ebling, Alessia Battisti, Marek Kostrzewa, Dominik Pfütze, Annette Rios, Andreas Säuberli, and Nicolas Spring. 2022. Automatic text simplification for German. Frontiers in Communication, Volume 7 - 2022. ↗
  21. ELIS Research. 2025. European language industry survey 2025: Trends, expectations and concerns of the European language industry. Accessed: 16 Oct 2025. find
  22. Inmaculada Fajardo, Vicenta Ávila, Antonio Ferrer, Gema Tavares, Marcos Gómez, and Ana Hernández. 2014. Easy-to-read texts for students with intellectual disability: linguistic factors affecting comprehension. Journal of applied research in intellectual disabilities, 27(3):212–225. ↗
  23. Christian Federmann. 2018. Appraise evaluation framework for machine translation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 86–88, Santa Fe, New Mexico. Association for Computational Linguistics. find
  24. W. M. L. Finlay and E. Lyons. 2001. Methodological issues in interviewing and using self-report questionnaires with people with mental retardation. Psychological Assessment, 13(3):319–335. ↗
  25. Sian Gooding. 2022. On the ethical considerations of text simplification. In Ninth Workshop on Speech and Language Processing for Assistive Technologies (SLPAT-2022), pages 50–57, Dublin, Ireland. Association for Computational Linguistics. ↗
  26. Sian Gooding, Yevgeni Berzak, Tony Mak, and Matt Sharifi. 2021. Predicting text readability from scrolling interactions. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 380–390, Online. Association for Computational Linguistics. find
  27. Natalia Grabar and Horacio Saggion. 2022. Evaluation of automatic text simplification: Where are we now, where should we go from here. In Actes de la 29e Conférence sur le Traitement Automatique des Langues Naturelles. Volume 1 : conférence principale, pages 453–463, Avignon, France. ATALA. find
  28. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. find
  29. Theo Guidroz, Diego Ardila, Jimmy Li, Adam Mansour, Paul Jhun, Nina Gonzalez, Xiang Ji, Mike Sanchez, Sujay Kakarmath, Mathias MJ Bellaiche, Miguel Ángel Garrido, Faruk Ahmed, Divyansh Choudhary, Jay Hartford, Chenwei Xu, Henry Javier Serrano Echeverria, Yifan Wang, Jeff Shaffer, Eric, Cao, Yossi Matias, Avinatan Hassidim, Dale R Webster, Yun Liu, Sho Fujiwara, Peggy Bui, and Quang Duong. 2025. LLM-based text simplification and its effect on user comprehension and cognitive load. ↗
  30. Matt Huenerfauth, Lijun Feng, and Noémie Elhadad. 2009. Comparing evaluation techniques for text readability software for adults with intellectual disabilities. In Proceedings of the 11th International ACM SIGACCESS Conference on Computers and Accessibility, Assets ’09, page 3–10, Pittsburgh, Pennsylvania, USA. Association for Computing Machinery. find
  31. Glyn Jones. 2020. Designing multiple-choice test items. In The Routledge Handbook of Second Language Acquisition and Language Testing, chapter 9, pages 90–101. Routledge. ↗
  32. Martin Kappus and Maureen Ehrensberger-Dow. 2020. The ergonomics of translation tools: understanding when less is actually more. The Interpreter and Translator Trainer, 14(4):386–404. ↗
  33. Tannon Kew, Alison Chi, Laura Vásquez-Rodríguez, Sweta Agrawal, Dennis Aumiller, Fernando Alva-Manchego, and Matthew Shardlow. 2023. BLESS: Benchmarking large language models on sentence simplification. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 13291–13309, Singapore. Association for Computational Linguistics. find
  34. Maarit Koponen. 2016. Is machine translation post-editing worth the effort? A survey of research into post-editing and effort. The Journal of Specialised Translation, (25):131–148. find
  35. Hans P. Krings. 2001. Repairing Texts: Empirical Investigations of Machine Translation Post-editing Processes. Translation studies. Kent State University Press, Kent, Ohio and London. find
  36. Isabel Lacruz, Michael Denkowski, and Alon Lavie. 2014. Cognitive demand and cognitive effort in post-editing. In Proceedings of the 11th Conference of the Association for Machine Translation in the Americas, pages 73–84, Vancouver, Canada. Association for Machine Translation in the Americas. find
  37. Isabel Lacruz, Gregory M. Shreve, and Erik Angelone. 2012. Average pause ratio as an indicator of cognitive effort in post-editing: A case study. In Workshop on Post-Editing Technology and Practice, San Diego, California, USA. Association for Machine Translation in the Americas. find
  38. Gondy Leroy, James E Endicott, David Kauchak, Obay Mouradi, and Melissa Just. 2013. User evaluation of the effects of a text simplification algorithm using term familiarity on perception, understanding, learning, and information retention. Journal of medical Internet research, 15(7):e144. find
  39. Gondy Leroy, David Kauchak, Diane Haeger, and Douglas Spegman. 2022. Evaluation of an online text simplification editor using manual and automated metrics for perceived and actual text difficulty. JAMIA Open, 5(2):ooac044. find
  40. John Michael Linacre. 1989. Many-faceted Rasch measurement. Ph.D. thesis, The University of Chicago. find
  41. Christiane Maaß. 2020. Easy Language–Plain Language–Easy Language Plus: Balancing comprehensibility and acceptability. Frank & Timme, Berlin. find
  42. Margot Madina, Itziar Gonzalez-Dios, and Melanie Siegel. 2024. Towards reliable E2R texts: A proposal for standardized evaluation practices. In Computers Helping People with Special Needs, pages 224–231, Cham. Springer Nature Switzerland. find
  43. Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2020. Zero-shot crosslingual sentence simplification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5109–5126, Online. Association for Computational Linguistics. find
  44. Louis Martin, Angela Fan, Éric de la Clergerie, Antoine Bordes, and Benoît Sagot. 2022. MUSS: Multilingual unsupervised sentence simplification by mining paraphrases. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1651–1664, Marseille, France. European Language Resources Association.
  45. Paloma Martínez, Alberto Ramos, and Lourdes Moreno. 2024. Exploring Large Language Models to generate Easy to Read content. Frontiers in Computer Science, Volume 6 - 2024. find
  46. Kai North, Tharindu Ranasinghe, Matthew Shardlow, and Marcos Zampieri. 2025. Deep learning approaches to lexical simplification: A survey. Journal of Intelligent Information Systems, 63(1):111–134. ↗
  47. Jipeng Qiang, Minjiang Huang, Yi Zhu, Yunhao Yuan, Chaowei Zhang, and Kui Yu. 2025. Redefining simplicity: Benchmarking Large Language Models from lexical to document simplification. ↗
  48. Ehud Reiter. 2025. We should evaluate real-world impact. Computational Linguistics, pages 1–13. ↗
  49. Luz Rello, Ricardo Baeza-Yates, Stefan Bott, and Horacio Saggion. 2013a. Simplify or help? Text simplification strategies for people with dyslexia. In Proceedings of the 10th International Cross-Disciplinary Conference on Web Accessibility, W4A ’13, Rio de Janeiro, Brazil. Association for Computing Machinery. find
  50. Luz Rello, Ricardo Baeza-Yates, Laura Dempere-Marco, and Horacio Saggion. 2013b. Frequent words improve readability and short words improve understandability for people with dyslexia. In INTERACT ’13, pages 203–219, Cape Town, South Africa. Springer. find
  51. Luz Rello, Susana Bautista, Ricardo Baeza-Yates, Pablo Gervás, Raquel Hervás, and Horacio Saggion. 2013c. One half or 50%? An eye-tracking study of number representation readability. In IN-TERACT ’13, pages 229–245, Cape Town, South Africa. Springer. find
  52. Michael Ryan, Tarek Naous, and Wei Xu. 2023. Revisiting non-English text simplification: A unified multilingual benchmark. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4898–4927, Toronto, Canada. Association for Computational Linguistics. find
  53. Horacio Saggion, Sanja Štajner, Stefan Bott, Simon Mille, Luz Rello, and Biljana Drndarevic. 2015. Making it Simplext: Implementation and evaluation of a text simplification system for Spanish. ACM Trans. Access. Comput., 6(4). find
  54. Fumiko Samejima. 1997. Graded response model. In Wim J. van der Linden and Ronald K. Hambleton, editors, Handbook of Modern Item Response Theory, pages 85–100. Springer New York, New York, NY. ↗
  55. Andreas Säuberli, Franz Holzknecht, Patrick Haller, Silvana Deilen, Laura Schiffl, Silvia Hansen-Schirra, and Sarah Ebling. 2024. Digital comprehensibility assessment of simplified texts among persons with intellectual disabilities. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, CHI ’24, New York, NY, USA. Association for Computing Machinery. find
  56. Scarton Scarton, Mikel L. Forcada, Miquel Esplà-Gomis, and Lucia Specia. 2019. Estimating post-editing effort: a study on human judgements, task-based and reference-based metrics of MT quality. In Proceedings of the 16th International Conference on Spoken Language Translation, Hong Kong. Association for Computational Linguistics. ↗
  57. Sanja Štajner and Sergiu Nisioi. 2018. A detailed evaluation of neural sequence-to-sequence models for in-domain and cross-domain text simplification. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  58. Marie Stevenson. 2010. Researching reading. In Brian Paltridge and Aek Phakiti, editors, Continuum Companion to Research Methods in Applied Linguistics, pages 174–190. Continuum London, England. find
  59. Regina Stodden. 2021. When the scale is unclear – analysis of the interpretation of rating scales in human evaluation of text simplification. In Proceedings of the First Workshop on Current Trends in Text Simplification (CTTS 2021) co-located with the 37th Conference of the Spanish Society for Natural Language Processing (SE-PLN2021), Online (initially located in Málaga, Spain), September 21st, 2021, volume 2944 of CEUR Workshop Proceedings. CEUR-WS.org. find
  60. Regina Stodden. 2026. Automatic German Text Simplification: Data, Evaluation, and Models, 1 edition, volume 21 of Easy – Plain – Accessible. Frank & Timme. find
  61. Andreas Säuberli, Silvia Hansen-Schirra, Franz Holzknecht, Silke Gutermuth, Silvana Deilen, Laura Schiffl, and Sarah Ebling. 2023. Enabling text comprehensibility assessment for people with intellectual disabilities using a mobile application. Frontiers in Communication, 8. ↗
  62. Susanne Trauzettel-Klosinski, Klaus Dietz, and the IReST Study Group. 2012. Standardized Assessment of Reading Performance: The New International Reading Speed Texts IReST. Investigative Ophthalmology & Visual Science, 53(9):5452– 5461. find
  63. Lucas Nunes Vieira. 2017. Cognitive effort and different task foci in post-editing of machine translation: A think-aloud study. Across Languages and Cultures, 18(1):79–105. ↗
  64. Sanja Štajner. 2021. Automatic text simplification for social good: Progress and challenges. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 2637– 2652, Online. Association for Computational Linguistics. ↗
  65. Raeesa Yousaf, Marina Walther, and Michael Gertz. 2025. Accessible language simplification: Large Language Models for generating Easy German. In KI 2025: Advances in Artificial Intelligence, pages 204–217, Cham. Springer Nature Switzerland. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/carrer-2026-evaluating-llm-based-text-simplification/