Elamur Research⌕
☰
home / papers / 2026 / Assessing Quality of Experience in Natural Language Generation of German Text

Assessing Quality of Experience in Natural Language Generation of German Text

Paper · 2026 · Dinh Nam Pham, Shushen Manakhimova, Vivien Macketanz et al.

Summary

Surface-overlap metrics such as BLEU and ROUGE track human judgments of generated text poorly, and German has received less attention in NLG evaluation than English. The authors borrow Quality of Experience (QoE), a user-perception framework from telecommunications and multimedia, and apply it to German summarization (ATS) and machine translation (MT). They release TextQ-German, a set of six subsets with crowdsourced ratings from German speakers. Instead of fixing quality criteria in advance, they derive the perceptual dimensions for each task from the ratings themselves. They then train three kinds of QoE predictors: fine-tuned transformers, interpretable linguistic-feature regressors, and hybrids of the two. The hybrids beat pure transformers in nearly every setting, and feature-only models come close to fine-tuned language models. The corpus also includes LLM-generated outputs with overall QoE scores, and separately collected validation sets suggest the models generalize to unseen data. The result is an open benchmark and a set of baselines for user-centred evaluation of German NLG.

Key points

  • The six subsets are TextQ-ATS and TextQ-MT (dimension-level ratings), TextQ-ATS-LLM and TextQ-MT-LLM (LLM outputs with overall QoE), and TextQ-ATS-Val and TextQ-MT-Val (held-out validation).
  • Summarization source material comes partly from the GeWiki corpus of the SwissText & KONVENS 2020 summarization challenge.
  • The linguistic features include lexical diversity measures (type token ratio, Yule's K, Maas, Herdan), Flesch reading ease, sentence-length statistics, and German morphosyntactic ratios (dative and genitive nouns to nouns, haben to verb, participle verbs to verbs, passive voice count).
  • Feature subsets were chosen per quality dimension with RFE, SFS, Lasso and Elastic Net. Dimension-level errors in the appendix table are reported as RMSE, and the dimensions visible there include Grammaticality, Transparency and Precision.
  • Data and code are on GitHub (DFKI-NLP/TextQ), Hugging Face and Kaggle. The work was funded by DFG project 436,813,723.

Interesting ideas

  • I1. Quality dimensions are discovered from how users actually rate texts, not imposed from a checklist, which keeps the evaluation axes tied to perceived quality.
  • I2. Interpretable linguistic features are combined with a learned representation, and the features alone already land close to a fine-tuned model.
  • I3. Several feature-selection strategies (RFE, SFS, Lasso, Elastic Net) are compared for each dimension, yielding a short and inspectable list of features per quality axis.
  • I4. Morphosyntactic ratios such as case distribution, auxiliary choice, passive share and tense counts serve as cheap proxies for how a text is perceived.
  • I5. The dataset is extended with LLM outputs and with independently collected validation sets, testing whether predictors survive a shift in the generating systems.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (back): Rating generated or round-tripped text from the compressor this way could reveal which properties readers actually notice, giving generation targets grounded in perception.
  • I2 (learn): A feature-based quality scorer could act as an explainable baseline or negative control beside opaque scorers, showing how much of the signal is recoverable from transparent cues.
  • I3 (learn): The same comparison could prune large diagnostic feature sets used to evaluate parsers or readers down to a few readable signals per failure type.
  • I4 (tokens): Morphology and tagger output in crate en could feed similar ratio profiles to flag unnatural or degenerate text, such as generated output or noisy input, before parsing.
  • I5 (learn): Evaluation could likewise keep a held-out set from a different source or generator, so improvements are checked against distribution shift and not only in-domain fit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany77 works · 29,479 citations🎓Technische Universität BerlinBerlin · Germany22 works · 957 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Text generation · 202 works

Cite

Cite as Dinh Nam Pham, Shushen Manakhimova, Vivien Macketanz and Sebastian Moller. 2026. Assessing Quality of Experience in Natural Language Generation of German Text. arXiv:2608.18888.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2608.18888 · arXiv 2608.18888

@misc{pham2026assessing,
  title = {{Assessing Quality of Experience in Natural Language Generation of German Text}},
  author = {Dinh Nam Pham, Shushen Manakhimova, Vivien Macketanz and Sebastian Moller},
  year = {2026},
  doi = {10.48550/arxiv.2608.18888},
  eprint = {2608.18888},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2608.18888}
}

References

49 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anschütz, M., & Groh, G. (2022, September). TUM social computing at GermEval 2022: Towards the significance of text statistics and neural embeddings in text complexity prediction. In S. Möller, S. Mohtaj, & B. Naderi (Eds.), Proceedings of the germeval 2022 workshop on text complexity assessment of german text (pp. 21–26). Association for Computational Linguistics. https://aclanthology.org/2022.germeval-1.4/ Bansal, S., & Aggarwal, C. (2026). Textstat (Version 0.7.13). https://pypi.org/project/textstat/ ↗
  2. Barrault, L., Bojar, O., Costa-jussà, M. R., Federmann, C., Fishel, M., Graham, Y., Haddow, B., Huck, M., Koehn, P., Malmasi, S., Monz, C., Müller, M., Pal, S., Post, M., & Zampieri, M. (2019, August). Findings of the 2019 conference on machine translation (WMT19). In O. Bojar, R. Chatterjee, C. Federmann, M. Fishel, Y. Graham, B. Haddow, M. Huck, A. J. Yepes, P. Koehn, A. Martins, C. Monz, M. Negri, A. Névéol, M. Neves, M. Post, M. Turchi, & K. Verspoor (Eds.), Proceedings of the fourth conference on machine translation (volume 2: Shared task papers, day 1) (pp. 1–61). Association for Computational Linguistics. https://doi.org/10.18653/v1/W19-5301 ↗
  3. Bayerische Staatsbibliothek. (2025a). Bert-base-german-cased (revision 43cce13). https://doi.org/10.57967/hf/4377 ↗
  4. Bayerische Staatsbibliothek. (2025b). Bert-base-german-uncased (revision b705f0e). https://doi.org/10.57967/hf/4378 ↗
  5. Belz, A., & Reiter, E. (2006, April). Comparing automatic and human evaluation of NLG systems. In D. McCarthy & S. Wintner (Eds.), 11th conference of the European chapter of the association for computational linguistics (pp. 313– 320). Association for Computational Linguistics. https://aclanthology.org/E06-1040/ ↗
  6. Chan, B., Schweter, S., & Möller, T. (2020, December). German‘s next language model. In D. Scott, N. Bel, & C. Zong (Eds.), Proceedings of the 28th international conference on computational linguistics (pp. 6788–6796). International Committee on Computational Linguistics. https://doi.org/10.18653/v1/2020. coling-main.598 ↗
  7. Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. (2019, June). BERT: Pre-training of deep bidirectional transformers for language understanding. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers) (pp. 4171–4186). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1423 ↗
  8. Dohare, S., Karnick, H., & Gupta, V. (2017). Text summarization using abstract meaning representation. arXiv preprint arXiv:1706.01678 . http://arxiv.org/abs/1706.01678 ↗
  9. Fabbri, A. R., Kryściński, W., McCann, B., Xiong, C., Socher, R., & Radev, D. (2021). SummEval: Re-evaluating summarization evaluation (B. Roark & A. Nenkova, Eds.). Transactions of the Association for Computational Linguistics , 9 , 391– 409. https://doi.org/10.1162/tacl a 00373 ↗
  10. Fisher, R. (1925). Statistical methods for research workers . Oliver; Boyd. find
  11. Frefel, D. (2020, May). Summarization corpora of Wikipedia articles. In N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, & S. Piperidis (Eds.), Proceedings of the twelfth language resources and evaluation conference (pp. 6651–6655). European Language Resources Association. https://aclanthology.org/2020.lrec-1.821/ ↗
  12. Frefel, D., Vogel, M., & Märki, F. (2020). 2nd german text summarization challenge. In S. Ebling, D. Tuggener, M. Hürlimann, M. Cieliebak, & M. Volk (Eds.), Proceedings of the 5th swiss text analytics conference and the 16th conference on natural language processing, swisstext/konvens 2020, zurich, switzerland, june 23-25, 2020 [online only] . CEUR-WS.org. https://ceur - ws . org / Vol - 2624/germeval-task3-paper1.pdf Freitag, M., Foster, G., Grangier, D., Ratnakar, V., Tan, Q., & Macherey, W. (2021). Experts, errors, and context: A large-scale study of human evaluation for machine translation (B. Roark & A. Nenkova, Eds.). Transactions of the Association for Computational Linguistics , 9 , 1460–1474. https://doi . org / 10 . 1162/tacl a 00437 ↗
  13. Grubbs, F. E. (1950). Sample Criteria for Testing Outlying Observations. The Annals of Mathematical Statistics , 21 (1), 27–58. https://doi.org/10 . 1214 / aoms / 1177729885 ↗
  14. Gupta, A., Stead, T. S., & Ganti, L. (2024). Determining a meaningful r-squared value in clinical medicine. Academic Medicine & Surgery . ↗
  15. Howcroft, D. M., Belz, A., Clinciu, M.-A., Gkatzia, D., Hasan, S. A., Mahamood, S., Mille, S., van Miltenburg, E., Santhanam, S., & Rieser, V. (2020, December). Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. In B. Davis, Y. Graham, J. Kelleher, & Y. Sripada (Eds.), Proceedings of the 13th international conference on natural language generation (pp. 169–182). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.inlg-1.23 ↗
  16. IBM Corp. (2021). IBM SPSS Statistics for Macintosh, Version 28.0 . Armonk, NY. find
  17. International Telecommunication Union. (2017, November). Vocabulary for performance, quality of service and quality of experience (ITU-T Recommendation No. P.10/G.100). International Telecommunication Union. Geneva, Switzerland. find
  18. International Telecommunication Union. (2023, October). Subjective video quality assessment methods for multimedia applications (ITU-T Recommendation No. P.910). International Telecommunication Union. Geneva, Switzerland. find
  19. Jarque, C. M., & Bera, A. K. (1980). Efficient tests for normality, homoscedasticity and serial independence of regression residuals. Economics Letters , 6 (3), 255– 259. https://doi.org/10.1016/0165-1765(80)90024-5 ↗
  20. Levene, H. (1960, January). Robust tests for equality of variance. Stanford University Press. find
  21. Li, D., Jiang, T., & Jiang, M. (2019). Quality assessment of in-the-wild videos. Proceedings of the 27th ACM International Conference on Multimedia , 2351– 2359. https://doi.org/10.1145/3343031.3351028 ↗
  22. Lin, C.-Y. (2004). ROUGE: A package for automatic evaluation of summaries. Text Summarization Branches Out , 74–81. https://aclanthology.org/W04-1013/ ↗
  23. Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., & Zhu, C. (2023, December). G-eval: NLG evaluation using gpt-4 with better human alignment. In H. Bouamor, J. Pino, & K. Bali (Eds.), Proceedings of the 2023 conference on empirical methods in natural language processing (pp. 2511–2522). Association for Computational Linguistics. https://doi.org/10.18653/v1/2023.emnlp-main.153 ↗
  24. Ma, C., Shi, Z., Lu, Z., Xie, S., Chao, F., & Sui, Y. (2025). A survey on image quality assessment: Insights, analysis, and future outlook. arXiv preprint arXiv:2502.08540 . https://arxiv.org/abs/2502.08540 ↗
  25. Manakhimova, S., Macketanz, V., & Möller, S. (2025, March). Quality of experience of german machine translation and automatic text summarization. In S. Grawunder (Ed.), Studientexte zur sprachkommunikation: Elektronische sprachsignalverarbeitung 2025 (pp. 212–222). TUDpress, Dresden. https://www.essv.de/pdf/2025 212 222.pdf Mihalcea, R., & Tarau, P. (2004, July). TextRank: Bringing order into text. In D. Lin & D. Wu (Eds.), Proceedings of the 2004 conference on empirical methods in natural language processing (pp. 404–411). Association for Computational Linguistics. https://aclanthology.org/W04-3252/ ↗
  26. Mohtaj, S., Naderi, B., & Möller, S. (2022, September). Overview of the GermEval 2022 shared task on text complexity assessment of German text. In S. Möller, S. Mohtaj, & B. Naderi (Eds.), Proceedings of the germeval 2022 workshop on text complexity assessment of german text (pp. 1–9). Association for Computational Linguistics. https://aclanthology.org/2022.germeval-1.1/ ↗
  27. Mohtaj, S., Schmitt, V., Khamsehashari, R., & Möller, S. (2023). Multi-task learning for German text readability assessment. CLiC-it . find
  28. Möller, S., & Raake, A. (Eds.). (2014). Quality of experience . Springer International Publishing. https://doi.org/10.1007/978-3-319-02681-7 ↗
  29. Montani, I., Honnibal, M., Honnibal, M., Boyd, A., Landeghem, S. V., & Peters, H. (2023, October). Explosion/spacy: V3.7.2: Fixes for apis and requirements (Version v3.7.2). Zenodo. https://doi.org/10.5281/zenodo.10009823 ↗
  30. Naderi, B. (2018). Motivation of workers on microtask crowdsourcing platforms (1st). Springer Publishing Company, Incorporated. find
  31. Naderi, B., Mohtaj, S., Ensikat, K., & Möller, S. (2019). Subjective assessment of text complexity: A dataset for german language. arXiv preprint arXiv:1904.07733 . https://arxiv.org/abs/1904.07733 ↗
  32. Naderi, B., Mohtaj, S., Karan, K., & Möller, S. (2019). Automated text readability assessment for german language: A quality of experience approach. 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX) , 1–3. https://doi.org/10.1109/QoMEX.2019.8743194 ↗
  33. Naderi, B., Polzehl, T., Beyer, A., Pilz, T., & Möller, S. (2014). Crowdee: Mobile crowdsourcing micro-task platform for celebrating the diversity of languages. In H. Li, H. M. Meng, B. Ma, E. Chng, & L. Xie (Eds.), 15th annual conference of the international speech communication association, INTERSPEECH 2014, singapore, september 14-18, 2014 (pp. 1496–1497). ISCA. find
  34. Naderi, B., Wechsung, I., & Möller, S. (2015). Effect of being observed on the reliability of responses in crowdsourcing micro-task platforms. 2015 Seventh International Workshop on Quality of Multimedia Experience (QoMEX) , 1–2. https://doi.org/10.1109/QoMEX.2015.7148091 ↗
  35. Novikova, J., Dušek, O., Cercas Curry, A., & Rieser, V. (2017, September). Why we need new evaluation metrics for NLG. In M. Palmer, R. Hwa, & S. Riedel (Eds.), Proceedings of the 2017 conference on empirical methods in natural language processing (pp. 2241–2252). Association for Computational Linguistics. https://doi.org/10.18653/v1/D17-1238 ↗
  36. Osgood, C. E., Suci, G. J., & Tannenbaum, P. H. (1957). The measurement of meaning . Univer. Illinois Press. find
  37. Ozili, P. K. (2022). The acceptable r-square in empirical modelling for social science research. SSRN Electron. J. ↗
  38. Papineni, K., Roukos, S., Ward, T., & Zhu, W.-J. (2002, July). Bleu: A method for automatic evaluation of machine translation. In P. Isabelle, E. Charniak, & D. Lin (Eds.), Proceedings of the 40th annual meeting of the association for computational linguistics (pp. 311–318). Association for Computational Linguistics. https://doi.org/10.3115/1073083.1073135 ↗
  39. Pham, D. N., Macketanz, V., Manakhimova, S., & Möller, S. (2025, September). Modeling quality of experience in German automatic text summarization and machine translation. In C. Wartena & U. Heid (Eds.), Proceedings of the 21st conference on natural language processing (konvens 2025): Workshops (pp. 169–175). HsH Applied Academics. https://aclanthology. org / 2025 . konvens-2.12/ Rei, R., Stewart, C., Farinha, A. C., & Lavie, A. (2020, November). COMET: A neural framework for MT evaluation. In B. Webber, T. Cohn, Y. He, & Y. Liu (Eds.), Proceedings of the 2020 conference on empirical methods in natural language processing (emnlp) (pp. 2685–2702). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.213 ↗
  40. See, A., Liu, P. J., & Manning, C. D. (2017, July). Get to the point: Summarization with pointer-generator networks. In R. Barzilay & M.-Y. Kan (Eds.), Proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 1073–1083). Association for Computational Linguistics. https://doi.org/10.18653/v1/P17-1099 ↗
  41. Seiffe, L., Kallel, F., Möller, S., Naderi, B., & Roller, R. (2022, June). Subjective text complexity assessment for German. In N. Calzolari, F. Béchet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, J. Odijk, & S. Piperidis (Eds.), Proceedings of the thirteenth language resources and evaluation conference (pp. 707–714). European Language Resources Association. https://aclanthology.org/2022.lrec-1.74/ ↗
  42. van der Lee, C., Gatt, A., van Miltenburg, E., Wubben, S., & Krahmer, E. (2019, October). Best practices for the human evaluation of automatically generated text. In K. van Deemter, C. Lin, & H. Takamura (Eds.), Proceedings of the 12th international conference on natural language generation (pp. 355–368). Association for Computational Linguistics. https://doi.org/10.18653/v1/W19-8643 ↗
  43. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, & R. Garnett (Eds.), Advances in neural information processing systems (Vol. 30). Curran Associates, Inc.
  44. Wältermann, M., Raake, A., & Möller, S. (2010). Quality dimensions of narrowband and wideband speech transmission. Acta Acustica united with Acustica , 96 (6), 1090–1103. ↗
  45. Wang, S., Yu, W., Chen, X., Tian, X., Zhang, J., Lu, L., Tsao, Y., Yamagishi, J., Wang, Y., & Zhang, C. (2025, July). QualiSpeech: A speech quality assessment dataset with natural language reasoning and descriptions. In W. Che, J. Nabende, E. Shutova, & M. T. Pilehvar (Eds.), Proceedings of the 63rd annual meeting of the association for computational linguistics (volume 1: Long papers) (pp. 23588–23609). Association for Computational Linguistics. https://doi.org/10.18653/v1/2025.acl-long.1150 ↗
  46. Welch, B. L. (1947). The generalization of ‘student’s’ problem when several different population variances are involved. Biometrika , 34 (1/2), 28–35. Retrieved May 4, 2026, from http://www.jstor.org/stable/2332510 ↗
  47. Willmott, C. J., & Matsuura, K. (2005). Advantages of the Mean Absolute Error (MAE) over the Root Mean Square Error (RMSE) in assessing average model performance. Climate Research , 30 , 79–82. ↗
  48. Yang, B., Wang, L., Wong, D. F., Chao, L. S., & Tu, Z. (2019, June). Convolutional self-attention networks. In J. Burstein, C. Doran, & T. Solorio (Eds.), Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers) (pp. 4040–4045). Association for Computational Linguistics. https://doi.org/10.18653/v1/N19-1407 ↗
  49. Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., & Artzi, Y. (2020). Bertscore: Evaluating text generation with BERT. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . https://openreview.net/forum?id=SkeHuCVFDr ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pham-2026-assessing-quality-experience-natural-language/