Elamur Research⌕
☰
home / papers / 2026 / Quality and Comprehensibility of Interlingual Subtitles Produced by Humans or with Machines

Quality and Comprehensibility of Interlingual Subtitles Produced by Humans or with Machines

Summary

This study asks whether subtitles generated fully automatically can match professionally made ones. The authors took English-language video material and produced German interlingual subtitles with three commercial tools: HappyScribe, CapCut and Amberscript. They compared these outputs with each other and with a human-made German reference.

Quality was judged with the FAR model (Functional Equivalence, Acceptability, Readability), a framework from audiovisual translation research. The automatic metrics BLEU, BLEURT and BERTScore were also computed. Comprehensibility was measured with the Flesch Reading Ease formula and examined qualitatively using the Hamburg Comprehensibility Concept.

The machine-generated subtitles fell short of the human ones on both quality and understandability. They dropped or misrecognised meaning-bearing words, translated too literally, and segmented text badly. They also mishandled named entities and idioms and favoured heavy formal compounds.

Automatic scores reached human-like levels only for short segments. Surface readability indices hid the differences between versions. The work matters because it shows that current tools still need human intervention to produce subtitles fit for broadcast.

Key points

  • Setup: English→German subtitles from HappyScribe, CapCut and Amberscript were evaluated against a human reference (UT-h-de). The analysis was text-centred rather than fully multimodal.
  • Comprehensibility scores: Flesch Reading Ease barely differs across versions: 72 for the human and Amberscript subtitles, 73 for HappyScribe and CapCut. The authors stress that this metric ignores semantic coherence.
  • Typical errors by system:
    • CapCut shows the most literal, word-for-word renderings.
    • Amberscript produces repeated or misaligned segments.
    • All systems struggle with ASR-induced omissions and substitutions, which hurt Functional Equivalence.
  • Compounds: The AI systems choose formal compounds such as Innenrevision or Dienstaufsichtsbehörde, where the human subtitler wrote the simpler interne Ermittlung.
  • Limitations:
    • A single rater applied the FAR annotation, with no interrater check.
    • Only one language pair was studied.
    • The system architectures are opaque.
    • Planned future work covers other language pairs, post-editing and audience reception studies.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{schluter2026quality,
  title = {{Quality and Comprehensibility of Interlingual Subtitles Produced by Humans or with Machines}},
  author = {Schlüter, Lara Shoana and Lapshinova-Koltunski, Ekaterina and Jaki, Sylvia},
  booktitle = {Proceedings of the 26th Annual Conference of the European Association for Machine Translation (Volume 1)},
  year = {2026},
  url = {https://aclanthology.org/2026.eamt-1.53/}
}

References

52 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agnetta, Marco. 2025. On the assessment and acceptance of ai technologies in the avt industry: The associations’ perspective. Lebende Sprachen, 70(1):149–176. find
  2. Al Sawi, Islam and Rania Allam. 2024. Exploring challenges in audiovisual translation: A comparative analysis of human- and ai-generated arabic subtitles in birdman. PLOS ONE, 19(10):1–20, 10. ↗
  3. Ballod, Matthias. 2020. Klar-text in Organisationen: Ein Ratgeber zur Optimierung administrativer Informationen. Springer Fachmedien Wiesbaden, Wiesbaden. find
  4. Bellés-Calvera, Lucı́a and Rocı́o Caro Quintana. 2021. Audiovisual translation through NMT and subtitling in the netflix series ‘cable girls’. In Mitkov, Ruslan, Vilelmini Sosoni, Julie Christine Giguère, Elena Murgolo, and Elizabeth Deysel, editors, Proceedings of the Translation and Interpreting Technology Online Conference, pages 142–148, Held Online, July. INCOMA Ltd. find
  5. Bolaños Garcı́a-Escribano, Alejandro. 2025. Practices, Education and Technology in Audiovisual Translation. Routledge. ↗
  6. Brambilla, Marina, Valentina Crestani, and Fabio Mollica. 2016. Untertitelung: interlinguale, intralinguale und intersemiotische Aspekte. Peter Lang Verlag, Berlin, Deutschland. ↗
  7. Calvo-Ferrer, José Ramón. 2023. Can you tell the difference? A study of human vs machine-translated subtitles. Perspectives, 32(6):1115–1132. find
  8. Christmann, Ursula and Norbert Groeben. 2018. Verständlichkeit: die psychologische perspektive. In Maaß, Christiane and Isabel Rink, editors, Handbuch Barrierefreie Kommunikation, volume 3 of Kommunikation – Partizipation – Inklusion, pages 123–146. Frank & Timme, Berlin. find
  9. Davitti, Elena, Annalisa Sandrelli, Tomasz Korybski, Yuan Zou, Constantin Orasan, and Sabine Braun. 2024. Using asr tools to produce automatic subtitles for tv broadcasting: A cross-linguistic comparative analysis. Journal of Audiovisual Translation, 7(2):1–35, Dec. ↗
  10. Deilen, Silvana. 2022. Optische Gliederung von Komposita in Leichter Sprache. Frank and Timme, Berlin. ↗
  11. Diaz Cintas, Jorge and Aline Remael. 2007. Audiovisual Translation: Subtitling. Kinderhook: St. Jerome Publishing, Manchester. find
  12. Dı́az Cintas, Jorge. 2015. Technological strides in subtitling. In Chan, Sin-wai, editor, The Routledge Encyclopedia of Translation Technology, pages 632– 643. Routledge, London and New York. ↗
  13. Dziuk Lameira, Katharina. 2023. Zur komplexität von texten. von der lesbarkeitsformel zur textlinguistis-chen komplexität. In Schrott, Angela, Johanna Wolf, and Christine Pflüger, editors, Textkomplexität und Textverstehen: Studien zur Verständlichkeit von Texten, pages 69–98. De Gruyter, Berlin, Boston. find
  14. Fernandes, Rafael and Marcos Lopes. 2024. Open-source LLMs vs. NMT systems: Translating spatial language in EN-PT-br subtitles. In Martindale, Marianna, Janice Campbell, Konstantin Savenkov, and Shivali Goel, editors, Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 2: Presentations), pages 152– 153, Chicago, USA, September. Association for Machine Translation in the Americas. find
  15. Hagström, Hanna and Jan Pedersen. 2022. Subtitles in the 2020s: The influence of machine translation. Journal of Audiovisual Translation, 5(1):207–225, Dec. ↗
  16. Hof, Lea. 2025. Maschinelle untertitelung mittels pipeline- und end-to-end-ansatz. ein vergleich von youtube und whisper. In Agnetta, Marco, Astrid Schmidhofer, and Alena Petrova, editors, Bild – Ton – Sprachtransfer: Neue Perspektiven auf Audiovisuelle Translation und Media Accessibility, Tran-sKultur, pages 39–68. Frank & Timme, Berlin. ↗
  17. Jaki, Sylvia, Maren Bolz, and Sophie Röther. 2024. Ki-technologien in der audiovisuellen translation. trans-kom, 17:320–342. find
  18. Jüngst, Heike Elisabeth. 2020. Audiovisuelles Übersetzen. Ein Lehr- und Arbeitsbuch. Narr Francke Attempto, Tübingen, 2. überarbeitete und erweiterte auflage edition. find
  19. Karakanta, Alina, Luisa Bentivogli, Mauro Cettolo, Matteo Negri, and Marco Turchi. 2022. Post-editing in automatic subtitling: A subtitlers’ perspective. In Moniz, Helena and Lieve et al. Macken, editors, Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 261–270, Ghent, Belgium. European Association for Machine Translation. hdl:1854/LU-8756355. find
  20. Kelle, Udo. 2022. Mixed methods. In Baur, Nina and Jörg Blasius, editors, Handbuch Methoden der empirischen Sozialforschung, pages 163–177. Springer VS, Wiesbaden. find
  21. Koponen, Maarit, Umut Sulubacak, Kaisa Vitikainen, and Jörg Tiedemann. 2020. MT for subtitling: User evaluation of post-editing productivity. In Martins, André, Helena Moniz, Sara Fumega, Bruno Martins, Fernando Batista, Luisa Coheur, Carla Parra, Isabel Trancoso, Marco Turchi, Arianna Bisazza, Joss Moorkens, Ana Guerberof, Mary Nurminen, Lena Marg, and Mikel L. Forcada, editors, Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 115–124, Lisboa, Portugal, November. European Association for Machine Translation. find
  22. Korycińska-Wegner, Małgorzata. 2024. Linguistische zugänge. In Künzli, Alexander and Klaus Kaindl, editors, Handbuch Audiovisuelle Translation. Arbeitsmittel für Wissenschaft, Studium, Praxis, pages 61–74. Frank & Timme, Berlin. find
  23. Künzli, Alexander. 2017. Die Untertitelung – von der Produktion zur Rezeption, volume 90 of Arbeiten zur Theorie und Praxis des Übersetzens und Dolmetschens, TRANSÜD. Frank & Timme, Berlin. find
  24. Künzli, Alexander. 2020. From inconspicuousness to flow – the CIA model of subtitle quality. Perspectives, 29(3):326–338. ↗
  25. Künzli, Alexander. 2024a. Normen und qualität in der AVT. In Künzli, Alexander and Klaus Kaindl, editors, Handbuch Audiovisuelle Translation. Arbeitsmittel für Wissenschaft, Studium, Praxis, pages 345–360. Frank & Timme, Berlin. find
  26. Künzli, Alexander. 2024b. Untertitelung. In Künzli, Alexander and Klaus Kaindl, editors, Handbuch Audiovisuelle Translation. Arbeitsmittel für Wissenschaft, Studium, Praxis, pages 107–122. Frank & Timme, Berlin. find
  27. Künzli, Alexander. 2025. Untertiteln im stream-ingzeitalter – überlegungen von untertitel-expertinnen im deutschsprachigen raum zum kom-petenzprofil. Fachsprache. Journal of Professional and Scientific Communication, 47(3–4):129–144. find
  28. Kurch, Alexander, Nathalie Mälzer, and Katja Münch. 2015. Qualitätsstudie zu live-untertitelungen – am beispiel des “tv-duells”. trans-kom, 8(1):144–163. find
  29. Kurch, Alexander. 2018. Produktionsprozesse der hörgeschädigten-untertitelungen und audiodeskription: Potenziale teilautomatisierter prozessbeschle-unigung mittels (sprach-)technologien. In Maaß, Christiane and Isabel Rink, editors, Handbuch Barrierefreie Kommunikation, volume 3 of Kommunikation – Partizipation – Inklusion, pages 437–454. Frank & Timme, Berlin. find
  30. Lapshinova-Koltunski, Ekaterina, Sylvia Jaki, Maren Bolz, and Merle Sauter. 2025. Human- or machine-translated subtitles: Who can tell them apart? In Bouillon, Pierrette, Johanna Gerlach, Sabrina Girletti, Lise Volkart, Raphael Rubino, Rico Sennrich, Ana C. Farinha, Marco Gaido, Joke Daems, Dorothy Kenny, Helena Moniz, and Sara Szoc, editors, Proceedings of Machine Translation Summit XX: Volume 1, pages 496–505, Geneva, Switzerland, June. European Association for Machine Translation. find
  31. Loock, Rudy, Benjamin Holt, and Sophie Léchauguette. 2022. The use of online translators by students not enrolled in a professional translation program: beyond copying and pasting for a professional use. In Macken, Lieve and et al., editors, Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 23–29, Ghent, Belgium, June. European Association for Machine Translation. hdl:1854/LU-8756355. find
  32. Ludera, Ewa, Agnieszka Szarkowska, and David Orrego-Carmona. 2024. Expertise in interlingual subtitling: applying the FAR model to study the quality of subtitles created by professional and trainee subtitlers. The International Journal of Translation and Interpreting Research, 16(1):55–75. ↗
  33. Mälzer, Nathalie and Maria Wünsche. 2018. Untertitelung für Hörgeschädigte (SDH). In Maaß, Christiane and Isabel Rink, editors, Handbuch Barrierefreie Kommunikation, volume 3 of Kommunikation – Partizipation – Inklusion, pages 327–344. Frank & Timme, Berlin. find
  34. Mälzer, Nathalie and Maria Wünsche. 2019. Handlungsempfehlungen für die TV-untertitelung für gehörlose und schwerhörige kinder zwischen acht und zwölf jahren. Technical report, Universität Hildesheim, Institut für Übersetzungswissenschaft & Fachkommunikation, Hildesheim. find
  35. Martı́nez Martı́nez, José Manuel and Mihaela Vela. 2016. SubCo: A learner translation corpus of human and machine subtitles. In Calzolari, Nicoletta, Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 2246–2254, Portorož, Slovenia, May. European Language Resources Association (ELRA). find
  36. Mondello, Ashley, Romina Cini, Sahil Rasane, Alina Karakanta, and Laura Casanellas. 2025. Using AI tools in multimedia localization workflows: a productivity evaluation. In Bouillon, Pierrette, Johanna Gerlach, Sabrina Girletti, Lise Volkart, Raphael Rubino, Rico Sennrich, Samuel Läubli, Martin Volk, Miquel Esplà-Gomis, Vincent Vandeghinste, Helena Moniz, and Sara Szoc, editors, Proceedings of Machine Translation Summit XX: Volume 2, pages 1– 7, Geneva, Switzerland, June. European Association for Machine Translation. find
  37. Nardi, Antonella. 2016. Sprachlich-textuelle faktoren im untertitelungsprozess: Ein modell zur übersetzerausbildung deutsch-italienisch. trans-kom, 9(1):34–57. find
  38. Nikoloudaki, Dionysia. 2024. Rendering patriarchy through gendered translator gaze in Romeo and Juliet. inTRAlinea. Special Issue: Translating Threat. find
  39. Ottmann, Angelika and Carmen Canfora. 2020. Risiken und haftungsfragen bei neuronaler maschineller übersetzung. In Bund der Dolmetscher und Übersetzer (BDÜ), editor, Maschinelle Übersetzung für Übersetzungsprofis, pages 171–184. BDÜ Fachverlag, Bonn. find
  40. Papi, Sara, Marco Gaido, Alina Karakanta, Mauro Cettolo, Matteo Negri, and Marco Turchi. 2023. Direct speech translation for automatic subtitling. Transactions of the Association for Computational Linguistics, 11:1355–1376. ↗
  41. Pedersen, Jan. 2017. The FAR model: assessing quality in interlingual subtitling. The Journal of Specialised Translation, 28:210–229. find
  42. Rei, Ricardo, Nuno Miguel Guerreiro, and Fernando Batista. 2020. Automatic truecasing of video subtitles using BERT: A multilingual adaptable approach. In Lesot, Marie-Jeanne et al., editors, Information Processing and Management of Uncertainty in Knowledge-Based Systems, volume 1237 of Communications in Computer and Information Science, pages 708–721, Cham. Springer. find
  43. Romero-Fresco, Pablo and Franz Pöchhacker. 2017. Quality assessment in interlingual live subtitling: The NTR model. Linguistica Antverpiensia, New Series – Themes in Translation Studies, 16:149–167. ↗
  44. Salesky, Elizabeth, Marcello Federico, and Marine Carpuat, editors. 2024. Proceedings of the 21st International Conference on Spoken Language Translation (IWSLT 2024), Bangkok, Thailand (in-person and online), August. Association for Computational Linguistics. find
  45. Schierl, Frederike. 2023. Reception of machine-translated and human-translated subtitles – a case study. In Yamada, Masaru and Felix do Carmo, editors, Proceedings of Machine Translation Summit XIX, Vol. 2: Users Track, pages 42–53, Macau SAR, China, September. Asia-Pacific Association for Machine Translation. find
  46. Schmalz, Antonia. 2019. Maschinelle übersetzung. In Wittpahl, Volker, editor, Künstliche Intelligenz, pages 194–211. Springer Vieweg, Berlin, Heidelberg. find
  47. Schmitz, Anke. 2016. Verständlichkeit von Sach-texten. Wirkung der globalen Textkohäsion auf das Textverständnis von Schülern. Psychologische Genese der Sprach- und Schreibkompetenz. Springer VS, Wiesbaden. find
  48. Tang, Yunlong, Jing Bi, Siting Xu, Luchuan Song, Susan Liang, Teng Wang, Daoan Zhang, Jie An, Jingyang Lin, Rongyi Zhu, Ali Vosoughi, Chao Huang, Zeliang Zhang, Pinxin Liu, Mingqian Feng, Feng Zheng, Jianguo Zhang, Ping Luo, Jiebo Luo, and Chenliang Xu. 2026. Video understanding with large language models: A survey. IEEE Transactions on Circuits and Systems for Video Technology, 36(2):1355–1376. ↗ our card
  49. Tardel, Anke. 2023. A proposed workflow model for researching production processes in subtitling. trans-kom, 16(1):140–173. find
  50. Vanroy, Bram, Arda Tezcan, and Lieve Macken. 2023. MATEO: MAchine Translation Evaluation Online. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 499–500, Tampere, Finland, June. European Association for Machine Translation. find
  51. Wünsche, Maria. 2022. Untertitel im Kinderfernse-hen: Perspektiven aus Translationswissenschaft und Verständlichkeitsforschung, volume 46 of TransKul-tur. Frank & Timme, Berlin. find
  52. Zhang, Tianyi, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTscore: Evaluating text generation with BERT. arXiv preprint arXiv:1904.09675. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/schluter-2026-quality-comprehensibility-interlingual-subtitles-produced/