Elamur Research⌕
☰
home / papers / 2026 / Hurdles of Automatic Metric for Speech Translation Evaluation

Hurdles of Automatic Metric for Speech Translation Evaluation

Summary

Speech translation is still mostly scored with text-only metrics, which cannot account for spoken cues such as prosody or hesitations. The authors test whether giving a reference-free quality estimation (QE) metric access to the source audio improves it. They build two audio-aware metrics and evaluate them on the development set of the IWSLT 2026 metrics shared task:

  • Speech LLM: few-shot prompting of Phi-4-multimodal-instruct.
  • COMET+audio: a regression model in the style of CometKiwi that combines InfoXLM text embeddings with Whisper audio embeddings.

Neither approach reliably beats its text-only counterpart. Feeding a model audio that belongs to a different sample hurts little, and sometimes even helps, which suggests both metrics essentially ignore the audio. A manual analysis of 100 samples offers explanations:

  • The content is mostly technical or very short, leaving little paralinguistic signal.
  • Recordings often fail to match their transcripts.

The work matters because it warns that multimodal speech-translation metrics need cleaner, prosody-rich evaluation data. The authors caution that their findings may reflect this particular dataset rather than the task in general.

Key points

  • COMET+audio uses a shared InfoXLM-base text encoder and a Whisper-medium audio encoder, both adapted with LoRA (rank 8, alpha 32). Audio is attention-pooled, and the inputs are fused via two bilinear layers (audio–target and transcript–target) before a regression head.
  • Speech LLM samples dynamic few-shot examples from five score bins and averages five generated scores. Without the few-shot examples, the model gravitated toward extreme scores. Many prompt variants, including UniPrompt optimization, confidence resampling and sibling translations, did not help.
  • Segment-level correlation for Speech LLM was 27.4 with text only versus 18.4 with text plus audio. For COMET+audio, text plus audio and text only were nearly tied (18.3 vs 18.2; MAE 18.20 vs 18.21).
  • Using text with wrong audio produced the best system-level score for Speech LLM (90.1). Translation-only inputs also scored surprisingly well, hinting that the evaluation data favors fluency.
  • In the 100 annotated samples, 76% were technical and 20% information-sharing. About 34% showed audio–transcript mismatches, often because the audio cut off mid-phrase.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.iwslt-1.34

@inproceedings{zarzu2026hurdles,
  title = {{Hurdles of Automatic Metric for Speech Translation Evaluation}},
  author = {Zarzu, Victor Eugen and Zouhar, Vilem},
  booktitle = {Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.iwslt-1.34},
  url = {https://doi.org/10.18653/v1/2026.iwslt-1.34}
}

References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Idris Abdulmumin, Victor Agostinelli, Tanel Alumäe, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Claudia Borg, Fethi Bougares, Roldano Cat toni, Mauro Cettolo, Lizhong Chen, William Chen, Raj Dabre, Yannick Estève, Marcello Federico, Mark Fishel, Marco Gaido, Dávid Javorský, Marek Kasztelnik, Fortuné Kponou, Mateusz Krubiński, Tsz Kin Lam, Danni Liu, Evgeny Matusov, Chan dresh Kumar Maurya, John P. McCrae, Salima Mdhaffar, Yasmin Moslem, Kenton Murray, Satoshi Nakamura, Matteo Negri, Jan Niehues, Atul Kr. Ojha, John E. Ortega, Sara Papi, Pavel Pecina, Peter Polák, Piotr Połeć, Ashwin Sankar, Beat rice Savoldi, Nivedita Sethiya, Claytone Sikasote, Matthias Sperber, Sebastian Stüker, Katsuhito Su doh, Brian Thompson, Marco Turchi, Alex Waibel, Patrick Wilken, Rodolfo Zevallos, Vilém Zouhar, and Maike Züfle. 2025. Findings of the IWSLT 2025 Evaluation Campaign. In Proceedings of the 22nd International Conference on Spoken Language Translation (IWSLT 2025), pages 412–481, Vienna, Austria (in-person and online). find
  2. Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Jun-Kun Chen, Weizhu Chen, Yen-Chun Chen, Yi-ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, Min Gao, Amit Garg, Abhishek Goswami, Junheng Hao, Amr Hendy, Yuxuan Hu, Xin Jin, Mahmoud Khademi, Dongwoo Kim, Young Jin Kim, Gina Lee, Jinyu Li, Yunsheng Li, Chen Liang, Xihui Lin, Zeqi Lin, Mengchen Liu, Yang Liu, Gilsinia Lopez, Chong Luo, Piyush Madan, Vadim Mazalov, Arindam Mitra, Ali Mousavi, Anh Nguyen, Jing Pan, Daniel Perez-Becker, Jacob Platin, Thomas Portet, Kai Qiu, Bo Ren, Liliang Ren, Sambuddha Roy, Ning Shang, Yelong Shen, Saksham Singhal, Sub hojit Som, Xia Song, Tetyana Sych, Praneetha Vad damanu, Shuohang Wang, Yiming Wang, Zhenghao Wang, Haibin Wu, Haoran Xu, Weijian Xu, Yifan Yang, Ziyi Yang, Donghan Yu, Ishmam Zabir, Jian wen Zhang, Li Lyna Zhang, Yunan Zhang, and Xiren Zhou. 2025. Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via David Ifeoluwa Adelani, Victor Agostinelli, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Se bastien Bratières, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Lizhong Chen, Marcello Federico, Marco Gaido, Mahendra Gupta, HyoJung Han, Ali Hatami, David Javorský, Yejin Jeon, Marek Kasztel nik, Antoine Laurent, Danni Liu, Nam Luu, Min Ma, Dominik Macháček, Marie Maltais, Evgeny Ma tusov, Chandresh Kumar Maurya, John P. McCrae, Chutong Meng, Mohammadamini Mohammad, Yas min Moslem, Kenton Murray, Satoshi Nakamura, Matteo Negri, Jan Niehues, Atul Kr. Ojha, John Ortega, Siqi Ouyang, Sara Papi, Peter Polák, Fabian Retkowski, Beatrice Savoldi, Claytone Sika sote, Matthias Sperber, Sebastian Stüker, Katsuhito Sudoh, Marie Tahon, Marco Turchi, Alex Waibel, Patrick Wilken, Rodolfo Zevallos, Vilém Zouhar, and Maike Züfle. 2026. Speech Translation and Met rics in 2026: Findings of the IWSLT Campaign. In Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026), San Diego, California, US. find
  3. Hassan Akbari, Dan Kondratyuk, Yin Cui, Rachel Hornung, Huisheng Wang, and Hartwig Adam. 2023. Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Pro cessing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. find
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam Mc Candlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Pro cessing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual. ↗
  5. Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 3576–3588. ↗ our card
  6. Thierry Etchegoyhen, Haritz Arzelus, Harritxu Gete, Aitor Alvarez, Iván G. Torre, Juan Manuel Martín-Doñas, Ander González-Docasal, and Edson Benites Fernandez. 2022. Cascade or Direct Speech Transla tion? A Case Study. Applied Sciences 12(3). find
  7. HyoJung Han, Kevin Duh, and Marine Carpuat. 2024. SpeechQE: Estimating the Quality of Direct Speech Translation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process ing, pages 21852–21867, Miami, Florida, USA. find
  8. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In The Tenth Inter national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. find
  9. Cong Hua, Qianqian Xu, Shilong Bao, Zhiyong Yang, and Qingming Huang. 2024. ReconBoost: Boosting Can Achieve Modality Reconcilement. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, pages 19573–19597. ↗
  10. Gurusha Juneja, Gautam Jajoo, Hua Li, Jian Jiao, Nagarajan Natarajan, and Amit Sharma. 2025. Task Facet Learning: A Structured Approach To Prompt Optimization. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Aus tria, July 27 - August 1, 2025, pages 23473–23496. find
  11. Tom Kocmi and Christian Federmann. 2023. Large Language Models Are State-of-the-Art Evaluators of Translation Quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, EAMT 2023, Tampere, Fin land, 12-15 June 2023, pages 193–203. ↗
  12. Alon Lavie, Greg Hanneman, Sweta Agrawal, Diptesh Kanojia, Chi-Kiu Lo, Vilém Zouhar, Frederic Blain, Chrysoula Zerva, Eleftherios Avramidis, Sourabh Deoghare, Archchana Sindhujan, Jiayi Wang, David Ifeoluwa Adelani, Brian Thompson, Tom Kocmi, Markus Freitag, and Daniel Deutsch. 2025. Findings of the WMT25 Shared Task on Automated Trans lation Evaluation Systems: Linguistic Diversity is Challenging and References Still Help. In Proceed ings of the Tenth Conference on Machine Transla tion, pages 436–483, Suzhou, China. find
  13. Ilya Loshchilov and Frank Hutter. 2019. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. find
  14. H. Ney. 1999. Speech translation: coupling of recognition and translation. In 1999 IEEE Inter national Conference on Acoustics, Speech, and Signal Processing. Proceedings. ICASSP99 (Cat. No.99CH36258), pages 517-520 vol.1. find
  15. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock man, Christine McLeavey, and Ilya Sutskever. 2023. Robust Speech Recognition via Large-Scale Weak Supervision. In International Conference on Ma chine Learning, ICML 2023, 23-29 July 2023, Hon olulu, Hawaii, USA, pages 28492–28518. find our card
  16. Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. COMET: A Neural Framework for MT Evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process ing, EMNLP 2020, Online, November 16-20, 2020, pages 2685–2702. find our card
  17. Ricardo Rei, Marcos V. Treviso, Nuno Miguel Guer reiro, Chrysoula Zerva, Ana C. Farinha, Christine Maroti, José G. C. Souza, Taisiya Glushkova, Duarte M. Alves, Luísa Coheur, Alon Lavie, and André F. T. Martins. 2022. CometKiwi: IST-Unbabel 2022 Sub mission for the Quality Estimation Shared Task. In Proceedings of the Seventh Conference on Machine Translation, WMT 2022, Abu Dhabi, United Arab Emirates (Hybrid), December 7-8, 2022, pages 634– 645. find
  18. Communication Seamless, Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenzek, Ethan Ye, Bapi Akula, Peng-Jen Chen, Naji El Hachem, Brian Ellis, Gabriel Mejia Gonzalez, Justin Haaheim, Prangthip Hansanti, Russ Howes, Bernie Huang, Min-Jae Hwang, Hirofumi Inaguma, Somya Jain, Elahe Kalbassi, Amanda Kallet, Ilia Kulikov, Janice Lam, Daniel Li, Xutai Ma, Ruslan Mavlyutov, Ben jamin N. Peloquin, Mohamed Ramadan, Abinesh Ramakrishnan, Anna Y. Sun, Kevin Tran, Tuan Tran, Igor Tufanov, Vish Vogeti, Carleigh Wood, Yilin Yang, Bokai Yu, Pierre Andrews, Can Balioglu, Marta R. Costa-jussà, Onur Celebi, Maha Elbayad, Cynthia Gao, Francisco Guzmán, Justine Kao, Ann Lee, Alexandre Mourachko, Juan Pino, Sravya Pop uri, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, Paden Tomasello, Changhan Wang, Jeff Wang, and Skyler Wang. 2023. SeamlessM4T-Mas sively Multilingual & Multimodal Machine Transla tion. CoRR abs/2308.11596. arXiv: 2308.11596. ↗ our card
  19. Matthias Sperber and Matthias Paulik. 2020. Speech Translation and the End-to-End Promise: Taking Stock of Where We Are. In Proceedings of the 58th Annual Meeting of the Association for Compu tational Linguistics, pages 7409–7421, Online. find
  20. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022. Chain-of-Thought Prompt ing Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Pro cessing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/zarzu-2026-hurdles-automatic-metric-speech-translation/