Elamur Research⌕
☰
home / papers / 2026 / A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation

A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation

Summary

The paper addresses the lack of a reproducible way to evaluate simultaneous speech-to-speech translation (SimulS2ST) on long, unsegmented speech. Earlier work mostly used short or pre-segmented audio, and the earlier pBAL approach was never released and was built for cascade systems. The authors propose a pipeline that needs only source speech, sentence-split source transcripts and reference translations.

The pipeline works in three steps:

  • It transcribes the system's output speech in chunks with Qwen3-ASR-1.7B and gets token end times from Qwen3-ForcedAligner-0.6B.
  • It splits the transcript into sentences with spaCy and aligns them to source sentences using SEGALE, an embedding-based aligner built on Vecalign that supports many-to-many and null matches.
  • It computes YAAL-style latency and xCOMET quality for each aligned group and averages them into system scores.

The authors tested SeamlessStreaming, Seed LiveInterpret 2.0 and Hibiki-Zero on ACL 60/60 and Audio-NTREX-4L. The results show that delay keeps growing over talks that last several minutes. This points to a limitation of current systems that short-clip benchmarks hide. The code is open-sourced.

Key points

  • Scoring rules: Speech is transcribed in 180-second chunks, and timestamps are shifted back onto the global timeline. Groups showing over- or under-translation are left out of latency and receive the metric's minimum quality score (0 for COMET, −25 for MetricX).
  • Datasets:
    • ACL 60/60 dev: five English talks of about 10 minutes each, translated into German, Japanese and Chinese.
    • Audio-NTREX-4L test: clips of about 45 seconds, from French, German, Portuguese and Spanish into English.
  • System results: Seed LiveInterpret 2.0 gives the best xCOMET-XL scores but runs more than 2 s slower than the other systems on Audio-NTREX-4L. On En→Ja it reaches about 9.4 s latency with an xCOMET-XL of only 45.48. Both Qwen3-ASR and WhisperX show high CER (~27%) on its Japanese output, which points to poor speech synthesis rather than ASR errors.
  • Segmentation: On ACL 60/60 En→Zh, SEGALE segments with 90.9% accuracy, compared with 79.1% for SoftSegmenter.
  • Latency accumulation: It is linked to target speech running longer than the source. For En→Ja, end offsets exceed 200 s late in a talk. For En→Zh, where target speech is shorter, Seed LiveInterpret 2.0 stays stable.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.iwslt-1.3

@inproceedings{xue2026practical,
  title = {{A Practical Evaluation Method for Long-Form Simultaneous Speech-to-Speech Translation}},
  author = {Xue, Yulin and Ouyang, Siqi and Li, Lei},
  booktitle = {Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.iwslt-1.3},
  url = {https://doi.org/10.18653/v1/2026.iwslt-1.3}
}

References

23 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. WhisperX: Time-Accurate Speech Transcription of Long-Form Audio. In Interspeech 2023, pages 4489–4493. find
  2. Shanbo Cheng, Yu Bao, Zhichao Huang, Yu Lu, Ningxin Peng, Lu Xu, Runsheng Yu, Rong Cao, Yujiao Du, Ting Han, Yuxiang Hu, Zeyang Li, Sitong Liu, Shengtao Ma, Shiguang Pan, Jiongchen Xiao, Nuo Xu, Meng Yang, Rong Ye, and 9 others. 2025. Seed liveinterpret 2.0: End-to-end simultaneous speech-to-speech translation with your voice. Preprint, arXiv:2507.17527. ↗
  3. Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Coria Meglioli, David Dale, Ning Dong, Mark Duppenthaler, Paul-Ambroise Duquenne, Brian Ellis, Hady Elsahar, Justin Haaheim, John Hoffman, Min-Jae Hwang, Hirofumi Inaguma, Christopher Klaiber, Ilia Kulikov, Pengwei Li, Daniel Licht, Jean Maillard, and 46 others. 2023. Seamless: Multilingual expressive and streaming speech translation. Preprint, arXiv:2312.05187. ↗ our card
  4. Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. 2024. Moshi: a speech-text foundation model for real-time dialogue. Preprint, arXiv:2410.00037. ↗ our card
  5. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xCOMET: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995. ↗ our card
  6. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 492–504, Miami, Florida, USA. Association for Computational Linguistics. find our card
  7. Tom Labiausse, Romain Fabre, Yannick Estève, Alexandre Défossez, and Neil Zeghidour. 2026. Simultaneous speech-to-speech translation without aligned data. Preprint, arXiv:2602.11072. ↗
  8. Xutai Ma, Hongyu Gong, Danni Liu, Ann Lee, Yun Tang, Peng-Jen Chen, Wei-Ning Hsu, Phillip Koehn, and Juan Pino. 2022. Direct simultaneous speech-to-speech translation with variational monotonic multi-head attention. Preprint, arXiv:2110.08250. ↗
  9. Xutai Ma, Anna Sun, Siqi Ouyang, Hirofumi Inaguma, and Paden Tomasello. 2023. Efficient monotonic multihead attention. Preprint, arXiv:2312.04515. ↗
  10. Evgeny Matusov, Gregor Leusch, Oliver Bender, and Hermann Ney. 2005. Evaluating machine translation output with automatic sentence segmentation. In Proceedings of the Second International Workshop on Spoken Language Translation, Pittsburgh, Pennsylvania, USA. find our card
  11. Sara Papi, Marco Gaido, Matteo Negri, and Luisa Bentivogli. 2024. StreamAtt: Direct streaming speech-to-text translation with attention-based audio history selection. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3692–3707, Bangkok, Thailand. Association for Computational Linguistics. find
  12. Peter Polák, Sara Papi, Luisa Bentivogli, and Ondřej Bojar. 2026. Better late than never: Meta-evaluation of latency metrics for simultaneous speech-to-text translation. Preprint, arXiv:2509.17349. ↗
  13. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  14. Elizabeth Salesky, Kareem Darwish, Mohamed Al-Badrashiny, Mona Diab, and Jan Niehues. 2023. Evaluating multilingual speech translation under realistic conditions with resegmentation and terminology. In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 62–78, Toronto, Canada (in-person and online). Association for Computational Linguistics. find
  15. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300. ↗ our card
  16. Tom Labiausse, Laurent Mazaré, Edouard Grave, Alexandre Défossez, and Neil Zeghidour. 2025. High-fidelity simultaneous speech-to-speech translation. In Forty-second International Conference on Machine Learning. ↗
  17. Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, and Junyang Lin. 2026. Qwen3-asr technical report. Preprint, arXiv:2601.21337. ↗
  18. Danni Liu, Changhan Wang, Hongyu Gong, Xutai Ma, Yun Tang, and Juan Pino. 2022. From Start to Finish: Latency Reduction Strategies for Incremental Speech Synthesis in Simultaneous Speech-to-Speech Translation. In Interspeech 2022, pages 1771–1775. find
  19. Katsuhito Sudoh, Takatomo Kano, Sashi Novitasari, Tomoya Yanagita, Sakriani Sakti, and Satoshi Nakamura. 2020. Simultaneous speech-to-speech translation system with neural incremental asr, mt, and tts. Preprint, arXiv:2011.04845. ↗
  20. Brian Thompson and Philipp Koehn. 2020. Exploiting sentence order in document alignment. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5997–6007, Online. Association for Computational Linguistics. find
  21. Kuang-Da Wang, Shuoyang Ding, Chao-Han Huck Yang, Ping-Chun Hsieh, Wen-Chih Peng, Vitaly Lavrukhin, and Boris Ginsburg. 2025. Extending automatic machine translation evaluation to book-length documents. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 32323–32339, Suzhou, China. Association for Computational Linguistics. find
  22. Shaolei Zhang, Qingkai Fang, Shoutao Guo, Zhengrui Ma, Min Zhang, and Yang Feng. 2024. Stream-Speech: Simultaneous speech-to-speech translation with multi-task learning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8964– 8986, Bangkok, Thailand. Association for Computational Linguistics. find
  23. Renjie Zheng, Mingbo Ma, Baigong Zheng, Kaibo Liu, Jiahong Yuan, Kenneth Church, and Liang Huang. 2020. Fluent and low-latency simultaneous speech-to-speech translation with self-adaptive training. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3928–3937, Online. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/xue-2026-practical-evaluation-method-long-form/