Elamur Research⌕
☰
home / papers / 2026 / Test-Time Adaptation of an Offline Multimodal Foundation Model for Simultaneous Speech Translation

Test-Time Adaptation of an Offline Multimodal Foundation Model for Simultaneous Speech Translation

Summary

The paper addresses end-to-end simultaneous speech-to-text translation (SimulST), where systems usually need complex architectures and dedicated training. The authors, competing in the IWSLT 2026 SimulST shared task (speech-to-text, no extra context), build a training-free system instead. Streaming audio is cut into variable-length chunks using Silero VAD with pause detection plus minimum and maximum chunk durations. Translation is done by the off-the-shelf multimodal MoE model Qwen3-Omni, adapted only at inference time. Each audio chunk becomes a user turn in a multi-turn conversation. A deterministic wait-(k, s, n) policy, derived from wait-k, decides when the model reads and writes. Prefilling each response with the most recent translation tokens keeps turns coherent, and key-value caching avoids recomputation. Repetition penalties and detection keep the output clean. In five-fold cross-validation on the official development sets, the system generally beats the cascaded baseline on the quality–latency trade-off. This includes a strong zero-shot result for Czech-to-English. The work matters because it shows that strong SimulST is possible without fine-tuning.

Key points

  • Language directions and metrics: The system covers en→de, en→zh, en→it and cs→en, scored with COMET and LongYAAL (CU) in low- and high-latency regimes.
  • Policy settings: The model first reads k chunks, then reads one chunk per turn (s = 1) and writes at most n units. A unit is a word for English, German and Italian, and a character for Chinese.
  • Tuned hyperparameters: The three tuned settings are the maximum chunk duration d_max, the lag k and the output limit n. The authors stress that these must be coordinated, because too small an n makes the translation fall behind the speech.
  • Example results from Table 5:
    • In the high-latency regime for en→de, COMET rose from 0.8616 to 0.8995, while latency fell from about 3797 ms to 2511 ms.
    • In the low-latency regime for cs→en, COMET rose from 0.4978 to 0.6356.
    • For en→zh in the high-latency regime, latency dropped sharply, but COMET dipped slightly (0.8063 → 0.7670).
  • Limitations: Errors can accumulate across conversation turns. The hyperparameter search was coarse and used small single-domain data. The large model also adds computation-aware latency.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.iwslt-1.27

@inproceedings{xing2026test,
  title = {{Test-Time Adaptation of an Offline Multimodal Foundation Model for Simultaneous Speech Translation}},
  author = {Xing, Yi and Yu, Manli and Liu, Pengfei and Meng, Helen},
  booktitle = {Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.iwslt-1.27},
  url = {https://doi.org/10.18653/v1/2026.iwslt-1.27}
}

References

32 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Ifeoluwa Adelani, Victor Agostinelli, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Sebastien Bratières, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Lizhong Chen, Marcello Federico, Marco Gaido, Mahendra Gupta, HyoJung Han, Ali Hatami, David Javorský, Yejin Jeon, Marek Kasztelni… find
  2. Anthropic. 2026. Prompting best practices: Migrating away from prefilled responses. find
  3. Dom Barnard. 2022. Average Speaking Rate and Words per Minute. find
  4. Keqi Deng, Wenxi Chen, Xie Chen, and Phil Woodland. 2025. SimulS2S-LLM: Unlocking simultaneous inference of speech LLMs for speech-to-speech translation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16718–16734, Vienna, Aus… find
  5. Qian Dong, Yaoming Zhu, Mingxuan Wang, and Lei Li. 2022. Learning When to Translate for Streaming Speech. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 680–694, Dublin, Ireland. Association for Computational Linguistics. find
  6. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995. find our card
  7. Yuka Ko, Ryo Fukuda, Yuta Nishikawa, Yasumasa Kano, Katsuhito Sudoh, and Satoshi Nakamura. 2023. Tagged end-to-end simultaneous speech translation training using simultaneous interpretation data. In Proceedings of the 20th International Conference on Spoken Language Translation (IWSLT 2023), pages 3… find
  8. Roman Koshkin, Katsuhito Sudoh, and Satoshi Nakamura. 2024. TransLLaMa: LLM-based Simultaneous Translation System. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 461–476, Miami, Florida, USA. Association for Computational Linguistics. find
  9. Danni Liu, Gerasimos Spanakis, and Jan Niehues. 2020. Low-Latency Sequence-to-Sequence Speech Recognition and Translation by Partial Hypothesis Selection. In Interspeech 2020, pages 3620–3624. find
  10. Xiaoqian Liu, Guoqiang Hu, Yangfan Du, Erfeng He, YingFeng Luo, Chen Xu, Tong Xiao, and Jingbo Zhu. 2024. Recent advances in end-to-end simultaneous speech translation. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI-24, pages 8142–8150. Internatio… find
  11. Mingbo Ma, Liang Huang, Hao Xiong, Renjie Zheng, Kaibo Liu, Baigong Zheng, Chuanqiang Zhang, Zhongjun He, Hairong Liu, Xing Li, Hua Wu, and Haifeng Wang. 2019. STACL: Simultaneous translation with implicit anticipation and controllable latency using prefix-to-prefix framework. In Proceedings of the… find our card
  12. Xutai Ma, Juan Pino, and Philipp Koehn. 2020. SimulMT to SimulST: Adapting Simultaneous Text Translation to End-to-End Simultaneous Speech Translation. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint C… find
  13. Dominik Macháček and Peter Polák. 2025. Simultaneous translation with offline speech and LLM models in CUNI submission to IWSLT 2025. In Proceedings of the 22nd International Conference on Spoken Language Translation (IWSLT 2025), pages 389–398, Vienna, Austria (in-person and online). Association fo… find
  14. Ha Nguyen, Yannick Estève, and Laurent Besacier. 2021. An Empirical Study of End-To-End Simultaneous Speech Translation Decoding Strategies. In ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7528–7532. find
  15. Jingwei Ni, Zhijing Jin, Markus Freitag, Mrinmaya Sachan, and Bernhard Schölkopf. 2022. Original or translated? a causal analysis of the impact of translationese on machine translation performance. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computation… find
  16. Siqi Ouyang, Xi Xu, and Lei Li. 2025. InfiniSST: Simultaneous translation of unbounded speech with large language model. In Findings of the Association for Computational Linguistics: ACL 2025, pages 3032–3046, Vienna, Austria. Association for Computational Linguistics. find
  17. Sara Papi, Marco Gaido, Matteo Negri, and Marco Turchi. 2022. Does simultaneous speech translation need simultaneous models? In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 141–153, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  18. Sara Papi, Matteo Negri, and Marco Turchi. 2023. Attention as a Guide for Simultaneous Speech Translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13340–13356, Toronto, Canada. Association for Computational Linguistics… find
  19. Sara Papi, Peter Polák, Dominik Macháček, and Ondřej Bojar. 2025. How “real” is your real-time simultaneous speech-to-text translation system? Transactions of the Association for Computational Linguistics, 13:281–313. find
  20. Sara Papi, Maike Züfle, Marco Gaido, Beatrice Savoldi, Danni Liu, Ioannis Douros, Luisa Bentivogli, and Jan Niehues. 2026. MCIF: Multimodal crosslingual instruction-following benchmark from scientific talks. In The Fourteenth International Conference on Learning Representations. find
  21. Peter Polák, Sara Papi, Luisa Bentivogli, and Ondřej Bojar. 2025. Better Late Than Never: Evaluation of Latency Metrics for Simultaneous Speech-to-Text Translation. arXiv preprint. ArXiv:2509.17349 [cs]. find
  22. European Conference on Speech Communication and Technology (Eurospeech 2003), pages 1821–1824. find
  23. Xian Shi, Xiong Wang, Zhifang Guo, Yongqi Wang, Pei Zhang, Xinyu Zhang, Zishan Guo, Hongkun Hao, Yu Xi, Baosong Yang, Jin Xu, Jingren Zhou, and Junyang Lin. 2026. Qwen3-ASR Technical Report. arXiv preprint. ArXiv:2601.21337 [cs]. find
  24. Silero Team. 2024. Silero vad: pre-trained enterprise-grade voice activity detector (vad), number detector and language classifier. https://github.com/ snakers4/silero-vad. ↗
  25. Trusted Translations, Inc. 2026. Translation Quote per Word. find
  26. Ioannis Tsiamas, Gerard I. Gállego, José A. R. Fonollosa, and Marta R. Costa-jussà. 2022. SHAS: Approaching optimal Segmentation for End-to-End Speech Translation. In Interspeech 2022, pages 106–110. find
  27. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025. Qwen3-Omni Technical Report. arXiv preprint. ArXiv:2509.17765… find our card
  28. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 Technical Report. arXiv preprint. ArXiv:2505.09388 [… find our card
  29. Ruiqing Zhang, Zhongjun He, Hua Wu, and Haifeng Wang. 2022. Learning Adaptive Segmentation Policy for End-to-End Simultaneous Translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7862–7874, Dublin, Ireland. Association… find
  30. Shaolei Zhang and Yang Feng. 2023. End-to-End Simultaneous Speech Translation with Differentiable Segmentation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7659–7680, Toronto, Canada. Association for Computational Linguistics. find
  31. Peter Polák, Brian Yan, Shinji Watanabe, Alex Waibel, and Ondřej Bojar. 2023. Incremental Blockwise Beam Search for Simultaneous Speech Translation with Controllable Quality-Latency Tradeoff. In Interspeech 2023, pages 3979–3983. find
  32. Josef Psutka, Pavel Ircing, J.V. Psutka, Vlasta Radova, William J. Byrne, Jan Hajic, Jiri Mirovsky, and Samuel Gustman. 2003. Large vocabulary ASR for spontaneous czech in the MALACH project. In 8th find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/xing-2026-test-time-adaptation-offline-multimodal/