CUHKSZ Simultaneous Speech Translation System for IWSLT 2026
Summary
This is the CUHKSZ entry to the IWSLT 2026 Simultaneous Speech Translation task for English→Chinese and English→German, covering both the main and Extra Context tracks. Systems must translate long unsegmented audio and are scored with the computation-aware LongYAAL latency metric. Instead of chaining a speech encoder, projector and text decoder, the authors use a single natively multimodal model, Qwen3-Omni-30B-A3B. They train only LoRA adapters on its LLM ("Thinker") and keep the audio encoder frozen. Speech translation data is scarce, so they turn English ASR corpora (LibriSpeech, Common Voice, VoxPopuli) plus CoVoST 2 into pseudo-parallel streaming data. In one call, an LLM annotator generates syntax-aware chunks, aligned partial translations and <wait> decisions, which teaches the model when to listen and when to output. A lightweight Python agent on vLLM supplies fixed audio chunks, trims the dialogue history and filters emissions. On the dev set, the system reaches 40.5 / 27.7 BLEU under 2 s and 42.1 / 30.5 BLEU under 4 s. This shows that one multimodal LLM can learn a streaming policy without hand-written heuristics.
Key points
- Dev-set results (computation-aware latency):
- Low regime, En→Zh: 40.5 BLEU / 73.5 XCOMET-XL at 1.95 s.
- Low regime, En→De: 27.7 BLEU / 85.0 XCOMET-XL at 1.72 s.
- High regime, En→Zh: 42.1 BLEU / 75.7 XCOMET-XL at 2.16 s.
- High regime, En→De: 30.5 BLEU / 86.6 XCOMET-XL at 2.29 s.
- Training data uses only the IWSLT "Constrained with LLMs" allow-list. About 220K chunk-level examples remained after quality filtering. The paper names Qwen3-30B-Instruct as the target synthesizer in the abstract but Qwen3-32B as the annotator in the method section.
- A single LLM annotation call replaces the earlier spaCy + Whisper + SimAlign pipeline for chunking, alignment, reordering and <wait> labels.
- Chunk size (
chunk_sec) is the only latency control: 0.92 s / 1.60 s for low latency and 1.28 s / 2.56 s for high latency (En→Zh / En→De). Inference runs on a single NVIDIA A800 GPU. - Extra Context results:
- Injecting PDF-derived named entities or abstracts into the prompt raised En→Zh XCOMET by up to +0.71.
- On En→De, quality dropped slightly, but latency was lower.
Cite
@inproceedings{yang2026cuhksz,
title = {{CUHKSZ Simultaneous Speech Translation System for IWSLT 2026}},
author = {Yang, Zeyu and Nakamura, Satoshi},
booktitle = {Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026)},
year = {2026},
doi = {10.18653/v1/2026.iwslt-1.13},
url = {https://doi.org/10.18653/v1/2026.iwslt-1.13}
}References
- David Ifeoluwa Adelani, Victor Agostinelli, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Sebastien Bratières, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Lizhong Chen, Marcello Federico, Marco Gaido, Mahendra Gupta, HyoJung Han, Ali Hatami, David Javorský, Yejin Jeon, Marek Kaszteln… find
- Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. 2020. Common voice: A massively-multilingual speech corpus. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC), pa… find our card
- Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2023. xCOMET: Transparent machine translation evaluation through fine-grained error detection. arXiv preprint arXiv:2310.10482. find our card
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR). find
- Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning (ICML), pages 28492–28518. find our card
- Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, and 1 others. 2023. SeamlessM4T: Massively multilingual and multimodal machine translation. arXiv preprint arXiv:2308.11596. find our card
- Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021a. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proceedings of the 59th Annual… find our card
- Changhan Wang, Anne Wu, and Juan Pino. 2021b. CoVoST 2 and massively multilingual speech translation. In Proc. find
- Interspeech 2021, pages 2247–2251. find
- Zeyu Yang, Lai Wei, Roman Koshkin, Xi Chen, and Satoshi Nakamura. 2026. SASST: Leveraging syntax-aware chunking and LLMs for simultaneous speech translation. Proceedings of the AAAI Conference on Artificial Intelligence, 40(40):34358–34367. find
- Roman Koshkin, Katsuhito Sudoh, and Satoshi Nakamura. 2024. LLMs are zero-shot context-aware simultaneous translators. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1192–1207, Miami, Florida, USA. Association for Computational Linguistics. find
- Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. LibriSpeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. IEEE. find
- Sara Papi, Matteo Negri, and Marco Turchi. 2023. Attention as a guide for simultaneous speech translation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13340–13356, Toronto, Canada. Association for Computational Linguistics… find
This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-cuhksz-simultaneous-speech-translation-system/