FBK's Long-form SpeechLLMs for IWSLT 2026 Instruction Following
Summary
For the IWSLT 2026 Instruction Following shared task, FBK entered systems in both the short-form and long-form tracks under the constrained setting. The long-form track is harder because extended audio strains the LLM context and tends to produce unstable output. The team built a SpeechLLM from three parts: a SeamlessM4T-v2-large speech encoder, an adapter that compresses and projects speech features, and a Qwen3-4B-Instruct decoder adapted with LoRA. They trained it on a multilingual mix of ASR, ST and SQA data enriched with synthetic translations and question-answer pairs, then extended it to long audio. For long-form inference they compared three ways of segmenting speech. They also proposed HIFS, a metric that discounts each task score by its hallucination rate. The short-form system scored well on MCIF. For long audio, simple 30-second chunks proved the most robust choice. Errors were dominated by repeated hallucinated insertions, which hurt ASR and SSUM most. Most short-form ability survived the long-form extension. The study gives practical guidance on segmentation and hallucination-aware evaluation for long-form SpeechLLMs.
Key points
- Architecture: SeamlessM4T-v2-large Conformer encoder (50 Hz), an inherited adapter that downsamples about 8× to roughly 6.25 Hz, and a linear projection from 1024 to 2560 dimensions into Qwen3-4B-Instruct. The model has 4.73B parameters in total, of which 112.2M are trainable (LoRA rank 8).
- Training data: CoVoST2, EuroParlST, GigaST (M subset) and LibriSQA, totalling about 5.06M samples and 8,436.63 hours. The data was filtered with COMET (thresholds of roughly 0.75–0.85), and 5% "Not answerable" SQA examples were added.
- Short track: SIFS of 2.0708 on MCIF.
- Long track (HIFS by segmentation method):
- fixed 30-second window: 2.0663 (primary system)
- hybrid segmentation: 2.0139
- CRDNN-based VAD: 1.9591
- Observed issues: Hallucinations appeared mainly as repetitions and were only partly fixed by regex post-editing. In the official results, SSUM and ACHAP stayed weak across all segmentation methods.
Cite
@inproceedings{xie2026fbk,
title = {{FBK's Long-form SpeechLLMs for IWSLT 2026 Instruction Following}},
author = {Xie, Zhihang and Gaido, Marco and Papi, Sara and Negri, Matteo and Bentivogli, Luisa},
booktitle = {Proceedings of the 23rd International Conference on Spoken Language Translation (IWSLT 2026)},
year = {2026},
doi = {10.18653/v1/2026.iwslt-1.29},
url = {https://doi.org/10.18653/v1/2026.iwslt-1.29}
}References
- Idris Abdulmumin and 1 others. 2025. Findings of the IWSLT 2025 Evaluation Campaign. In Proc. of IWSLT. find
- David Ifeoluwa Adelani, Victor Agostinelli, Antonios Anastasopoulos, Luisa Bentivogli, Ondřej Bojar, Sebastien Bratières, Marine Carpuat, Roldano Cattoni, Mauro Cettolo, Lizhong Chen, Marcello Federico, Marco Gaido, Mahendra Gupta, HyoJung Han, Ali Hatami, David Javorský, Yejin Jeon, Marek Kaszteln… find
- Loïc Barrault and 1 others. 2023. SeamlessM4T: Massively Multilingual & Multimodal Machine Translation. In Proc. of WMT. find our card
- Xi Chen, Songyang Zhang, Qibing Bai, Kai Chen, and Satoshi Nakamura. 2024. LLaST: Improved End-to-end Speech Translation System Leveraged by Large Language Models. arXiv preprint. find
- Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In Proc. of ACL. find
- Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Ke Li, Junteng Jia, Yuan Shangguan, Jay Mahadeokar, Ozlem Kalinli, Christian Fuegen, and Mike Seltzer. 2024. AudioChatLlama: Towards General-Purpose Speech Abilities for LLMs. In Proc. of NAACL. find
- Jennifer Drexler Fox, Desh Raj, Natalie Delworth, Quinn McNamara, Corey Miller, and Migüel Jetté. 2024. Updated Corpora and Benchmarks for Long-Form Speech Recognition. In Proc. of ICASSP. find
- Marco Gaido, Matteo Negri, Mauro Cettolo, and Marco Turchi. 2021. Beyond Voice Activity Detection: Hybrid Audio Segmentation for Direct Speech Translation. In Proc. of ICNLSP. find
- Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. LoRA: Low-Rank Adaptation of Large Language Models. In Proc. of ICLR. find
- Chao-Wei Huang, Hui Lu, Hongyu Gong, Hirofumi Inaguma, Ilia Kulikov, Ruslan Mavlyutov, and Sravya Popuri. 2024. Investigating Decoder-only Large Language Models for Speech-to-text Translation. In Proc. of Interspeech. find
- Javier Iranzo-Sánchez, Joan Albert Silvestre-Cerdà, Javier Jorge, Nahuel Roselló, Adrià Giménez, Albert Sanchis, Jorge Civera, and Alfons Juan. 2020. Europarl-ST: A Multilingual Corpus for Speech Translation of Parliamentary Debates. In Proc. of Interspeech. find
- Nithin Rao Koluguri, Samuel Kriman, Georgy Zelen-froind, Somshubra Majumdar, Dima Rekesh, Vahid Noroozi, Jagadeesh Balam, and Boris Ginsburg. 2023. Investigating End-to-End ASR Architectures for Long Form Audio Transcription. In Proc. of Interspeech. find
- Sai Koneru, Maike Züfle, Thai-Binh Nguyen, Seyma-nur Akti, Jan Niehues, and Alexander Waibel. 2025. KIT’s Offline Speech Translation and Instruction Following Submission for IWSLT 2025. In Proc. of IWSLT. find
- Tsz Kin Lam, Marco Gaido, Sara Papi, Luisa Bentivogli, and Barry Haddow. 2025. Prepending or Cross-Attention for Speech-to-Text? An Empirical Comparison. In Proc. of NAACL. find
- Beomseok Lee, Marcely Zanon Boito, Laurent Besacier, and Ioan Calapodescu. 2025. NAVER LABS Europe Submission to the Instruction-following Track. In Proc. of IWSLT. find
- Microsoft. 2025. Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs. arXiv preprint. find our card
- Sara Papi, Maike Züfle, Marco Gaido, Beatrice Savoldi, Danni Liu, Ioannis Douros, Luisa Bentivogli, and Jan Niehues. 2026. MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talks. In Proc. of ICLR. find
- Tomasz Potapczyk and Pawel Przybysz. 2020. SRPOL’s System for the IWSLT 2020 End-to-End Speech Translation Task. In Proc. of IWSLT. find
- Mirco Ravanelli, Titouan Parcollet, Peter Plantinga, Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatabad, Abdelwahab Heba, Jianyuan Zhong, Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva, François Grondin, William Aris, Hwidong Na, Yan Gao, and 2 ot… find
- Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task. In Proc. of WMT. find our card
- Fabian Retkowski and Alexander Waibel. 2024. From Text Segmentation to Smart Chaptering: A Novel Benchmark for Structuring Video Transcriptions. In Proc. of ECACL. find
- Changhan Wang, Anne Wu, and Juan Pino. 2020. CoVoST 2 and Massively Multilingual Speech-to-Text Translation. arXiv preprint. find
- Rong Ye, Chengqi Zhao, Tom Ko, Chutong Meng, Tao Wang, Mingxuan Wang, and Jun Cao. 2023. GigaST: A 10,000-hour Pseudo Speech Translation Corpus. arXiv preprint. find
- Zihan Zhao, Yiyang Jiang, Heyang Liu, Yu Wang, and Yanfeng Wang. 2024. LibriSQA: A novel dataset and framework for spoken question answering with large language models. Trans. on Artificial Intelligence. find
- Maike Züfle, Sara Papi, Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, and Jan Niehues. 2025. NUT-SHELL: A Dataset for Abstract Generation from Scientific Talks. In Proc. of IWSLT. find
This page in Markdown → https://md.research.elamur.ai/free/en/papers/xie-2026-fbk-s-long-form-speechllms/