Elamur Research⌕
☰
home / papers / 2026 / ATIR: Towards Audio-Text Interleaved Contextual Retrieval

ATIR: Towards Audio-Text Interleaved Contextual Retrieval

Paper · 2026 · Tong Zhao, Chenghao Zhang, Yutao Zhu et al.

Summary

The paper tackles retrieval where queries and documents mix spoken and written segments in sequence, as in multi-turn chats with voice assistants or recorded lectures. Existing audio-text retrievers handle only single-turn or single-modality inputs. Applying MLLMs directly is costly, and long audio token sequences add noise. The authors define the Audio-Text Interleaved contextual Retrieval (ATIR) task and build a benchmark from LibriSpeech, CoQA, and SVQ, covering four contextual retrieval task types. Their synthesis pipeline uses MLLMs to:

  • create related documents and multi-turn interleaved queries,
  • generate missing audio under four acoustic conditions,
  • mine hard negatives and filter weak samples through self-evaluation.

They also propose ATIR-Qwen-3B, a bi-encoder retriever trained in two stages. Its token selector discards redundant audio tokens so that audio does not crowd out text in the embedding. The model beats text-modal, cross-modal, and fused-modal baselines and has lower latency because it skips the ASR step. This makes it a practical foundation for audio-aware RAG and conversational search.

Key points

  • Benchmark sources are LibriSpeech, CoQA, and SVQ. Audio is synthesized in clean, background-speech, traffic, and media-noise environments.
  • Ablation (average across tasks):
    • without the ATIR Selector: −1.05% Recall@1 and −1.42% nDCG@5;
    • without Stage I training: −3.27% / −3.92%;
    • without Stage II training: −5.86% / −6.75%.
  • The ATIR Selector reaches 78.86% Recall@1 and 85.09 nDCG@5. Average pooling of audio tokens (2/4/8 segments) scores lower, at 76.54–77.21 Recall@1.
  • Text baselines were run on Whisper-Large-V3 transcripts, which have a 2.81% normalized WER. Using the original source text instead improves these baselines but does not close the gap to ATIR-Qwen-3B.
  • Shuffling the order or position of audio turns lowers performance (e.g., IAT→T Recall@1 drops from 81.74 to 79.54). This shows the model depends on the interleaving structure itself.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Renmin University of ChinaBeijing · China139 works · 17,752 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 worksSignal Processing · 493 works

🧭 Topics 2Research subjects and languages the work is about.Information retrieval & search · 552 worksSpeech & spoken language · 519 works

Cite

Cite as Tong Zhao, Chenghao Zhang, Yutao Zhu and Zhicheng Dou. 2026. ATIR: Towards Audio-Text Interleaved Contextual Retrieval. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1006

@inproceedings{zhao2026atir,
  title = {{ATIR: Towards Audio-Text Interleaved Contextual Retrieval}},
  author = {Tong Zhao, Chenghao Zhang, Yutao Zhu and Zhicheng Dou},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1006},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1006}
}

References

53 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in neural information processing systems, 33:12449–12460.
  2. Adhiraj Banerjee and Vipul Arora. 2023. wav2tok: Deep sequence tokenizer for audio retrieval. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. find
  3. Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. 2024a. Mllm-asa-judge: Assessing multimodal llm-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning. find
  4. Haonan Chen, Liang Wang, Nan Yang, Yutao Zhu, Ziliang Zhao, Furu Wei, and Zhicheng Dou. 2025a. mme5: Improving multimodal multilingual embeddings via high-quality synthetic data. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, pages… find
  5. Jianlyu Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024b. M3-embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand an…
  6. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, and 1 others. 2022. Wavlm: Large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 16(6):1505–1518. find
  7. Yifu Chen, Shengpeng Ji, Haoxiao Wang, Ziqing Wang, Siyu Chen, Jinzheng He, Jin Xu, and Zhou Zhao. 2025b. Wavrag: Audio-integrated retrieval augmented generation for spoken dialogue models. arXiv preprint arXiv:2502.14727.
  8. Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, Shi Guang, and Haoqi Fan. 2025a. Emerging properties in unified multimodal pretraining. CoRR, abs/2505.14683. — [arXiv:2505.14683](https://arxiv.org/abs/2505.14683) find
  9. Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song, and 1 others. 2025b. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505.14683. — [arXiv:2505.14683](https://arxiv.org/abs/2505.14683) find
  10. Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023. CLAP learning audio concepts from natural language supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing ICASSP 2023, Rhodes Island, Greece, June 4-10, 2023, pages 1–5. IEEE. find
  11. Benjamin Elizalde, Soham Deshmukh, and Huaming Wang. 2024. Natural language supervision for general-purpose audio representations. In IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2024, Seoul, Republic of Korea, April 14-19, 2024, pages 336–340. IEEE. find
  12. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, Céline Hudelot, and Pierre Colombo. 2025. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenRevie…
  13. Clive Gomes, Hyejin Park, Patrick Kollman, Yi Song, If-fanice Houndayi, and Ankit Shah. 2022. Automated audio captioning and language-based audio retrieval. arXiv preprint arXiv:2207.04156. — [arXiv:2207.04156](https://arxiv.org/abs/2207.04156) find
  14. Tiancheng Gu, Kaicheng Yang, Ziyong Feng, Xingjun Wang, Yanzhao Zhang, Dingkun Long, Yingda Chen, Weidong Cai, and Jiankang Deng. 2025. Breaking the modality barrier: Universal embedding learning with multimodal llms. CoRR, abs/2504.17432. — [arXiv:2504.17432](https://arxiv.org/abs/2504.17432) find
  15. Georg Heigold, Ehsan Variani, Tom Bagby, Cyril Allauzen, Ji Ma, Shankar Kumar, and Michael Riley. Massive sound embedding benchmark (mseb). In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  16. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
  17. A Sophia Koepke, Andreea-Maria Oncescu, João F Henriques, Zeynep Akata, and Samuel Albanie. 2022. Audio retrieval with natural language queries: A benchmark study. IEEE Transactions on Multimedia, 25:2675–2685. find
  18. Zhibin Lan, Liqiang Niu, Fandong Meng, Jie Zhou, and Jinsong Su. 2025. Ume-r1: Exploring reasoning-driven generative multimodal embeddings. arXiv preprint arXiv:2511.00405. — [arXiv:2511.00405](https://arxiv.org/abs/2511.00405) find
  19. Wei Li, Hehe Fan, Yongkang Wong, Yi Yang, and Mohan Kankanhalli. 2024. Improving context understanding in multimodal large language models via multimodal composition learning. In Forty-first International Conference on Machine Learning. find
  20. Yucheng Li, Bo Dong, Frank Guerin, and Chenghua Lin. 2023a. Compressing context to enhance inference efficiency of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 6342–6353. Associatio…
  21. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023b. Towards general text embeddings with multi-stage contrastive learning. CoRR, abs/2308.03281.
  22. Jing Liu, Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang, and Jinhui Tang. 2025. VALOR: vision-audio-language omni-perception pre-training model and dataset. IEEE Trans. Pattern Anal. Mach. Intell., 47(2):708–724. find
  23. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net. find
  24. Christopher D Manning. 2008. Introduction to information retrieval. Syngress Publishing,. find
  25. Annamaria Mesaros, Aleksandr Diment, Benjamin Elizalde, Toni Heittola, Emmanuel Vincent, Bhiksha Raj, and Tuomas Virtanen. 2019. Sound event detection in the dcase 2017 challenge. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 27(6):992–1006. find
  26. Hokuto Munakata, Taichi Nishimura, Shota Nakada, and Tatsuya Komatsu. 2025. Language-based audio moment retrieval. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE. find
  27. Daisuke Niizumi, Daiki Takeuchi, Masahiro Yasuda, Binh Thien Nguyen, Yasunori Ohishi, and Noboru Harada. 2025. M2D-CLAP: exploring general-purpose audio-language representations beyond CLAP. IEEE Access, 13:163313–163330. find
  28. Andreea-Maria Oncescu, A Koepke, Joao F Henriques, Zeynep Akata, and Samuel Albanie. 2021. Audio retrieval with natural language queries. arXiv preprint arXiv:2105.02192. — [arXiv:2105.02192](https://arxiv.org/abs/2105.02192) find
  29. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.
  30. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In Acoustics, Speech and Signal Processing (ICASSP), 2015 find
  31. IEEE International Conference on, pages 5206–5210. IEEE. find
  32. Zhiliang Peng, Jianwei Yu, Wenhui Wang, Yaoyao Chang, Yutao Sun, Li Dong, Yi Zhu, Weijiang Xu, Hangbo Bao, Zehua Wang, Shaohan Huang, Yan Xia, and Furu Wei. 2025. Vibevoice technical report. CoRR, abs/2508.19205. — [arXiv:2508.19205](https://arxiv.org/abs/2508.19205) find
  33. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learn…
  34. Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266.
  35. Changli Tang, Qinfan Xiao, Ke Mei, Tianyi Wang, Fengyun Rao, and Chao Zhang. 2025. WAVE: learning unified & versatile audio-visual embeddings with multimodal LLM. CoRR, abs/2509.21990. — [arXiv:2509.21990](https://arxiv.org/abs/2509.21990) find
  36. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2024. SALMONN: towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenRev…
  37. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual E5 text embeddings: A technical report. CoRR, abs/2402.05672.
  38. Cong Wei, Yang Chen, Haonan Chen, Hexiang Hu, Ge Zhang, Jie Fu, Alan Ritter, and Wenhu Chen. 2024. Uniir: Training and benchmarking universal multimodal information retrievers. In European Conference on Computer Vision, pages 387–404. Springer. find
  39. Yijing Wu, SaiKrishna Rallabandi, Ravisutha Srini-vasamurthy, Parag Pravin Dakle, Alolika Gon, and Preethi Raghavan. 2023. Heysquad: A spoken question answering dataset. CoRR, abs/2304.13689. — [arXiv:2304.13689](https://arxiv.org/abs/2304.13689) find
  40. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff, Defu Lian, and Jian-Yun Nie. 2024. C-pack: Packed resources for general chinese embeddings. In Proceedings of the 47th international ACM SIGIR conference on research and development in information retrieval, pages 641–649.
  41. Jinheng Xie, Zhenheng Yang, and Mike Zheng Shou. 2025. Show-o2: Improved native unified multimodal models. arXiv preprint arXiv:2506.15564. — [arXiv:2506.15564](https://arxiv.org/abs/2506.15564) find
  42. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. 2025a. Qwen2.5-omni technical report. CoRR, abs/2503.20215.
  43. Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, and 1 others. 2025b. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215. — Qwen2.5-Omni Technical Report
  44. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, Yuanjun Lv, Yongqi Wang, Dake Guo, He Wang, Linhan Ma, Pei Zhang, Xinyu Zhang, Hongkun Hao, Zishan Guo, and 19 others. 2025c. Qwen3-omni technical report. CoRR, abs/2509.17765.
  45. Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, and 1 others. 2025d. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765.
  46. Mengyao Xu, Wenfei Zhou, Yauhen Babakhin, Gabriel de Souza Pereira Moreira, Ronay Ak, Radek Osmulski, Bo Liu, Even Oldridge, and Benedikt Schifferer. 2025e. Omni-embed-nemotron: A unified multimodal retrieval model for text, image, audio, and video. CoRR, abs/2510.03458. — [arXiv:2510.03458](https://arxiv.org/abs/2510.03458) find
  47. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 40 others. 2025. Qwen3 technical report. CoRR, abs/2505.09388.
  48. Dongchao Yang, Haohan Guo, Yuanyuan Wang, Rongjie Huang, Xiang Li, Xu Tan, Xixin Wu, and Helen Meng. 2024. Uniaudio 1.5: Large language model-driven audio codec is A few-shot audio task learner. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processi… find
  49. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2024. A survey on multimodal large language models. National Science Review, 11(12):nwae403.
  50. Dong Yu and Lin Deng. 2016. Automatic speech recognition, volume 1. Springer. find
  51. Chenghao Zhang, Guanting Dong, Xinyu Yang, and Zhicheng Dou. 2025a. Towards mixed-modal retrieval for universal retrieval-augmented generation. Preprint, arXiv:2510.17354. — [arXiv:2510.17354](https://arxiv.org/abs/2510.17354) find
  52. Xin Zhang, Xiang Lyu, Zhihao Du, Qian Chen, Dong Zhang, Hangrui Hu, Chaohong Tan, Tianyu Zhao, Yuxuan Wang, Bin Zhang, Heng Lu, Yaqian Zhou, and Xipeng Qiu. 2024. Intrinsicvoice: Empowering llms with intrinsic real-time voice interaction abilities. CoRR, abs/2410.08035. — [arXiv:2410.08035](https://arxiv.org/abs/2410.08035) find
  53. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025b. Qwen3 embedding: Advancing text embedding and reranking through foundation models. CoRR, abs/2506.05176.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2026-atir-audio-text-interleaved-contextual/