Elamur Research⌕
☰
home / papers / 2026 / Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models

Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models

Paper · 2026 · Bajian Xiang, Tingwei Guo, Xuan Chen et al.

Summary

Large Speech Language Models (LSLMs) turn audio into very long token streams, typically 12.5–25 tokens per second. These sequences are much longer than the meaning they carry, which makes inference expensive. The authors test whether every speech token really needs its own representation inside the LLM backbone.

They first run oracle experiments. Word timestamps from the Montreal Forced Aligner define word-aligned units, and audio tokens within each unit are dropped or merged at a single layer of Qwen2-Audio or Kimi-Audio. ASR error on Librispeech-test-clean measures how much content survives. The results show a depth-dependent hierarchy:

  • early layers are fragile and carry acoustic detail;
  • middle layers form an unstable acoustic-to-semantic transition;
  • deep layers tolerate drastic reduction.

Layer-wise cosine-similarity analysis supports this picture. Building on it, the authors propose Affinity Pooling, a training-free method that merges neighbouring tokens whose similarity is high. Its variant, Dual Affinity Pooling (DAP), applies the merging at the input embedding and again at a deep layer. DAP lowers prefill compute with no loss in accuracy, offering inexpensive efficiency gains for deployed speech models.

Key points

  • Models and setup: Qwen2-Audio (32 layers, 25 tokens/s) and Kimi-Audio (28 layers, 12.5 tokens/s). Oracle interventions use Random Drop, Uniform Drop and Uniform Merge, evaluated with WER and a clamped WER (cWER) on Librispeech-test-clean.
  • Deep-layer redundancy: In deep layers (l ≥ 25), keeping only about 25.67% of audio tokens still gives near-baseline performance. Merging consistently beats dropping, and regular sampling beats random sampling.
  • Efficiency of DAP: DAP reduces prefilling FLOPs by 27.48% across three tasks. On long utterances it gives about 1.7× memory saving (dynamic memory falls from 1.99 GB to 1.17 GB) and about 1.1× faster time-to-first-token.
  • Comparison with fixed-rate baselines: At a 60% input-token budget, Affinity Pooling reaches a mean WER of 5.70%. It outperforms both signal-level speedup and linear interpolation of embeddings.
  • Best configuration: The ablations favour a strict adjacency window (ω=1) at the input layer and a wider window (ω=3) at a deep layer (l=29 for Qwen2-Audio).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Bajian Xiang, Tingwei Guo, Xuan Chen and Yang Han. 2026. Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.742

@inproceedings{xiang2026do,
  title = {{Do We Need Distinct Representations for Every Speech Token? Unveiling and Exploiting Redundancy in Large Speech Language Models}},
  author = {Bajian Xiang, Tingwei Guo, Xuan Chen and Yang Han},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.742},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.742}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Swarup Ranjan Behera, Abhishek Dhiman, Karthik Gowda, and Aalekhya Satya Narayani. 2024. Fastast: Accelerating audio spectrogram transformer via token merging and cross-model knowledge distillation. Preprint, arXiv:2406.07676. — [arXiv:2406.07676](https://arxiv.org/abs/2406.07676) find
  2. Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. 2023. Token merging: Your vit but faster. Preprint, arXiv:2210.09461. — [arXiv:2210.09461](https://arxiv.org/abs/2210.09461) find
  3. Fan Bu, Yuhao Zhang, Xidong Wang, Benyou Wang, Qun Liu, and Haizhou Li. 2024. Roadmap towards superhuman speech understanding using large language models. Preprint, arXiv:2410.13268. — [arXiv:2410.13268](https://arxiv.org/abs/2410.13268) find
  4. Yiming Chen, Xianghu Yue, Chen Zhang, Xiaoxue Gao, Robby T. Tan, and Haizhou Li. 2024. Voicebench: Benchmarking llm-based voice assistants. Preprint, arXiv:2410.17196.
  5. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. 2024. Qwen2-audio technical report. Preprint, arXiv:2407.10759.
  6. Wenqian Cui, Dianzhi Yu, Xiaoqi Jiao, Ziqiao Meng, Guangyan Zhang, Qichao Wang, Yiwen Guo, and Irwin King. 2025. Recent advances in speech language models: A survey. Preprint, arXiv:2410.03751.
  7. Tianyu Fu, Tengxuan Liu, Qinghao Han, Guohao Dai, Shengen Yan, Huazhong Yang, Xuefei Ning, and Yu Wang. 2025. Framefusion: Combining similarity and importance for video token reduction on large vision language models. Preprint, arXiv:2501.01986. — [arXiv:2501.01986](https://arxiv.org/abs/2501.01986) find
  8. Chaoqun He, Renjie Luo, Shengding Hu, Yuanqian Zhao, Jie Zhou, Hanghao Wu, Jiajie Zhang, Xu Han, Zhiyuan Liu, and Maosong Sun. 2024. Ultraeval: A lightweight platform for flexible and comprehensive evaluation for llms. Preprint, arXiv:2404.07584. find
  9. Shengpeng Ji, Yifu Chen, Minghui Fang, Jialong Zuo, Jingyu Lu, Hanting Wang, Ziyue Jiang, Long Zhou, Shujie Liu, Xize Cheng, Xiaoda Yang, Zehan Wang, Qian Yang, Jian Li, Yidi Jiang, Jingzhen He, Yunfei Chu, Jin Xu, and Zhou Zhao. 2024. Wavchat: A survey of spoken dialogue models. Preprint, arXiv:241… find
  10. KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, and 21 others. 2025. Kimi-audio technical report. Preprint, arXiv:2504.18425.
  11. Yuri Kuratov, Mikhail Arkhipov, Aydar Bulatov, and Mikhail Burtsev. 2025. Cramming 1568 tokens into a single vector and back again: Exploring the limits of embedding space capacity. Preprint, arXiv:2502.13063. find
  12. Taehan Lee and Hyukjun Lee. 2025. Token Pruning in Audio Transformers: Optimizing Performance and Decoding Patch Importance. IOS Press. find
  13. Tianpeng Li, Jun Liu, Tao Zhang, Yuanbo Fang, Da Pan, Mingrui Wang, Zheng Liang, Zehuan Li, Mingan Lin, Guosheng Dong, Jianhua Xu, Haoze Sun, Zenan Zhou, and Weipeng Chen. 2025. Baichuan-audio: A unified framework for end-to-end speech interaction. Preprint, arXiv:2502.17239.
  14. Yuang Li, Yu Wu, Jinyu Li, and Shujie Liu. 2023. Accelerating transducers through adjacent token merging. Preprint, arXiv:2306.16009. find
  15. Yueqian Lin, Yuzhe Fu, Jingyang Zhang, Yudong Liu, Jianyi Zhang, Jingwei Sun, Hai "Helen" Li, and Yiran Chen. 2025. Speechprune: Context-aware token pruning for speech information retrieval. Preprint, arXiv:2412.12009. — [arXiv:2412.12009](https://arxiv.org/abs/2412.12009) find
  16. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206–5210. find
  17. Kele Shao, Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. 2025a. Holitom: Holistic token merging for fast video large language models. Preprint, arXiv:2505.21334. — [arXiv:2505.21334](https://arxiv.org/abs/2505.21334) find
  18. Kele Shao, Keda Tao, Kejia Zhang, Sicheng Feng, Mu Cai, Yuzhang Shang, Haoxuan You, Can Qin, Yang Sui, and Huan Wang. 2025b. When tokens talk too much: A survey of multimodal long-context token compression across images, videos, and audios. Preprint, arXiv:2507.20198. — [arXiv:2507.20198](https://arxiv.org/abs/2507.20198) find
  19. Zhiyuan Tang, Dong Wang, Yanguang Xu, Jianwei Sun, Xiaoning Lei, Shuaijiang Zhao, Cheng Wen, Xi Tan, Chuandong Xie, Shuran Zhou, Rui Yan, Chenjia Lv, Yang Han, Wei Zou, and Xiangang Li. 2021. Kespeech: An open source speech dataset of mandarin and its eight subdialects. In NeurIPS Datasets and Bench… find
  20. Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. 2025. Dycoke: Dynamic compression of tokens for fast video large language models. Preprint, arXiv:2411.15024. — [arXiv:2411.15024](https://arxiv.org/abs/2411.15024) find
  21. Changhan Wang, Anne Wu, and Juan Pino. 2020. Covost 2 and massively multilingual speech-to-text translation. Preprint, arXiv:2007.10310. find
  22. Hankun Wang, Yiwei Guo, Chongtian Shao, Bohan Li, Xie Chen, and Kai Yu. 2025a. Codecslime: Temporal redundancy compression of neural speech codec via dynamic frame rate. Preprint, arXiv:2506.21074. — [arXiv:2506.21074](https://arxiv.org/abs/2506.21074) find
  23. Hualei Wang, Yiming Li, Shuo Ma, Hong Liu, and Xiangdong Wang. 2025b. Listening between the frames: Bridging temporal gaps in large audio-language models. Preprint, arXiv:2511.11039. — [arXiv:2511.11039](https://arxiv.org/abs/2511.11039) find
  24. Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, and Linfeng Zhang. 2025. Token pruning in multimodal large language models: Are we solving the right problem? Preprint, arXiv:2502.11501. find
  25. Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. 2024. Visionzip: Longer is better but not necessary in vision language models. Preprint, arXiv:2412.04467. — [arXiv:2412.04467](https://arxiv.org/abs/2412.04467) find
  26. Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. 2024. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot. Preprint, arXiv:2412.02612.
  27. Rui-Chen Zheng, Wenrui Liu, Hui-Peng Du, Qinglin Zhang, Chong Deng, Qian Chen, Wen Wang, Yang Ai, and Zhen-Hua Ling. 2025. Say more with less: Variable-frame-rate speech tokenization via adaptive clustering and implicit duration coding. Preprint, arXiv:2509.04685. — [arXiv:2509.04685](https://arxiv.org/abs/2509.04685) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiang-2026-do-we-need-distinct-representations/