Elamur Research⌕
☰
home / papers / 2023 / BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing

BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing

Paper · 2023 · Chen Wang, Minpeng Liao, Zhongqiang Huang et al.

Summary

Text-trained large language models struggle to take speech as input. Cascaded systems that feed ASR output into an LLM keep the two components separate, and end-to-end speech LLMs depend on speech instruction data, which is rare. The authors first show that training an adapter only to reproduce transcripts makes the LLM ignore instructions and just transcribe. BLSP instead treats each ASR pair as two parallel inputs. In the first step, a frozen instruction-tuned LLM writes a continuation of the textual transcript. In the second step, a small adapter bridging the two modalities, placed after the frozen speech encoder and feeding into the LLM, is trained so that the LLM, given the audio, produces that same continuation. Only the adapter is learned, and only ordinary ASR corpora are required. The resulting model handles ASR, speech translation, spoken language understanding and multi-turn spoken dialogue. It performs competitively with cascades and even works zero-shot on non-English speech, although a noticeable quality gap remains. The work offers a cheap recipe for adding speech input to LLMs without collecting speech instructions.

Key points

  • Training data: The adapter is trained on LibriSpeech, GigaSpeech and Common Voice 2.0, using 8.8 million (speech, continuation) pairs. The continuations come from Llama-2 fine-tuned on Alpaca-52k (3 epochs, batch size 128).
  • Adapter design: Three 1-D convolution layers (kernel 5, stride 2) shorten the speech sequence 8×, followed by a bottleneck layer with hidden size 512. It is trained for one epoch with batch size 768.
  • BLSP+RP variant: This variant mixes repetition data (speech mapped to its transcript) with continuation data at a 1:9 ratio, which helps fine-grained lexical modeling.
  • Motivating analysis: A t-SNE analysis on LibriSpeech shows that an ASR-trained adapter maps speech to nearly identical representations regardless of the task instruction (CW, SA, SR, ST).
  • Limitations: Alignment is mostly semantic, so phonetic detail is lost. Paralinguistic cues are not modeled, and replies stay in English even for Mandarin input. Code is available at github.com/cwang621/blsp.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China168 works · 15,386 citations🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🔬Institute of AutomationBeijing · China70 works · 14,251 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSpeech & spoken language · 480 works

Cite

Cite as Chen Wang, Minpeng Liao, Zhongqiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong and Jiajun Zhang. 2023. BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing. arXiv:2309.00916.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2309.00916 · arXiv 2309.00916

@misc{wang2023blsp,
  title = {{BLSP: Bootstrapping Language-Speech Pre-training via Behavior Alignment of Continuation Writing}},
  author = {Chen Wang, Minpeng Liao, Zhongqiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong and Jiajun Zhang},
  year = {2023},
  doi = {10.48550/arxiv.2309.00916},
  eprint = {2309.00916},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2309.00916}
}

References

24 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock… find
  2. Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. 2023. Llama-adapter v2: Parameter-efficient visual instruction model. arXiv preprint arXiv:2304.15010. find
  3. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. find
  4. Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. 2023. Listen, think, and understand. arXiv preprint arXiv:2305.10790. — [arXiv:2305.10790](https://arxiv.org/abs/2305.10790) find
  5. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Belgium, Brussels. Association for Computational Linguistics.
  6. François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Esteve. 2018. Ted-lium 3: Twice as much data and corpus repartition for experiments on speaker adaptation. In Speech and Computer: 20th International Conference, SPECOM 2018, Leipzig, Germany, September 18–22, 2018, P… find
  7. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PM…
  8. Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790–2799. PMLR. find
  9. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. arxiv. arXiv preprint arXiv:2212.04356.
  10. Alaa Saade, Joseph Dureau, David Leroy, Francesco Caltagirone, Alice Coucke, Adrien Ball, Clément Doumouro, Thibaut Lavril, Alexandre Caulier, Théodore Bluche, et al. 2019. Spoken language understanding on the edge. In 2019 Fifth Workshop on Energy Efficient Machine Learning and Cognitive Computing-… find
  11. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  12. Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, et al. 2023. Audiogpt: Understanding and generating speech, music, sound, and talking head. arXiv preprint arXiv:2304.12995. — [arXiv:2304.12995](https://arxiv.org/abs/2304.12995) find
  13. Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303.17580.
  14. Siddique Latif, Moazzam Shoukat, Fahad Shamshad, Muhammad Usama, Heriberto Cuayáhuitl, and Björn W Schuller. 2023. Sparks of large audio models: A survey and outlook. arXiv preprint arXiv:2308.12792. — [arXiv:2308.12792](https://arxiv.org/abs/2308.12792) find
  15. Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu, and Kyu J Han. 2022. Slue: New benchmark tasks for spoken language understanding evaluation on natural speech. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 792… find
  16. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597.
  17. Yu Shu, Siwei Dong, Guangyao Chen, Wenhao Huang, Ruihua Zhang, Daochen Shi, Qiqi Xiang, and Yemin Shi. 2023. Llasm: Large language and speech model. find
  18. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. arXiv preprint arXiv:2304.08485.
  19. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. ↗
  20. Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. 2019. Speech model pre-training for end-to-end spoken language understanding. arXiv preprint arXiv:1904.03670. find
  21. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  22. Gen Luo, Yiyi Zhou, Tianhe Ren, Shengxin Chen, Xiaoshuai Sun, and Rongrong Ji. 2023. Cheap and quick: Efficient vision-language instruction tuning for large language models. arXiv preprint arXiv:2305.15023. — [arXiv:2305.15023](https://arxiv.org/abs/2305.15023) find
  23. Changhan Wang, Anne Wu, and Juan Pino. 2020. Covost 2 and massively multilingual speech-to-text translation. arXiv preprint arXiv:2007.10310. find
  24. R OpenAI. 2023. Gpt-4 technical report. arXiv, pages 2303–08774.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2023-blsp-bootstrapping-language-speech-pre/