Elamur Research⌕
☰
home / papers / 2024 / What Do Self-Supervised Speech Models Know About Words?

What Do Self-Supervised Speech Models Know About Words?

Paper · 2024 · Ankita Pasad, Chung-Ming Chien, Shane Settle et al.

Summary

Self-supervised speech models (S3Ms) do well on many speech tasks, but it is still unclear what they learn about units larger than phones. This study looks at how word-level information is represented across the layers of ten S3Ms. The models differ in training objective, input modality (speech-only or visually grounded) and size.

Trained probing classifiers can blur interpretation, so the authors avoid them. Instead, they use projection-weighted CCA to relate word-segment representations to four kinds of linguistic vectors:

  • word identity,
  • pronunciation, via acoustically grounded word embeddings,
  • part-of-speech features,
  • semantic attributes.

They also run three training-free evaluations: acoustic word discrimination, word segmentation and spoken sentence similarity.

The main findings are:

  • Frames inside a word segment are not equally useful, and identity cues cluster near the segment's middle.
  • Objective and size determine which layers hold linguistic content, usually intermediate ones.
  • Visually grounded models beat the speech-only models they were initialized from on several tasks.
  • Simple methods on frozen features beat earlier benchmark results.

These insights can guide decisions about layer selection, distillation and adaptation for downstream tasks.

Key points

  • The CCA targets are one-hot word IDs (500 dims), AGWEs trained on LibriSpeech (128), POS attributes derived from PTB (45) and semantic attributes from SemCor (41).
  • FaST-VGS+ and VG-HuBERT outperform wav2vec2-Base and HuBERT-Base on discrimination, segmentation and semantic similarity.
  • HuBERT and WavLM consistently rank above wav2vec2. Model rankings change between the LibriSpeech and Buckeye evaluation domains, while layer-wise trends stay stable.
  • A parameter-free segmentation algorithm beats more complex unsupervised approaches. A single-layer RNN trained on little labeled data reaches near-perfect word discrimination.
  • The paper appeared in TACL vol. 12 (2024), pp. 372–391. Code is available at github.com/ankitapasad/layerwise-analysis.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Toyota Technological Institute at ChicagoChicago · United States33 works · 13,261 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSignal Processing · 452 works

Cite

Cite as Ankita Pasad, Chung-Ming Chien, Shane Settle and Karen Livescu. 2024. What Do Self-Supervised Speech Models Know About Words?. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00656

@article{pasad2024what,
  title = {{What Do Self-Supervised Speech Models Know About Words?}},
  author = {Ankita Pasad, Chung-Ming Chien, Shane Settle and Karen Livescu},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00656},
  url = {https://doi.org/10.1162/tacl_a_00656}
}

References

100 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Badr M. Abdullah, Mohammed Maqsood Shaik, Bernd Möbius, and Dietrich Klakow. 2023. An information-theoretic analysis of self-supervised discrete representations of speech. In Interspeech. https://doi.org/10.21437/Interspeech.2023-2131 — [doi:10.21437/interspeech.2023-2131](https://doi.org/10.21437/interspeech.2023-2131) ↗
  2. Yonatan Belinkov. 2022. Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics. https://doi.org/10.1162/coli_a_00422 ↗
  3. Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. 2018. LRS3-TED: A large-scale dataset for visual speech recognition. arXiv preprint arXiv:1809.00496. find
  4. Yonatan Belinkov and James Glass. 2019. Analysis methods in neural language processing: A survey. Transactions of the Association for Computational Linguistics (TACL). https://doi.org/10.1162/tacl a 00254 ↗
  5. Robin Algayres, Tristan Ricoul, Julien Karadayi, Hugo Laurençon, Salah Zaiem, Abdelrahman Mohamed, Benoı̂t Sagot, and Emmanuel Dupoux. 2022. DP-Parse: Finding word boundaries from raw speech with an instance lexicon. Transactions of the Association for Computational Linguistics (TACL). https://doi… find
  6. Saurabhchand Bhati, Jesús Villalba, Piotr Żelasko, Laureano Moro-Velazquez, and Najim Dehak. 2021. Segmental contrastive predictive coding for unsupervised word segmentation. In Interspeech. https://doi.org/10.21437/Interspeech.2021-1874 ↗
  7. Robin Algayres, Mohamed Zaiem, Benoı̂t Sagot, and Emmanuel Dupoux. 2020. Evaluating the reliability of acoustic speech embeddings. In Interspeech. https://doi.org/10.21437/Interspeech.2020-2362 ↗
  8. Michael A. Carlin, Samuel Thomas, Aren Jansen, and Hynek Hermansky. 2011. Rapid evaluation of speech representations for spoken term discovery. In Interspeech. https://doi.org/10.21437/Interspeech.2011-304 — [doi:10.21437/interspeech.2011-304](https://doi.org/10.21437/interspeech.2011-304) ↗
  9. Heng-Jui Chang, Shu-wen Yang, and Hung-yi Lee. 2022. DistilHuBERT: Speech representation learning by layer-wise distillation of hidden-unit BERT. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://doi.org/10 .1109/ICASSP43922.2022.9747490 of deep bidirectio… find
  10. Ewan Dunbar, Julien Karadayi, Mathieu Bernard, Xuan-Nga Cao, Robin Algayres, Lucas Ondel, Laurent Besacier, Sakriani Sakti, and Emmanuel Dupoux. 2020. The zero resource speech challenge 2020: Discovering discrete subword and word units. In Interspeech. https://doi.org/10.21437/Interspeech.2020-2743 ↗
  11. Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin, Sanjeev Khundanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Yujun Wang, Zhao You, and Zhiyong Yan. 2021. GigaSpeech: An… find
  12. Zhiyun Fan, Meng Li, Shiyu Zhou, and Bo Xu. 2021. Exploring wav2vec 2.0 on speaker verification and language identification. In Interspeech. find
  13. Manaal Faruqui and Chris Dyer. 2014. Community evaluation and exchange of word vectors at wordvectors. org. In Association for Computational Linguistics (ACL): System Demonstrations. https://doi.org/10.3115/v1/P14-5004 ↗
  14. Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2022. WavLM: Large-scale self-supervised pre-training for full stack… find
  15. Manaal Faruqui, Yulia Tsvetkov, Pushpendre Rastogi, and Chris Dyer. 2016. Problems with evaluation of word embeddings using word similarity tasks. In 1st Workshop on Evaluating Vector-Space Representations for NLP. https://doi.org/10.18653/v1/W16-2506 ↗
  16. Hyeong-Seok Choi, Juheon Lee, Wansoo Kim, Jie Lee, Hoon Heo, and Kyogu Lee. 2021. Neural analysis and synthesis: Reconstructing speech from self-supervised representations. In Advances in Neural Information Processing Systems (NeurIPS). find
  17. Chi-Luen Feng, Po-chun Hsu, and Hung-yi Lee. 2022. Silence is sweeter than speech: Self-supervised model using silence to store speaker information. arXiv preprint arXiv:2205.03759. — [arXiv:2205.03759](https://arxiv.org/abs/2205.03759) find
  18. Tzeviya Sylvia Fuchs and Yedid Hoshen. 2023. Unsupervised word segmentation using temporal gradient pseudo-labels. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). find
  19. Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations. In International Conference on Language Resources and Evaluation (LREC).
  20. Santiago Cuervo, Maciej Grabias, Jan Chorowski, Grzegorz Ciesielski, Adrian Łańcucki, Paweł Rychlikowski, and Ricard Marxer. 2022. Contrastive prediction strategies for unsupervised segmentation and categorization of phonemes and words. In International Conference on Acoustics, Speech and Signal Pr… find
  21. John J. Godfrey, Edward C. Holliman, and Jane McDaniel. 1992. Switchboard: Telephone speech corpus for research and development. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). https://doi.org/10.1109/ICASSP.1992.225858 ↗
  22. Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. find
  23. BERT: Pre-training Zhang, Yonghui Wu, and Ruoming Pang. 2020. Conformer: Convolution-augmented transformer for speech recognition. In Interspeech. https://doi.org/10.21437/Interspeech.2020-3015 ↗
  24. Aren Jansen, Samuel Thomas, and Hynek Hermansky. 2013. Weak top-down constraints for unsupervised acoustic model training. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). https://doi.org/10.1109/ICASSP.2013.6639241 ↗
  25. David Harwath and James Glass. 2017. Learning word-like units from joint audio-visual analysis. In Association for Computational Linguistics (ACL). https://doi.org/10 .18653/v1/P17-1047 find
  26. Hang Ji, Tanvina Patel, and Odette Scharenborg. 2022. Predicting within and across language phoneme recognition performance of self-supervised learning speech pre-trained models. arXiv preprint arXiv:2206.12489. — [arXiv:2206.12489](https://arxiv.org/abs/2206.12489) find
  27. Wanjia He, Weiran Wang, and Karen Livescu. 2017. Multi-view recurrent neural acoustic word embeddings. In International Conference on Learning Representations (ICLR). find
  28. Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu, Pierre-Emmanuel Mazaré, Julien Karadayi, Vitaliy Liptchinsky, Ronan Collobert, Christian Fuegen, Tatiana Likhomanenko, Gabriel Synnaeve, Armand Joulin, Abdelrahman Mohamed, and Emmanuel Dupoux. 2020. Libri-Light: A benchmark… find
  29. John Hewitt and Percy Liang. 2019. Designing and interpreting probes with control tasks. In Empirical Methods in Natural Language Processing (EMNLP). https://doi.org/10 .18653/v1/D19-1275 find
  30. Harold Hotelling. 1936. Relations between two sets of variates. Biometrika. https://doi.org/10.1093/biomet/28.3-4.321 — [doi:10.1093/biomet/28.3-4.321](https://doi.org/10.1093/biomet/28.3-4.321) ↗
  31. Wei-Ning Hsu, David Harwath, Christopher Song, and James Glass. 2021a. Text-free image-to-speech synthesis using learned segmental units. In Association for Computational Linguistics (ACL). find
  32. Herman Kamper. 2022. Word segmentation on discovered phone units with dynamic programming and self-supervised scoring. IEEE/ACM Transactions on Audio, Speech, and Language Processing (TASLP). https://doi.org/10.1109/TASLP.2022.3229264 — [doi:10.1109/taslp.2022.3229264](https://doi.org/10.1109/taslp.2022.3229264) ↗
  33. Wei-Ning Hsu, Anuroop Sriram, Alexei Baevski, Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Jacob Kahn, Ann Lee, Ronan Collobert, Gabriel Synnaeve, and Michael Auli. 2021b. Robust wav2vec 2.0: Analyzing domain shift in self-supervised pre-training. In Interspeech. find
  34. Herman Kamper, Weiran Wang, and Karen Livescu. 2016. Deep convolutional acoustic word embeddings using word-pair side information. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). https://doi.org/10.1109/ICASSP.2016.7472619 ↗
  35. Wei-Ning Hsu, Yao-Hung Hubert Tsai, Benjamin Bolte, Ruslan Salakhutdinov, and Abdelrahman Mohamed. 2021c. HuBERT: How much can a bad teacher benefit asr pre-training? In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). find
  36. Eesung Kim, Jae-Jin Jeon, Hyeji Seo, and Hoon Kim. 2022. Automatic pronunciation assessment using self-supervised speech representation learning. In Interspeech. https://doi.org/10.21437/Interspeech.2022-10245 — [doi:10.21437/interspeech.2022-10245](https://doi.org/10.21437/interspeech.2022-10245) ↗
  37. Yushi Hu, Shane Settle, and Karen Livescu. 2020. Multilingual jointly trained acoustic and written word embeddings. In Interspeech. https://doi.org/10.21437/Interspeech.2020-2828 ↗
  38. Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. 2019. Similarity of neural network representations revisited. In International Conference on Machine Learning (ICML).
  39. Dongseong Hwang, Khe Chai Sim, Zhouyuan Huo, and Trevor Strohman. 2022. Pseudo label is better than human label. In Interspeech. https://doi.org/10.21437/Interspeech.2022-11034 ↗
  40. Felix Kreuk, Joseph Keshet, and Yossi Adi. 2020. Self-supervised contrastive learning for unsupervised phoneme segmentation. In Interspeech. https://doi.org/10.21437/Interspeech.2020-2398 large annotated corpus of English: The Penn treebank. Computational Linguistics. https://doi.org/10.21236/ADA2… — [doi:10.21236/ada273556](https://doi.org/10.21236/ada273556) ↗
  41. Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and Emmanuel Dupoux. 2021. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics (T…
  42. Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Interspeech. https://doi.org/10 .21437/Interspeech.2017-1386 find
  43. Danny Merkx, Stefan L. Frank, and Mirjam Ernestus. 2021. Semantic sentence similarity: Size does not always matter. In Interspeech. https://doi.org/10.21437/Interspeech.2021-1464 ↗
  44. Keith Levin, Katharine Henry, Aren Jansen, and Karen Livescu. 2013. Fixed-dimensional acoustic embeddings of variable-length segments in low-resource settings. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). https://doi.org/10.1109/ASRU.2013.6707765 — [doi:10.1109/asru.2013.6707765](https://doi.org/10.1109/asru.2013.6707765) ↗
  45. Danny Merkx, Sebastiaan Scholten, Stefan L. Frank, Mirjam Ernestus, and OdetteScharenborg. 2023. Modelling human word learning and recognition using visually grounded speech. Cognitive Computation. https://doi.org/10.1007/s12559-022-10059-7 — [doi:10.1007/s12559-022-10059-7](https://doi.org/10.1007/s12559-022-10059-7) ↗
  46. Yuanchao Li, Yumnah Mohamied, Peter Bell, and Catherine Lai. 2023a. Exploration of a self-supervised speech model: A study on emotional corpora. In IEEE Spoken Language Technology Workshop (SLT). find
  47. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Human Language Technology. https://doi.org/10.3115/1075671 .1075742 ↗
  48. Zhengyang Li, Thomas Graave, Jing Liu, Timo Lohrenz, Siegfried Kunzmann, and Tim Fingscheidt. 2023b. Parameter-efficient cross-language transfer learning for a language-modular audiovisual speech recognition. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). find
  49. Oli Liu, Hao Tang, and Sharon Goldwater. 2023. Self-supervised predictive coding models encode speaker and phonetic information in orthogonal subspaces. In Interspeech. https://doi.org/10.21437/Interspeech.2023 -871 — [doi:10.21437/interspeech.2023](https://doi.org/10.21437/interspeech.2023) ↗
  50. Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, and Yoshua Bengio. 2019. Speech model pre-training for end-to-end spoken language understanding. In Interspeech. https://doi.org/10.21437/Interspeech.2019-2396 ↗
  51. Abdelrahman Mohamed, Hung-yi Lee, Lasse Borgholt, Jakob D. Havtorn, Joakim Edin, Christian Igel, Katrin Kirchhoff, Shang-Wen Li, Karen Livescu, Lars Maaløe, Tara N. Sainath, and Shinji Watanabe. 2022. Self-supervised speech representation learning: A review. IEEE Journal of Selected Topics in Signal… find
  52. Ari Morcos, Maithra Raghu, and Samy Bengio. 2018. Insights on representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems (NeurIPS). find
  53. Tu Anh Nguyen, Maureen De Seyssel, Robin Algayres, Patricia Rozé, Ewan Dunbar, and Emmanuel Dupoux. 2022. Are word boundaries useful for unsupervised language learning? arXiv preprint arXiv:2210.02956. — [arXiv:2210.02956](https://arxiv.org/abs/2210.02956) find
  54. Danni Ma, Neville Ryant, and Mark Liberman. 2021. Probing acoustic representations for phonetic properties. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). https://doi.org/10 .1109/ICASSP39728.2021.9414776 find
  55. Tu Anh Nguyen, Maureen de Seyssel, Patricia Rozé, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, and Emmanuel Dupoux. 2020. find
  56. The Zero Resource Speech Benchmark 2021: Metrics and baselines for Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a unsupervised spoken language modeling. In NeurIPS Workshop on Self-Supervised Learning for Speech and Audio Processing. language modeling. In AAAI Work… find
  57. Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mohamed, and Emmanuel Dupoux. 2023. Generative spoken dialogue language modeling. Transactions of the Association for Computational Linguistics (TACL). https…
  58. Puyuan Peng and David Harwath. 2022b. Word discovery in visually grounded, self-supervised speech models. In Interspeech. find
  59. Puyuan Peng, Herman Kamper, and Karen Livescu. 2020. A correspondence variational autoencoder for unsupervised acoustic word embeddings. In NeurIPS Workshop on Self-Supervised Learning for Speech and Audio Processing. find
  60. Shruti Palaskar, Vikas Raunak, and Florian Metze. 2019. Learned in speech recognition: Contextual acoustic word embeddings. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). https://doi.org/10.1109/ICASSP.2019 .8683868 — [doi:10.1109/icassp.2019](https://doi.org/10.1109/icassp.2019) ↗
  61. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EN-MLP). https://doi.org/10.3115/v1/D14-1162 ↗
  62. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. LibriSpeech: An ASR corpus based on public domain audio books. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). https://doi.org/10.1109/ICASSP.2015.7178964 — [doi:10.1109/icassp.2015.7178964](https://doi.org/10.1109/icassp.2015.7178964) ↗
  63. Mark A. Pitt, Keith Johnson, Elizabeth Hume, Scott Kiesling, and William Raymond. 2005. The Buckeye corpus of conversational speech: Labeling conventions and a test of transcriber reliability. Speech Communication. https://doi.org/10.1016/j .specom.2004.09.001 — [doi:10.1016/j](https://doi.org/10.1016/j) find
  64. Ankita Pasad, Ju-Chieh Chou, and Karen Livescu. 2021. Layer-wise analysis of a self-supervised speech representation model. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). https://doi.org/10 .1109/ASRU51503.2021.9688093 find
  65. Archiki Prasad and Preethi Jyothi. 2020. How accents confound: Probing for accent information in end-to-end speech recognition systems. In Association for Computational Linguistics (ACL). https://doi.org/10.18653/v1/2020.acl-main.345 ↗
  66. Ankita Pasad, Bowen Shi, and Karen Livescu. 2023. Comparative layer-wise analysis of self-supervised speech models. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). https://doi.org/10.1109/ICASSP49357.2023 .10096149 — [doi:10.1109/icassp49357.2023](https://doi.org/10.1109/icassp49357.2023) ↗
  67. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. 2017. SVCCA: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In Neural Information Processing Systems (NIPS). find
  68. Okko Räsänen, Unto K. Laine, and Toomas Altosaar. 2011. Blind segmentation of speech using non-linear filtering methods. Speech Technologies. https://doi.org/10.5772/16433 — [doi:10.5772/16433](https://doi.org/10.5772/16433) ↗
  69. Ankita Pasad, Felix Wu, Suwon Shon, Karen Livescu, and Kyu J. Han. 2022. On the use of external data for spoken named entity recognition. In North American Chapter of the Association for Computational Linguistics (NAACL). https://doi.org/10.18653/v1/2022.naacl-main.53 ↗
  70. Abhilasha Ravichander, Yonatan Belinkov, and Eduard Hovy. 2021. Probing the probing paradigm: Does probing accuracy entail task relevance? In European Chapter of the Association for Computational Linguistics (EACL). https://doi.org/10 .18653/v1/2021.eacl-main.295
  71. Puyuan Peng and David Harwath. 2022a. Self-supervised representation learning for speech using visual grounding and masked Ramon Sanabria, Hao Tang, and Sharon Goldwater. 2021. On the difficulty of segmenting words with attention. In Second Workshop on Insights from Negative Results in NLP. https://… find
  72. Bowen Shi, Shane Settle, and Karen Livescu. 2021. Whole-word segmental speech recognition with acoustic word embeddings. In IEEE Spoken Language Technology Workshop (SLT). https://doi.org/10.1109/SLT48900.2021.9383578 ↗
  73. Ramon Sanabria, Hao Tang, and Sharon Goldwater. 2023. Analyzing acoustic word embeddings from pre-trained self-supervised speech models. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). https://doi.org/10 .1109/ICASSP49357.2023.10096099 find
  74. Suwon Shon, Siddhant Arora, Chyi-Jiunn Lin, Ankita Pasad, Felix Wu, Roshan Sharma, Wei-Lun Wu, Hung-Yi Lee, Karen Livescu, and Shinji Watanabe. 2023. SLUE Phase-2: A benchmark suite of diverse spoken language understanding tasks. In Association for Computational Linguistics (ACL). https://doi.org/10… find
  75. Naomi Saphra and Adam Lopez. 2019. Understanding learning dynamics of language models with SVCCA. In North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT).
  76. Shane Settle, Kartik Audhkhasi, Karen Livescu, and Michael Picheny. 2019. Acoustically grounded word embeddings for improved acoustics-to-word speech recognition. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). https://doi.org/10.1109/ICASSP.2019 .8682903 — [doi:10.1109/icassp.2019](https://doi.org/10.1109/icassp.2019) ↗
  77. Shane Settle and Karen Livescu. 2016. Discriminative acoustic word embeddings: Recurrent neural network-based approaches. In IEEE Spoken Language Technology Workshop (SLT). https://doi.org/10.1109/SLT.2016.7846310 ↗
  78. Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu, and Kyu J. Han. 2022. SLUE: New benchmark tasks for spoken language understanding evaluation on natural speech. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). https://doi.org/10 .1109… find
  79. Louis ten Bosch and Bert Cranen. 2007. A computational model for unsupervised word discovery. In Interspeech. https://doi.org/10 .21437/Interspeech.2007-429 find
  80. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. BERT rediscovers the classical NLP pipeline. In Association for Computational Linguistics (ACL). https://doi.org/10 .18653/v1/P19-1452
  81. Jui Shah, Yaman Kumar Singla, Changyou Chen, and Rajiv Ratn Shah. 2021. What all do audio transformer models hear? Probing acoustic representations for language delivery and its structure. In IEEE International Conference on Data Mining Workshops (ICDMW). find
  82. Gaofei Shen, Afra Alishahi, Arianna Bisazza, and Grzegorz Chrupała. 2023. Wave to syntax: Probing spoken language models for syntax. In Interspeech. https://doi.org/10 .21437/Interspeech.2023-679 find
  83. Hsiang-Sheng Tsai, Heng-Jui Chang, Wen-Chin Huang, Zili Huang, Kushal Lakhotia, Shu-wen Yang, Shuyan Dong, Andy T. Liu, Cheng-I Jeff Lai, Jiatong Shi, Xuankai Chang, Phil Hall, Hsuan-Jui Chen, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-yi Lee. 2022. SUPERB-SG: Enhanced speech proce… find
  84. Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, and Abdelrahman Mohamed. 2022. Learning audio-visual speech representation by masked multimodal cluster prediction. In International Conference on Learning Representations (ICLR). find
  85. Yulia Tsvetkov, Manaal Faruqui, and Chris Dyer. 2016. Correlation-based intrinsic evaluation of word vector representations. In 1st Workshop on Evaluating Vector-Space Representations for NLP. https://doi.org/10.18653/v1/W16-2520 ↗
  86. Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Guillaume Lample, and Chris Dyer. 2015. Evaluation of word vector representations by subspace alignment. In Empirical Methods in Natural Language Processing (EMNLP). https://doi.org/10.18653/v1/D15-1243 ↗
  87. Benjamin van Niekerk, Leanne Nortje, Matthew Baas, and Herman Kamper. 2021. Analyzing speaker information in self-supervised models to improve zero-resource speech processing. In Interspeech. https://doi.org/10.21437/Interspeech.2021-1182 — [doi:10.21437/interspeech.2021-1182](https://doi.org/10.21437/interspeech.2021-1182) ↗
  88. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Association for Computational L…
  89. Felix Wu, Kwangyoun Kim, Shinji Watanabe, Kyu J. Han, Ryan McDonald, Kilian Q. Weinberger, and Yoav Artzi. 2023. Wav2seq: Pre-training speech-to-text encoder-decoder models using pseudo languages. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). https://doi.org/… find
  90. Lisa Van Staden and Herman Kamper. 2021. A comparison of self-supervised speech representations as input features for unsupervised acoustic word embeddings. In IEEE Spoken Language Technology Workshop (SLT). https://doi.org/10.1109/SLT48900.2021.9383625 ↗
  91. Shuo Xie, Jiahao Qiu, Ankita Pasad, Li Du, Qing Qu, and Hongyuan Mei. 2022. Hidden state variability of pretrained language models can guide computation reduction for transfer learning. In Findings of Empirical Methods in Natural Language Processing (EMNLP). https://doi.org/10.18653 /v1/2022.finding… find
  92. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems (NIPS).
  93. Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern… — SciPy 1.0 — Fundamental Algorithms for Scientific Computing in Python
  94. Gene-Ping Yang, Yue Gu, Qingming Tang, Dongsu Du, and Yuzong Liu. 2023a. On-device constrained self-supervised speech representation learning for keyword spotting via knowledge distillation. In Interspeech. find
  95. Mu Yang, Ram CMC Shekar, Okim Kang, and John HL Hansen. 202b. What can an accent identifier learn? Probing phonetic and prosodic information in a wav2vec2-based accent identification model. In Interspeech. find
  96. Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, Tzu-Hsien Huang, Wei-Cheng Tseng, Ko-tik Lee, Da-Rong Liu, Zili Huang, Shuyan Dong, Shang-Wen Li, Shinji Watanabe, Abdelrahman Mohamed, and Hung-yi Lee.… find
  97. Elena Voita, Rico Sennrich, and Ivan Titov. 2019. The bottom-up evolution of representations in the transformer: A study with machine translation and language modeling objectives. In North American Chapter of the Association for Computational Linguistics (NAACL). https://doi.org/10.18653/v1/D19-144… ↗
  98. Salah Zaiem, Robin Algayres, Titouan Parcollet, Slim Essid, and Mirco Ravanelli. 2023a. Fine-tuning strategies for faster inference using speech self-supervised models: A comparative study. In IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). https://doi.org/10.1109… — [arXiv:2308.14456](https://arxiv.org/abs/2308.14456) find
  99. Jian Zhu, Zuoyu Tian, Yadong Liu, Cong Zhang, and Chia-wen Lo. 2022. Bootstrapping meaning through listening: Unsupervised learning of spoken sentence embeddings. In Findings of Empirical Methods in Natural Language Processing (EMNLP). https://doi.org/10 .18653/v1/2022.findings-emnlp.81 find
  100. Salah Zaiem, Youcef Kemiche, Titouan Parcollet, Slim Essid, and Mirco Ravanelli. 2023b. Speech self-supervised representations benchmarking: A case for larger probing heads. arXiv preprint find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pasad-2024-what-do-self-supervised-speech/