Elamur Research⌕
☰
home / papers / 2023 / SONAR: Sentence-Level Multimodal and Language-Agnostic Representations

SONAR: Sentence-Level Multimodal and Language-Agnostic Representations

Paper · 2023 · Paul-Ambroise Duquenne, Holger Schwenk and Benoit Sagot

Summary

SONAR addresses the need for a single fixed-size sentence embedding space that works across many languages and across both text and speech. The authors first train a text encoder-decoder covering 200 languages, initialized from the NLLB 1B dense model. Instead of token-level cross-attention, the decoder sees only one pooled sentence vector. They compare several training signals: translation, auto-encoding, denoising auto-encoding, and an MSE loss that pulls translations together in embedding space. Next, they distill this text space into speech by training language-family speech encoders, initialized from w2v-bert, as students of the text teacher, using only transcription data.

The resulting text encoder clearly beats LASER3 and LaBSE on the xsim and xsim++ similarity-search benchmarks. The decoder delivers text translation close to NLLB 1B despite the single-vector bottleneck. Pairing the speech encoders with the text decoder gives zero-shot speech-to-text translation that rivals Whisper. This matters because one shared space supports bitext mining, cross-modal retrieval and translation into many languages, including low-resource ones. All models are openly released.

Key points

  • Coverage and release: text encoders and decoders cover 200 languages, and speech encoders cover 37 languages. The models are published at github.com/facebookresearch/SONAR and were ported to fairseq2.
  • Effect of training objectives:
    • Translation is the main objective for producing language-agnostic embeddings.
    • Adding an MSE alignment loss improves cross-lingual similarity.
    • Denoising auto-encoding helps decoding, but in excess it weakens language-agnosticity.
  • Text similarity search: a single SONAR text encoder substantially outperforms LASER3 and LaBSE on xsim and xsim++.
  • Speech-to-English translation: on FLEURS (spBLEU, zero-shot, trained on ASR transcriptions only), SONAR averages 25.3 versus 24.5 for Whisper large v2.
    • SONAR is weaker on Mandarin, German and French.
    • It is stronger on Spanish, Dutch, Swahili, Uzbek and Indian languages.
    • Ukrainian is included. The PDF-extracted table appears to give about 30.7 for SONAR versus 32.5 for Whisper, which should be verified against the original.
  • Same-language decoding: decoding speech embeddings into text in the same language acts as zero-shot transcription. SONAR trails Whisper on BLEU because it often paraphrases, but the gap is much smaller on BERTScore.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Institut national de recherche en sciences et technologies du numériqueLe Chesnay-Rocquencourt · France85 works · 25,498 citations🔬ALMANACH: Modélisation et analyse linguistique automatique et humanités computationnellesParis · France20 works · 7,008 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 811 works

Cite

Cite as Paul-Ambroise Duquenne, Holger Schwenk and Benoit Sagot. 2023. SONAR: Sentence-Level Multimodal and Language-Agnostic Representations. arXiv:2308.11466.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.11466 · arXiv 2308.11466

@misc{duquenne2023sonar,
  title = {{SONAR: Sentence-Level Multimodal and Language-Agnostic Representations}},
  author = {Paul-Ambroise Duquenne, Holger Schwenk and Benoit Sagot},
  year = {2023},
  doi = {10.48550/arxiv.2308.11466},
  eprint = {2308.11466},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.11466}
}

References

33 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. 2019. Common voice: A massively-multilingual speech corpus. arXiv preprint arXiv:1912.06670.
  2. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. TACL, pages 597–610.
  3. Arun Babu, Changhan Wang, Andros Tjandra, Kushal Lakhotia, Qiantong Xu, Naman Goyal, Kritika Singh, Patrick von Platen, Yatharth Saraf, Juan Pino, et al. 2021. Xlsr: Self-supervised cross-lingual speech representation learning at scale. arXiv preprint arXiv:2111.09296.
  4. Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. 2020. wav2vec 2.0: A framework for self-supervised learning of speech representations. Advances in Neural Information Processing Systems, 33:12449– 12460.
  5. Ankur Bapna, Colin Cherry, Yu Zhang, Ye Jia, Melvin Johnson, Yong Cheng, Simran Khanuja, Jason Riesa, and Alexis Conneau. 2022. mslam: Massively multilingual joint pre-training for speech and text. arXiv preprint arXiv:2202.01374. find
  6. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
  7. Mingda Chen, Kevin Heffernan, Onur Çelebi, Alex Mourachko, and Holger Schwenk. 2023. xsim++: An improved proxy to bitext mining performance for low-resource languages. arXiv preprint arXiv:2306.12907. find
  8. Yu-An Chung, Yu Zhang, Wei Han, Chung-Cheng Chiu, James Qin, Ruoming Pang, and Yonghui Wu. 2021. W2v-bert: Combining contrastive learning and masked language modeling for self-supervised speech pre-training. In 2021 find
  9. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 244–250. IEEE. find
  10. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. arXiv preprint arXiv:2003.10555. find
  11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116.
  12. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In ACL.
  13. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364.
  14. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2023. Fleurs: Few-shot learning evaluation of universal representations of speech. In 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798–805. IEEE.
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
  16. Mattia A. Di Gangi, Roldano Cattoni, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2019. MuST-C: a Multilingual Speech Translation Corpus. In NAACL, pages 2012–2017.
  17. Paul-Ambroise Duquenne, Hongyu Gong, Ning Dong, Jingfei Du, Ann Lee, Vedanuj Goswani, Changhan Wang, Juan Pino, Benoît Sagot, and Holger Schwenk. 2022a. Speechmatrix: A large-scale mined corpus of multilingual speech-to-speech translations. arXiv preprint arXiv:2211.04508. find
  18. Paul-Ambroise Duquenne, Hongyu Gong, Benoît Sagot, and Holger Schwenk. 2022b. T-modules: Translation modules for zero-shot cross-modal machine translation. arXiv preprint arXiv:2205.12216. find
  19. Paul-Ambroise Duquenne, Hongyu Gong, and Holger Schwenk. 2021. Multimodal and multilingual embeddings for large-scale speech mining. Advances in Neural Information Processing Systems, 34. find
  20. Paul-Ambroise Duquenne, Holger Schwenk, and Benoît Sagot. 2023. Modular speech-to-text translation for zero-shot cross-modal transfer. In Interspeech. find
  21. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic bert sentence embedding. arXiv preprint arXiv:2007.01852.
  22. Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022. Bitext mining using distilled sentence representations for low-resource languages. arXiv preprint arXiv:2205.12654.
  23. Sameer Khurana, Antoine Laurent, and James Glass. 2022. Samu-xlsr: Semantically-aligned multimodal utterance-level cross-lingual speech representation. arXiv preprint arXiv:2205.08180. — [arXiv:2205.08180](https://arxiv.org/abs/2205.08180) find
  24. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. find
  25. Marta R NLLB Team, Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672. find
  26. Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr corpus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206–5210. IEEE. find
  27. Ngoc-Quan Pham, Jan Niehues, Thanh-Le Ha, and Alex Waibel. 2019. Improving zero-shot translation with language-independent constraints. find
  28. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak supervision. Technical report, Technical report, OpenAI, 2022. URL https://cdn.openai. com/papers/whisper. pdf. ↗
  29. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
  30. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In EMNLP, pages 4512–4525.
  31. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021. CCMatrix: Mining billions of high-quality parallel sentences on the web. In ACL, page 6490–6500.
  32. Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. 2021. VoxPopuli: A large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. In Proceedings of the 59th Annual…
  33. Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax. arXiv preprint arXiv:1902.08564. — [arXiv:1902.08564](https://arxiv.org/abs/1902.08564) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/duquenne-2023-sonar-sentence-level-multimodal-language/