Elamur Research⌕
☰
home / papers / 2022 / Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages

Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages

Summary

Multilingual sentence encoders that try to cover every language in one model have trouble with the long tail of low-resource languages. Training is unbalanced, compute costs are high, and new scripts are hard to add. The authors instead build several smaller encoders, each for one language or a family of related languages, that all map into one shared embedding space. They do this through teacher-student distillation. The teacher, LASER2, is a retrained LASER. The students (LASER3) are 12-layer transformers initialized from scratch and trained with a cosine loss against the teacher's embeddings. Each student gets its own SPM vocabulary. Training can also add an MLM objective on monolingual text and a progressive-distillation curriculum. Because the encoders stay mutually compatible, their outputs can be used directly for bitext mining. The approach clearly beats the original LASER on FLORES and matches or exceeds LaBSE. The authors cover 50 African languages, many of which no other encoder supports. Bitexts mined for 44 of them improve NMT into English by 3.8 BLEU on average. This shows that new languages can be added cheaply and incrementally.

Key points

  • Models:
    • LASER2 (teacher) is a 5-layer BiLSTM producing 1024-dim embeddings, trained on 93 languages with SPM and implemented in fairseq.
    • LASER3 students have about 250M parameters each.
    • Each student is anchored with 2M English monolingual sentences plus 2M English–Spanish CCMatrix pairs.
  • Joint training on monolingual data: combining the cosine loss with an MLM loss lowers the xsim error for Wolof from 21.1% to 5.9%.
  • Evaluation and mining: encoders are evaluated on FLORES, which includes 44 African languages, and on Tatoeba. Mining runs against 21.5 billion English sentences with a single margin threshold of 1.06 for all languages.
  • NMT gains:
    • Average BLEU rises from 11.0 to 14.8.
    • Somali improves the most, from 5.1 to 21.1 BLEU after adding 4.9M mined pairs.
    • 13 languages, including Akan and Wolof, stay below 5 BLEU, mostly because little monolingual data is available.
  • Release: the training code, models and mined bitexts are freely available in the fairseq repository (nllb/laser_distillation).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 74 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-emnlp.154

@inproceedings{heffernan2022bitext,
  title = {{Bitext Mining Using Distilled Sentence Representations for Low-Resource Languages}},
  author = {Heffernan, Kevin and Çelebi, Onur and Schwenk, Holger},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.154},
  url = {https://doi.org/10.18653/v1/2022.findings-emnlp.154}
}

References

34 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jade Abbott and Laura Martinus. 2019. Benchmarking neural machine translation for Southern African 9 https://github.com/facebookresearch/ LASER/ languages. In Proceedings of the 2019 Workshop on Widening NLP, pages 98–101, Florence, Italy. Association for Computational Linguistics. ↗
  2. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges. http://ar… find our card
  3. Mikel Artetxe and Holger Schwenk. 2019a. Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings. In ACL, pages 3197–3203. find our card
  4. Mikel Artetxe and Holger Schwenk. 2019b. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. TACL, pages 597–610. find our card
  5. Paul Azunre, Salomey Osei, Salomey Addo, Lawrence Asamoah Adu-Gyamfi, Stephen Moore, Bernard Adabankah, Bernard Opoku, Clara Asare-Nyarko, Samuel Nyarko, Cynthia Amoaba, et al. 2021. NLP for ghanaian languages. arXiv preprint arXiv:2103.15475. — [arXiv:2103.15475](https://arxiv.org/abs/2103.15475) find
  6. Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Firat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, Theresa Breiner, Vera Axelrod, Jason Riesa, Yuan Cao, Mia Xu Chen, Klaus Macherey, Maxim Krikun, Pidong Wang, Alexander Gutkin, Apurva Shah, Yanp… — [arXiv:2205.03983](https://arxiv.org/abs/2205.03983) find
  7. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In ACL, pages 8440–8451. find
  8. Alexis Conneau and Guillaume Lample. 2019. Cross-lingual language model pretraining. In NeurIPS. find
  9. Raj Dabre and Aneerav Sukhoo. 2022. Morisienmt: A dataset for mauritian creole machine translation. arXiv preprint arXiv:2206.02421. — [arXiv:2206.02421](https://arxiv.org/abs/2206.02421) find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186. find
  11. Chris Chinenye Emezue and Femi Pancrace Bonaventure Dossou. 2020. Ffr v1. 1: Fon-french neural machine translation. In Proceedings of the The Fourth Widening Natural Language Processing Workshop, pages 83–87. 2109 9 find
  12. Carlos Escolano, Marta R. Costa-jussà, José A. R. Fonollosa, and Mikel Artetxe. 2021. Multilingual machine translation: Closing the gap between shared and language-specific encoder-decoders. In EACL, pages 944–948, Online. ACL. find
  13. Martinus, Kolawole Tajudeen, Kevin Degila, Kelechi Ogueji, Kathleen Siminyu, Julia Kreutzer, Jason Webster, Jamiil Toure Ali, Jade Z. Abbott, Iroro Orife, Ignatius Ezeani, Idris Abdulkabir Dangana, Herman Kamper, Hady Elsahar, Goodness Duru, Ghollah Kioko, Espoir Murhabazi, Elan Van Biljon, Daniel W… find our card
  14. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020. Beyond english-centric multiling… find
  15. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Ari-vazhaga, and Wei Wang. 2020. Language-agnostic BERT sentence embedding. https://arxiv.org/ abs/2007.01852 . ↗ our card
  16. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2021. The FLORES-101 evaluation benchmark for low-resource and multilingual machine translation. https://arxiv.org/abs/ 2106.03193 . ↗
  17. Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. https://arxiv.org/abs/1802.06893 . ↗ our card
  18. Gilles Hacheme. 2021. English2gbe: A multilingual machine translation model for {Fon/Ewe}gbe. CoRR, abs/2112.11482. — [arXiv:2112.11482](https://arxiv.org/abs/2112.11482) find
  19. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020a. XTREME: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. In ICML, pages 4411–4421. find our card
  20. Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020b. XTREME: a massively multilingual multi-task benchmark for evaluating cross-lingual generalization. In https://arxiv.org/abs/2003.11080 . ↗ our card
  21. Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, Alexandre Muzio, Saksham Singhal, Hany Hassan Awadalla, Xia Song, and Furu Wei. 2021. DeltaLM: encoder-decoder pre-training for language generation and translation by augmenting pretrained multilingual encoders. https://arxiv.org/abs/2106.13736 . — [arXiv:2106.13736](https://arxiv.org/abs/2106.13736) ↗
  22. Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi E. Fasubaa, Tajudeen Kolawole, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Shamsuddeen Hassan Muhammad, Salomon Kabongo, Salomey Osei, Sackey Freshia, Rubungo Andre Niyongabo, Ricky Macharm, Perez Ogayo, Orevaoghene Ahia, Musie Meressa, M… find
  23. Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantad… find our card
  24. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese bert-networks. In EMNLP, pages 3982–3992. find
  25. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In EMNLP, pages 4512– 4525. find our card
  26. Sebastian Ruder, Noah Constant, Jan Botha, Aditya Siddhant, Orhan Firat, Jinlan Fu, Pengfei Liu, Junjie Hu, Dan Garrette, Graham Neubig, and Melvin Johnson. 2021. XTREME-R: Towards more challenging and nuanced multilingual evaluation. In EMNLP, pages 10215–10245. find our card
  27. Holger Schwenk, Guillaume Wenzek, Sergey Edunov, Edouard Grave, Armand Joulin, and Angela Fan. 2021. CCMatrix: Mining billions of high-quality parallel sentences on the web. In ACL, page 6490–6500. find our card
  28. Aditya Siddhant, Ankur Bapna, Orhan Firat, Yuan Cao, Mia Xu Chen, Isaac Caswell, and Xavier Garcia. 2110 10 2022. Towards the next 1000 languages in multilingual machine translation: Exploring the synergy between supervised and self-supervised learning. https://arxiv.org/abs/2201.03110 . — [arXiv:2201.03110](https://arxiv.org/abs/2201.03110) ↗
  29. Kathleen Siminyu, Godson Kalipe, Davor Orlic, Jade Abbott, Vukosi Marivate, Sackey Freshia, Prateek Sibal, Bhanu Neupane, David I. Adelani, Amelia Taylor, Jamiil Toure ALI, Kevin Degila, Momboladji Balogoun, Thierno Ibrahima DIOP, Davis David, Chayma Fourati, Hatem Haddad, and Malek Naski. 2021. AI4… — [arXiv:2104.02516](https://arxiv.org/abs/2104.02516) find
  30. A We trained our students to minimize the xsim score of each language with respect to the English LASER2 teacher. In order to best consider specificities of languages, several independent student models were trained: • Semitic languages: amh and tir • Kwa languages: aka, ewe, fon, and twi J. Tiedema… find
  31. Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Dongdong Zhang, and Furu Wei. 2022. Deep-Net: scaling transformers to 1,000 layers. https://arxiv.org/abs/2203.00555 . — [arXiv:2203.00555](https://arxiv.org/abs/2203.00555) ↗
  32. Zihan Wang, Karthikeyan K, Stephen Mayhew, and Dan Roth. 2020. Extending multilingual BERT to low-resource languages. In EMNLP, pages 2649–2656. find
  33. Yinfei Yang, Gustavo Hernandez Abrego, Steve Yuan, Qinlan Shen Mandy Guo, Daniel Cer, Brian Strope Yun-hsuan Sun and, and Ray Kurzweil. 2019. Improving multilingual sentence embedding using bidirectional dual encoder with additive margin softmax. In IJCAI, pages 5370–5378. find
  34. Analysis of zero-short performance of multiple student encoders • Senegambian languages: wol (+bsc, cou and fuv) • remaining 43 languages find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/heffernan-2022-bitext-mining-distilled-sentence-representations/