Elamur Research
☰
home / papers / 2020 / Multilingual Universal Sentence Encoder for Semantic Retrieval

Multilingual Universal Sentence Encoder for Semantic Retrieval

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.acl-demos.12.pdf

Summary

The paper addresses the need for ready-to-use sentence embeddings that place text from many languages in one semantic space, so that similar meanings can be retrieved within and across languages. The authors add three multilingual models to the universal sentence encoder (USE) family, covering 16 languages: USE Trans (Transformer, more accurate), USE CNN (cheaper and faster) and USE QA Trans+Cxt (a question-answering variant that also encodes context). A shared encoder is trained in a multi-task dual-encoder setup on three objectives. The first is question-answer ranking on pairs mined from forums. The second is translation-pair ranking on mined bitext. The third is NLI on SNLI, which was machine-translated into the 15 non-English languages. The models set a new state of the art on monolingual and cross-lingual semantic retrieval (SR). They are competitive on bitext retrieval (BR) and ReQA. On SentEval transfer tasks they come close to, and sometimes beat, English-only encoders. Because the models are public on TensorFlow Hub and need only TensorFlow, they are a practical basis for multilingual search.

Key points

  • Tokenization and data: All languages share a single 128k SentencePiece vocabulary built from 8M language-balanced sentences, with over 99% character coverage. Each language gets at least 60M QA training pairs, topped up by machine translation where needed.
  • Configurations:
    • Transformer: 6 layers, 8 heads, hidden size 512, inputs capped at 100 tokens.
    • CNN: 2 layers, filter widths 1/2/3/5, inputs capped at 256 tokens.
  • SR results (MAP@100): USE Trans scores 89.1 on Quora and 42.3 on AskUbuntu. The earlier best from Gillick et al. (2018) scored 87.5 and 37.3.
  • Bitext retrieval on the UN corpus (P@1): USE Trans reaches 86.1 on en-es, versus 89.0 for the specialized model of Yang et al. (2019). The gap is largest on en-zh.
  • Transfer and resources:
    • On STS Bench, multilingual USE Trans scores 0.837/0.825 (dev/test), above the English-only USE Transformer at 0.802/0.766.
    • The CNN model is the fastest and lightest.
    • GPU inference runs 2–3× faster than CPU.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

172 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 163 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-demos.12

@inproceedings{yang2020multilingual,
  title = {{Multilingual Universal Sentence Encoder for Semantic Retrieval}},
  author = {Yang, Yinfei and Cer, Daniel and Ahmad, Amin and Guo, Mandy and Law, Jax and Constant, Noah and Abrego, Gustavo Hernandez and Yuan, Steve and Tar, Chris and Sung, Yun-hsuan and Strope, Brian and Kurzweil, Ray},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations},
  year = {2020},
  doi = {10.18653/v1/2020.acl-demos.12},
  url = {https://doi.org/10.18653/v1/2020.acl-demos.12}
}

References

26 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. 2018. End-to-end retrieval in continuous space. CoRR, abs/1811.08008. — [arXiv:1811.08008](https://arxiv.org/abs/1811.08008) find
  2. Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yua… find
  3. Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. 2015. Cqadupstack: A benchmark data set for community question-answering research. In Proceedings of the 20th Australasian Document Computing Symposium, ADCS ’15, pages 3:1–3:8. find
  4. K. Sparck Jones, S. Walker, and S. E. Robertson. 2000. A probabilistic model of information retrieval: Development and comparative experiments. Inf. Process. Manage., 36(6):779–808. find
  5. Mikel Artetxe and Holger Schwenk. 2019. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597–610. find our card
  6. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746–1751. find our card
  7. Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450. find our card
  8. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642. find
  9. Taku Kudo and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66– 71. find our card
  10. Tolgahan Cakaloglu, Christian Szegedy, and Xiaowei Xu. 2018. Text embeddings for retrieval from a large knowledge base. CoRR, abs/1810.10176. — [arXiv:1810.10176](https://arxiv.org/abs/1810.10176) find
  11. Tao Lei, Hrishikesh Joshi, Regina Barzilay, Tommi Jaakkola, Kateryna Tymoshenko, Alessandro Moschitti, and Lluís Màrquez. 2016. Semi-supervised question retrieval with gated convolutions. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguis… find
  12. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14. find
  13. Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Langua… find our card
  14. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In International Conference on Learning Representations (ICLR). find our card
  15. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392. find our card
  16. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1870–1879. find our card
  17. Muthuraman Chidambaram, Yinfei Yang, Daniel Cer, Steve Yuan, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Learning cross-lingual sentence representations via a multi-task dual-encoder model. CoRR, abs/1810.12836. find
  18. Jakob Uszkoreit, Jay Ponte, Ashok Popat, and Moshe Dubiner. 2010. Large scale parallel document mining for machine translation. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 1101–1109. find
  19. Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC-2018). find our card
  20. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS, pages 6000–6010. find
  21. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 670–680, Copenhagen,… find our card
  22. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
  23. Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In COLING 2004: Proceedings of the 20th International Conference on Computational Linguistics, pages 350–356. find our card
  24. Yinfei Yang, Gustavo Hernández Ábrego, Steve Yuan, Mandy Guo, Qinlan Shen, Daniel Cer, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Improving multilingual sentence embedding using bi-directional dual encoder with additive margin softmax. CoRR, abs/1902.08564. — [arXiv:1902.08564](https://arxiv.org/abs/1902.08564) find
  25. Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Learning semantic textual similarity from conversations. In Proceedings of The Third Workshop on Representation Learning for NLP, pages 164– 174. find
  26. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The united nations parallel corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), pages 3530–3534. find our card