Elamur Research⌕
☰
home / papers / 2017 / Learning Joint Multilingual Sentence Representations with Neural Machine Translation

Learning Joint Multilingual Sentence Representations with Neural Machine Translation

Summary

The paper asks whether neural machine translation can act as a training signal for sentence embeddings that do not depend on the input language. The authors build an NMT system with one encoder and one decoder per language and compress each sentence into a single fixed-size vector. They drop the usual attention mechanism, so the decoders can be discarded after training and only the encoders remain. They describe a family of "partial training paths" (1:1, M:1, 1:N, M:N). Each path decides, per mini-batch, which encoders receive input and which decoders receive targets, and the authors discuss the parallel data each one needs. To measure quality, they define a cross-lingual similarity error rate that checks whether translations end up next to each other. Across six typologically diverse languages, the average error over all 21 language pairs is 1.2%. They also study how this changes when the search pool grows to 1.4 million out-of-domain sentences. Nearest neighbours turn out to be paraphrases with different wording, both within and across languages. This points to uses in multilingual search and cross-lingual transfer.

Key points

  • Six languages are covered: English, French, Spanish, Russian, Arabic and Chinese. Together they give 21 language pairs, and the cross-lingual similarity error averaged across them comes to 1.2%.
  • Retrieval is scaled up to 1.4M sentences drawn from an out-of-domain corpus. Qualitative search examples (monolingual and EN→FR/ES) come from Europarl and use cosine similarity.
  • The architecture has L encoders and L decoders and uses fixed-size sentence vectors without attention. Self-attention or inner-attention are mentioned as possible input-only alternatives.
  • Training strategies differ in their data needs:
    • 1:1 with a shared target, or M:1 with averaged embeddings, needs only bitexts with a pivot language. However, the pivot language itself never gets an embedding.
    • 1:N, which translates into all other languages without autoencoding, gives embeddings for every language but requires L-way parallel data such as Europarl, TED or UN.
  • Close neighbours in embedding space are semantically related but often differ strongly in syntax and structure.

Interesting ideas

  • I1. Shared decoder as an alignment pressure: routing many encoders into one shared decoder of limited capacity pushes all languages toward a common representation. This is a cheap way to align languages without an explicit distance loss.
  • I2. Per-mini-batch partial training paths: the active encoders and decoders can change from batch to batch, which lets one model mix bitexts, multi-way parallel text and even monolingual data. This is useful when a low-resource language such as Ukrainian has uneven parallel coverage.
  • I3. Avoid trivial autoencoding: feeding the target language back as input turns translation into an easier copying task, which may weaken the representation. Excluding the source language from the targets keeps the objective meaningful.
  • I4. Evaluate embeddings by translation retrieval: checking whether a sentence's translation is its nearest neighbour gives a simple, language-agnostic quality metric. Growing the search pool to millions of distractors adds a realistic stress test for any multilingual encoder.
  • I5. Inspect the score falloff qualitatively: listing the top neighbours with their cosine scores shows how similarity drops from exact translations to partial topical matches. This helps set retrieval thresholds, for example when mining parallel sentences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

98 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 94 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w17-2619

@inproceedings{schwenk2017learning,
  title = {{Learning Joint Multilingual Sentence Representations with Neural Machine Translation}},
  author = {Schwenk, Holger and Douze, Matthijs},
  booktitle = {Proceedings of the 2nd Workshop on Representation Learning for NLP},
  year = {2017},
  doi = {10.18653/v1/w17-2619},
  url = {https://doi.org/10.18653/v1/w17-2619}
}

References

40 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Gouews, Y. Bengio, and G. Corrado. 2014. Bilbowa: Fast bilingual distributed representations without word alignments. In ICML. find
  2. Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval workshop. find our card
  3. Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual models for compositional distributed semantics. In ACL. pages 58–68. find our card
  4. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. 2017. A simple but tough-to-beat baseline for sentence embeddings. In ICLR. find
  5. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with gpus. arXiv preprint arXiv:1702.08734 . find our card
  6. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  7. Melvin Johnson et al. 2016. Google’s multilingual neural machine translation system: Enabling zero-shot translation. In https://arxiv.org/ abs/1611.04558. ↗
  8. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. Wit 3 : Web inventory of transcribed and translated talks. In EAMT. pages 261–268. find
  9. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP. pages 1700–1709. find our card
  10. Sarath Chandar, Mitesh M. Khapra, Hugo Larochelle, and Balaraman Ravindran. 2016. Correlation neural networks. Neural Computation 28:257–285. find
  11. A. Klementiev, I. Titov, and B. Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Coling. find our card
  12. Sarath Chandar, Mitesh M. Khapra, Balaraman Ravindran, Vikas Raykar, and Amrita Saha. 2013. Multilingual deep learning. In NIPS DL wshop. find
  13. T. Kociský, K.M. Hermann, and P. Blunsom. 2014. Learning bilingual word representations by marginalizing alignments. In ACL. pages 224–229. find
  14. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP. find
  15. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In ICLR. find our card
  16. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML. pages 160–167. find
  17. Yang Liu, Chenjie Sun, Lei Lin, and Xiaolong Wang. 2016. Learning natural language inference using bidirectional lstm model and inner-attention. In https://arxiv.org/abs/1605.09090. — [arXiv:1605.09090](https://arxiv.org/abs/1605.09090) ↗
  18. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In https://arxiv.org/abs/1705.02364. ↗ our card
  19. Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015a. Multi-task sequence to sequence learning. In ICLR. find our card
  20. J. Coulmance, J.M. Marty, G. Wenzek, and A. Ben-haloum. 2015. Trans-gram, fast cross-lingual word embeddings. In EMNLP. find
  21. T. Luong, H. Pham, and C.D. Manning. 2015b. Bilingual word representations with monolingual quality in mind. In ACL workshop on Vector Space Modeling for NLP. pages 151–159. find our card
  22. Daxiang Dong, Huan Wu, Wei He, Dianhai Yu, and Haifeng wang. 2015. Multi-task learning for multiple language translation. In ACL. pages 1723–1732. find our card
  23. Tomas Mikolov, Wen tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous word space representations. In NAACL. pages 746–751. find
  24. Orhan Firat, Kyunghyun Choa, and Yoshua Bengio. 2016a. Multi-way, multilingual neural machine translation with shared attention mechanism. In NAACL. find
  25. Aditua Mogadala and Achim Rettinger. 2016. Bilingual word embeddings fomr parallel and non-parallel corpora for cross-language classification. In NAACL. pages 692–702. find
  26. Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos T. Yarman Vural, and Kyunghyun Cho. 2016b. Zero-resource translation with multi-lingual neural machine translation. In EMNLP. find our card
  27. Hideki Nakayama and Noriki Nishida. 2016. Zero-resource machine translation by multimodal encoder-decoder network with multimedia pivot. In https://arxiv.org/abs/1611.04503. — [arXiv:1611.04503](https://arxiv.org/abs/1611.04503) ↗
  28. Andrea Frome, Grep S. Corrado, Jonathon Shlens, Samy Bengio, Jeffrey Dean, marc’Aurelio Ranzato, and Thomas Mikolov. 2013. DeViSa:E a deep visual-semantic embedding model. In NIPS. find
  29. Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng. 2011. Multimodal deep learning. In ICML. and machine translation. In https://arxiv.org/abs/1610.05011. — [arXiv:1610.05011](https://arxiv.org/abs/1610.05011) ↗
  30. Hieu Pham, Minh-Thang Luong, and Christopher D. Manning. 2015. Learning distributed representations for multilingual text sequences. In Workshop on Vector Space Modeling for NLP. find
  31. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS. pages 649–657. find our card
  32. Amrita Saha, Mitesh M. Kharpa, Sarath Chandar, Janarthanan Rajendran, and Kyunghyun Cho. 2016. A correlational encoder decoder architecture for pivot based sequence generation. In https://arxiv.org/abs/1606.04754. ↗
  33. Han Zhao, Zhengdong Lu, and Pascal Poupart. 2015. Self-adaptive hierarchical sentence model. In https://arxiv.org/abs/1504.05070. — [arXiv:1504.05070](https://arxiv.org/abs/1504.05070) ↗
  34. Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. 2016a. Deep recurrent models with fast-forward connections for neural machine translation. TACL 4:371–383. find
  35. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL. pages 1715–1725. find
  36. Richard Socher, Jeffrey Pennington, Eric H. Huang, Andrew Y. Ng, and Christopher D. Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In EMNLP. find our card
  37. Xinjie Zhou, Xiaojun Wan, and Jianguo Xiao. 2016b. Cross-lingual sentiment classification with bilingual document representation learning. In ACL. find
  38. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS. pages 3104–3112. find
  39. M Yonghui Wu et al. 2016. Google’s neural machine translation system: Bridging the gap between human Barret Zoph and Kevin Knight. 2016. Multi-source neural translation. In NAACL. pages 30–34. find our card
  40. Ziemski, Marcin Juncys-Dowmunt, and B. Pouliquen. 2016. The united nations parallel corpus v1.0. In LREC. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schwenk-2017-learning-joint-multilingual-sentence-representations/