Elamur Research
☰
home / papers / 2017 / Offline bilingual word vectors, orthogonal transformations and the inverted softmax

Offline bilingual word vectors, orthogonal transformations and the inverted softmax

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1702.03859

Summary

The paper studies how to build bilingual word embeddings "offline": two monolingual vector spaces are trained separately and then aligned with a mapping learned from a translation dictionary, following Mikolov et al. (2013a). The authors argue that a mapping that is consistent in both translation directions must be orthogonal, essentially a rotation of one space onto the other. Such a mapping has an exact closed-form solution via the SVD and is equivalent to the orthogonal Procrustes problem. They also relate it to CCA and add optional dimensionality reduction based on the singular values. To counter hubness during retrieval, they introduce an "inverted softmax" that normalises similarity scores over source words. The method is evaluated on English–Italian word translation using Dinu et al.'s vectors, dictionary and test set, and on sentence retrieval with Europarl. It clearly outperforms Mikolov's least-squares mapping. Because the orthogonal mapping tolerates noisy supervision, it works even without an expert dictionary. This makes cross-lingual alignment cheap and feasible for language pairs with few bilingual resources.

Key points

  • With SVD plus the inverted softmax, English→Italian precision@1 rises from 34% (Mikolov) to 43%, and Italian→English rises from 25% to 37%.
  • The training dictionary contains 5,000 translation pairs. Replacing it with a pseudo-dictionary of identical character strings shared by both languages yields 40% and 34% precision, compared with only 1% and 3% for Mikolov's method.
  • A phrase dictionary built from the Europarl aligned corpus, using summed and normalised word vectors as sentence vectors, reaches 43% and 38%. This matches the expert word dictionary.
  • For sentence retrieval, the correct Italian translation of an English sentence is found among 200k candidates with 68% precision@1. In the English→Italian direction with the phrase dictionary, the inverted softmax (n_s = 12800) scores 0.825 @5 and 0.862 @10.
  • The paper places the method alongside closely related work: Xing et al. (2015), whose approach was approximate, Artetxe et al. (2016), which proposed the same approach independently, and Faruqui & Dyer's CCA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

188 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 172 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1702.03859 · arXiv 1702.03859

@misc{smith2017offline,
  title = {{Offline bilingual word vectors, orthogonal transformations and the inverted softmax}},
  author = {Smith, Samuel L. and Turban, David H. P. and Hamblin, Steven and Hammerla, Nils Y.},
  year = {2017},
  doi = {10.48550/arxiv.1702.03859},
  eprint = {1702.03859},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1702.03859}
}

References

20 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A Smith. Massively multilingual word embeddings. arXiv:1602.01925, 2016. find our card
  2. M Artetxe, G Labaka, and E Agirre. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2289–2294, 2016. find our card
  3. Sarath Chandar, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas C Raykar, and Amrita Saha. An autoencoder approach to learning bilingual word representations. In Advances in Neural Information Processing Systems, pp. 1853–1861, 2014. find
  4. Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. Improving zero-shot learning by mitigating the hubness problem. arXiv:1412.6568, 2014. — [arXiv:1412.6568](https://arxiv.org/abs/1412.6568) find
  5. Manaal Faruqui and Chris Dyer. Improving vector space word representations using multilingual correlation. In Proceedings of the 2014 conference of the Association for Computational Linguistics. Association for Computational Linguistics, 2014. find
  6. Stephan Gouws, Yoshua Bengio, and Greg Corrado. Bilbowa: Fast bilingual distributed representations without word alignments. In Proceedings of The 32nd International Conference on Machine Learning, pp. 748–756, 2015. find
  7. Aria Haghighi, Percy Liang, Taylor Berg-Kirkpatrick, and Dan Klein. Learning bilingual lexicons from monolingual corpora. In ACL, volume 2008, pp. 771–779, 2008. find
  8. Karl Moritz Hermann and Phil Blunsom. Multilingual distributed representations without word alignment. arXiv preprint arXiv:1312.6173, 2013. — [arXiv:1312.6173](https://arxiv.org/abs/1312.6173) find
  9. Yoon Kim. Convolutional neural networks for sentence classification. arXiv:1408.5882, 2014. find our card
  10. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in neural information processing systems, pp. 3294–3302, 2015. find
  11. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. Inducing crosslingual distributed representations of words. 2012. find
  12. Philipp Koehn. Europarl: A parallel corpus for statistical machine translation. In MT summit, volume 5, pp. 79–86, 2005. find
  13. Quoc V Le and Tomas Mikolov. Distributed representations of sentences and documents. In ICML, volume 14, pp. 1188–1196, 2014. find our card
  14. Ang Lu, Weiran Wang, Mohit Bansal, Kevin Gimpel, and Karen Livescu. Deep multilingual correlation for improved word embeddings. In Proceedings of NAACL, 2015. find
  15. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. Exploiting similarities among languages for machine translation. arXiv:1309.4168, 2013a. find
  16. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013b. find
  17. William H Press. Canonical correlation clarified by singular value decomposition. 2011. find
  18. Peter H Schönemann. A generalized solution of the orthogonal procrustes problem. Psychometrika, 31(1), 1966. find
  19. Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1006–1011, 2015. find
  20. Will Y Zou, Richard Socher, Daniel M Cer, and Christopher D Manning. Bilingual word embeddings for phrase-based machine translation. In EMNLP, pp. 1393–1398, 2013. find