Elamur Research
☰
home / papers / 2016 / Learning principled bilingual mappings of word embeddings while preserving monolingual invariance

Learning principled bilingual mappings of word embeddings while preserving monolingual invariance

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D16-1250.pdf

Summary

The paper addresses cross-lingual word embedding mapping: learning a linear transformation that maps independently trained source-language vectors into a target-language space, using a bilingual dictionary as supervision. Artetxe, Labaka and Agirre start from the least-squares objective of Mikolov et al. (2013b). They then add three constraints: an orthogonal mapping, length normalization, and dimension-wise mean centering. Orthogonality keeps dot products unchanged, so the source space stays intact for monolingual tasks. The resulting framework yields an exact, SVD-based closed-form solution instead of gradient descent. It also reframes earlier methods. Xing et al. (2015) amounts to an orthogonal mapping on unit-length vectors. Faruqui and Dyer's CCA approach is close to the full model but also forces dimensions to be uncorrelated with equal variance. The authors evaluate English–Italian translation induction and the English word analogy task. The full configuration achieves the best translation accuracy while leaving analogy performance almost unchanged. This shows that orthogonality, more than normalization, drives the improvements.

Key points

  • Data: The Dinu et al. (2015) English–Italian dataset provides 300-dimensional word2vec CBOW embeddings. English vectors come from ukWaC + Wikipedia + BNC (2.8B words) and Italian vectors from itWaC (1.6B words). It includes a Europarl-derived dictionary with 5,000 training pairs and 1,500 test pairs.
  • Translation induction (EN-IT accuracy):
    • Mikolov et al. (2013b): 34.93%
    • Xing et al. (2015): 36.87%
    • Faruqui and Dyer (2014), CCA with 180 dimensions: 37.80%
    • Proposed method (orthogonal + length normalization + mean centering): 39.27%
  • Monolingual analogy (Mikolov et al. 2013a questions, 30,000-word vocabulary threshold): The original embeddings score 76.66%. The proposed method loses only 0.07 points, compared with 2.86 for Mikolov et al. and 7.02 for CCA.
  • Exact solution: The optimal orthogonal mapping is W = VUᵀ, obtained from the SVD of ZᵀX. Its cost grows linearly with vocabulary size.
  • Code: The open-source Python/NumPy implementation is released as vecmap (github.com/artetxem/vecmap).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

164 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 158 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1250

@inproceedings{artetxe2016learning,
  title = {{Learning principled bilingual mappings of word embeddings while preserving monolingual invariance}},
  author = {Artetxe, Mikel and Labaka, Gorka and Agirre, Eneko},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1250},
  url = {https://doi.org/10.18653/v1/d16-1250}
}

References

8 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sarath Chandar A P, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas C Raykar, and Amrita Saha. 2014. An autoencoder approach to learning bilingual word representations. In Advances in Neural Information Processing Systems Acknowledgments 27, pages 1853–1861. Georgiana Din… find
  2. Min-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Bilingual word representations with monolingual quality in mind. In NAACL Workshop on Vector Space Modeling for NLP, pages 151–159. find our card
  3. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  4. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168. find
  5. Michael Wick, Pallika Kanani, and Adam Pocock. 2016. Minimally-constrained multilingual embeddings via artificial code-switching. In Thirtieth AAAI conference on Artificial Intelligence (AAAI). find
  6. Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006–1011. find
  7. Kai Zhao, Hany Hassan, and Michael Auli. 2015. Learning translation models from monolingual continuous representations. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1527–1536. find
  8. Will Y. Zou, Richard Socher, Daniel Cer, and Christopher D. Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1393–1398. find