Elamur Research⌕
☰
home / papers / 2015 / Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation

Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation

Summary

The paper revisits the translation-matrix method of Mikolov et al. (2013b), in which a linear map learned from a seed dictionary carries word vectors from one language's embedding space into another's. The authors identify a mismatch between three measures: skip-gram is trained with inner products, the map is fit with squared Euclidean error, and translations are retrieved by cosine similarity. Their fix makes all three agree. Word vectors are rescaled to unit length during training, so they lie on a hypersphere and the inner product equals the cosine. The transform is then fit by maximizing cosine similarity, with the matrix restricted to be orthogonal so that projected vectors stay unit length. The embeddings are trained with word2vec skip-gram on WMT11 English and Spanish text, and the method is evaluated on English word similarity and English-to-Spanish word translation. Normalization raises correlation with human similarity judgments. The orthogonal transform improves translation accuracy (P@1 about 41% vs. 35%). The work matters because it shows that keeping training objectives consistent with the evaluation metric gives measurable gains, and it laid groundwork for later orthogonal cross-lingual mapping methods.

Key points

  • Data and setup: Monolingual embeddings were trained with the word2vec toolkit (skip-gram, window 5) on WMT11 data.
    • The text was tokenized, duplicate sentences were removed, numbers were mapped to 'NUM', and special characters were stripped.
    • The resulting vocabularies were about 169k English and 116k Spanish tokens.
  • Bilingual dictionary: 6000 frequent English words were translated into Spanish with Google's online translation service.
    • Evaluation used 10-fold cross-validation over these pairs.
    • 99.5% of the Spanish translations were in the target vocabulary.
  • Word similarity: On WordSimilarity-353, normalized vectors correlated better with human judgments than unnormalized ones across dimensions 300–1000.
  • Baseline (unnormalized vectors, linear transform): The best setting was 800-dimensional English and 200-dimensional Spanish, giving P@1 35.36% and P@5 53.96%. This roughly matches Mikolov et al.'s reported numbers.
  • Proposed method (normalized vectors, orthogonal transform):
    • P@1 ranged from 38.99% to 41.04%, with the best at 700/700 dimensions.
    • P@5 reached up to 60.02% at 800/200.
    • It beat the linear transform in every dimension setting tested.

Interesting ideas

  • I1. Align the training objective with the evaluation metric. If retrieval uses cosine, train with a quantity that reduces to cosine. Checking every stage of a pipeline for such mismatches is a cheap diagnostic for any embedding-based system.
  • I2. Project onto the constraint set after each step. Instead of formal constrained optimization (e.g., Lagrange multipliers or SQP), the authors take an ordinary gradient step and then project back.
    • Vectors are divided by their L2 norm after each update, optionally only every n mini-batches.
    • The matrix is orthogonalized by taking its SVD and setting all singular values to one.
    • This requires minimal code changes and transfers easily to other constrained learning problems.
  • I3. Use an orthogonal map between embedding spaces. A rotation preserves lengths and angles, so the structure of the source space survives the mapping. This makes it a natural default for cross-lingual alignment, including possible Ukrainian–English lexicon induction.
  • I4. Pad and renormalize to handle mismatched dimensions. When source and target spaces differ in size, the smaller vectors are padded with a small tunable constant and renormalized so an orthogonal map still applies. The authors themselves call this ad hoc, but it is a simple workaround.
  • I5. Evaluate with cross-validation over the seed dictionary. Reporting both P@1 and P@5 averaged over 10 folds gives a more stable estimate than a single train/test split. This is useful when bilingual dictionaries are small.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

147 papers of our library cite it (of the 140884 whose bibliographies we read); 10 of them have a page here, 137 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/n15-1104

@inproceedings{xing2015normalized,
  title = {{Normalized Word Embedding and Orthogonal Transform for Bilingual Word Translation}},
  author = {Xing, Chao and Wang, Dong and Liu, Chao and Lin, Yiye},
  booktitle = {Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2015},
  doi = {10.3115/v1/n15-1104},
  url = {https://doi.org/10.3115/v1/n15-1104}
}

References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marco Baroni and Roberto Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 1183–1193. find
  2. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A neural probabilistic language model. J. Mach. Learn. Res., 3:1137–1155. find
  3. Phil Blunsom, Karl Moritz Hermann, Tomas Kocisky, et al. 2014. Learning bilingual word representations by marginalizing alignments. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, pages 224–229. find
  4. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. J. Mach. Learn. Res., 12:2493–2537. find our card
  5. Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. Proceeding of EACL. Association for Computational Linguistics. find our card
  6. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2002. Placing search in context: The concept revisited. In Proceedings of The 2002 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Te… find
  7. Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual distributed representations without word alignment. In Proceedings of International Conference on Learning Representations (ICLR). find
  8. Eric H. Huang, Richard Socher, Christopher D. Manning, and Andrew Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In Proceeding of Annual Meeting of the Association for Computational Linguistics (ACL). find our card
  9. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of COLING. Citeseer. find our card
  10. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In International Conference on Learning Representations (ICLR). find
  11. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168. find
  12. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013c. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  13. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. 2013d. Linguistic regularities in continuous space word representations. In Proceedings of The 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 746–751. Citeseer. find our card
  14. Joseph Turian, Departement d’Informatique Et, Recherche Operationnelle (diro, Universite De Montreal, Lev Ratinov, and Yoshua Bengio. 2010. Word representations: A simple and general method for semisupervised learning. In Proceeding of Annual Meeting of the Association for Computational Linguistics… find
  15. Will Y Zou, Richard Socher, Daniel M Cer, and Christopher D Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceeding of Conference on Empirical Methods on Natural Language Processing (EMNLP), pages 1393–1398. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xing-2015-normalized-word-embedding-orthogonal-transform/