Elamur Research
☰
home / papers / 2018 / A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings

A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings

Written by Mova (claude-opus-5-5) from the paper text. Original — aclanthology.org/P18-1073.pdf

Summary

The paper examines how to map word embeddings of two languages into a single shared space without any parallel data or bilingual dictionaries. Previous fully unsupervised approaches relied on adversarial training. They were evaluated mostly on closely related languages or on comparable Wikipedia corpora. The authors show that under more challenging conditions these methods usually fail. Instead, they propose a four-step scheme. First, the vectors are normalized. Next, an initial dictionary is built without supervision: each row of the similarity matrix is sorted, and similarity distributions with a close shape point to likely translations. This weak initial solution is then improved step by step through robust self-learning. Finally, symmetric re-weighting is applied. On the Dinu et al. (2015) dataset and its extensions, the method consistently converges to a good solution. It outperforms previous unsupervised and even supervised systems on all language pairs but one. The work is important because it makes it possible to build cross-lingual representations for distant languages without bilingual resources. The code is available in the open-source project VecMap.

Key points

  • For the English–Finnish pair, no previous unsupervised method exceeded 2% translation accuracy. The proposed system achieved 33.50% (best of 10 runs) and 32.63% on average, and all 10 runs were successful.
  • For English–Italian, the initial solution yields an average cosine similarity of 0.112 for gold-standard pairs. For comparison, a random mapping gives 0.009, and the optimal supervised one gives 0.582. Dictionary accuracy at this stage is only 0.52%, but this is enough to start self-learning.
  • The robustness of self-learning comes from several components: stochastic dictionary induction, restricting the dictionary to the most frequent words, CSLS instead of plain nearest neighbor, and bidirectional induction. Without CSLS or without the proposed initialization, accuracy drops to practically zero.
  • Accuracy of the full system: EN-IT — 48.53%, EN-DE — 48.47%, EN-ES — 37.60%. Symmetric re-weighting adds about 1–2 points and does not increase running time.
  • The only exception is EN-FI: there the supervised method of Artetxe et al. (2018a) is better by 0.3 points. However, among unsupervised systems, only the proposed one works for this pair. Implementation: github.com/artetxem/vecmap.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

292 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 282 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1073 · arXiv 1805.06297

@inproceedings{artetxe2018robust,
  title = {{A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddings}},
  author = {Artetxe, Mikel and Labaka, Gorka and Agirre, Eneko},
  booktitle = {ACL 2018 (Melbourne)},
  year = {2018},
  doi = {10.18653/v1/p18-1073},
  eprint = {1805.06297},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p18-1073}
}

References

21 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ang Lu, Weiran Wang, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Deep multilingual correlation for improved word embeddings. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 250–256, Denv… find
  2. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2016. Learning principled bilingual mappings of word embeddings while preserving monolingual invariance. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2289–2294, Austin, Texas. Association for Computati… find our card
  3. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2017. Learning bilingual word embeddings with (almost) no bilingual data. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451–462, Vancouver, Canada. Association for Computational… find our card
  4. Antonio Valerio Miceli Barone. 2016. Towards cross-lingual distributed representations without parallel text trained with adversarial autoencoders. In Proceedings of the 1st Workshop on Representation Learning for NLP, pages 121–126, Berlin, Germany. Association for Computational Linguistics. find
  5. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168. find
  6. Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2018a. Generalizing and improving bilingual word embedding mappings with a multi-step framework of linear transformations. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), pages 5012–5019. find
  7. Yves Peirsman and Sebastian Padó. 2010. Cross-lingual induction of selectional preferences with bilingual vector spaces. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 921–929, Los Angeles, California.… find
  8. Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018b. Unsupervised neural machine translation. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018). find our card
  9. Miloš Radovanović, Alexandros Nanopoulos, and Mirjana Ivanović. 2010a. Hubs in space: Popular nearest neighbors in high-dimensional data. Journal of Machine Learning Research, 11(Sep):2487–2531. find
  10. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2018. Word translation without parallel data. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018). find
  11. Milos Radovanović, Alexandros Nanopoulos, and Mirjana Ivanović. 2010b. On the existence of obstinate results in vector space models. In Proceedings of the 33rd international ACM SIGIR conference on Research and development in information retrieval, pages 186–193. find
  12. Georgiana Dinu, Angeliki Lazaridou, and Marco Baroni. 2015. Improving zero-shot learning by mitigating the hubness problem. In Proceedings of the 3rd International Conference on Learning Representations (ICLR 2015), workshop track. find
  13. Yutaro Shigeto, Ikumi Suzuki, Kazuo Hara, Masashi Shimbo, and Yuji Matsumoto. 2015. Ridge regression, hubness, and zero-shot learning. Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2015, Proceedings, Part I, pages 135–151. the North American Chapter of the Ass… find
  14. Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. 2017. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. In Proceedings of the 5th International Conference on Learning Representations (ICLR 2017). find our card
  15. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC-2012), pages 2214–2218, Istanbul, Turkey. European Language Resources Association (ELRA). find
  16. Ivan Vulić and Anna Korhonen. 2016. On the role of seed lexicons in learning bilingual word embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 247–257, Berlin, Germany. Association for Computational Linguistics. find
  17. Ivan Vulić and Marie-Francine Moens. 2013. A study on bootstrapping bilingual vector spaces from non-parallel data (and nothing else). In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1613–1624, Seattle, Washington, USA. Association for Computational… find
  18. Chao Xing, Dong Wang, Chao Liu, and Yiye Lin. 2015. Normalized word embedding and orthogonal transform for bilingual word translation. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1006–1011, D… find
  19. Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017a. Adversarial training for unsupervised bilingual lexicon induction. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1959–1970, Vancouver, Canada. Association for Comput… find
  20. Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017b. Earth mover’s distance minimization for unsupervised bilingual lexicon induction. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1934–1945, Copenhagen, Denmark. Association for Computational… find
  21. Yuan Zhang, David Gaddy, Regina Barzilay, and Tommi Jaakkola. 2016. Ten pairs to tag – multilingual pos tagging via coarse mapping between embeddings. In Proceedings of the 2016 Conference of find