Elamur Research
☰
home / papers / 2016 / Massively Multilingual Word Embeddings

Massively Multilingual Word Embeddings

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1602.01925

Summary

Earlier work on cross-lingual word vectors mostly covered language pairs and often depended on parallel text. This paper places words from more than fifty languages into a single common vector space using only monolingual corpora and bilingual dictionaries. It introduces two estimators:

  • multiCluster links translation-equivalent words into dictionary-derived clusters. It then trains skipgram on corpora in which each token is replaced by its cluster ID.
  • multiCCA learns CCA projections from bilingual dictionaries and maps each language's monolingual embeddings into the English space.

Both are trained for 59 languages. They are compared with two parallel-data baselines, multiSkip and translation-invariance. For evaluation, the authors adapt QVEC to the multilingual case (multiQVEC) and propose multiQVEC-CCA, which removes a theoretical flaw in the original metric. Both new metrics track downstream results (dependency parsing and document classification) more closely than earlier intrinsic measures. A public evaluation portal and open-source code give the field a shared benchmark. This is especially valuable for transferring models to lower-resource languages.

Key points

  • multiCluster and multiCCA need no parallel corpora. When parallel text is available, dictionaries can be extracted from it with fast_align, using a translation-probability threshold of τ = 0.1.
  • multiCCA uses English as the pivot space because English has the largest corpora and the most bilingual dictionaries.
  • With 59 languages, multiCCA beats multiCluster on all metrics. Merging translations into one vector adds semantic ambiguity, which hurts multiCluster at that scale.
  • The 12-language comparison covers English, French, German, Spanish, Swedish, Italian, Danish, Finnish, Hungarian, Czech, Greek and Bulgarian. In it, a dictionary-based method wins five evaluations, including both extrinsic tasks. Parsing scores were 61.0 for multiCluster versus 59.8 for translation-invariance, and multiCluster and multiCCA tied at 92.1 on document classification. multiSkip never ranks first.
  • The web portal scores uploaded embeddings on nine metrics and has separate development and test modes.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

135 papers of our library cite it (of the 140673 whose bibliographies we read); 8 of them have a page here, 127 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1602.01925 · arXiv 1602.01925

@misc{ammar2016massively,
  title = {{Massively Multilingual Word Embeddings}},
  author = {Ammar, Waleed and Mulcaire, George and Tsvetkov, Yulia and Lample, Guillaume and Dyer, Chris and Smith, Noah A.},
  year = {2016},
  doi = {10.48550/arxiv.1602.01925},
  eprint = {1602.01925},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1602.01925}
}

References

38 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Željko Agić, Maria Jesus Aranzabe, Aitziber Atutxa, Cristina Bosco, Jinho Choi, Marie-Catherine de Marneffe, Timothy Dozat, Richárd Farkas, Jennifer Foster, Filip Ginter, Iakes Goenaga, Koldo Gojenola, Yoav Goldberg, Jan Hajič, Anders Trærup Johannsen, Jenna Kanerva, Juha Kuokkala, Veronika Laip… find
  2. Eneko Agirre, Enrique Alfonseca, Keith Hall, Jana Kravalova, Marius Paşca, and Aitor Soroa. 2009. A study on similarity and relatedness using distributional and WordNet-based approaches. In Proc. of NAACL, pages 19–27. find
  3. Rami Al-Rfou’, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual nlp. In CONLL. find our card
  4. Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. JAIR. find
  5. José Camacho-Collados, Mohammad Taher Pilehvar, and Roberto Navigli. 2015. A framework for the construction of monolingual and cross-lingual word similarity datasets. In Proc. of ACL. find
  6. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In Proc. of EMNLP. find
  7. Jocelyn Coulmance, Jean-Marc Marty, Guillaume Wenzek, and Amine Benhalloum. 2015. Trans-gram, fast cross-lingual word-embeddings. In Proc. of EMNLP. find
  8. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM Model 2. In Proc. of NAACL. find
  9. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proc. of ACL. find our card
  10. Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. Proc. of EACL. find
  11. Matt Gardner, Kejun Huang, Evangelos Papalexakis, Xiao Fu, Partha Talukdar, Christos Faloutsos, Nicholas Sidiropoulos, and Tom Mitchell. 2015. Translation invariant word embeddings. In Proc. of EMNLP. find
  12. Stephan Gouws, Yoshua Bengio, and Greg Corrado. 2014. Bilbowa: Fast bilingual distributed representations without word alignments. arXiv preprint arXiv:1410.2455. — [arXiv:1410.2455](https://arxiv.org/abs/1410.2455) find
  13. Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2015. Cross-lingual dependency parsing based on distributed representations. In Proc. of ACL. find
  14. Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2016. A representation learning framework for multi-source transfer parsing. In Proc. of AAAI. find
  15. Michael U Gutmann and Aapo Hyvärinen. 2012. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. JMLR. find
  16. Aria Haghighi, Percy Liang, Taylor Berg-Kirkpatrick, and Dan Klein. 2008. Learning bilingual lexicons from monolingual corpora. In Proc. of ACL. find
  17. Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual Models for Compositional Distributional Semantics. In Proc. of ACL. find
  18. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proc. of COLING. find
  19. Tomáš Kociskỳ, Karl Moritz Hermann, and Phil Blunsom. 2014. Learning bilingual word representations by marginalizing alignments. In arXiv preprint arXiv:1405.0947. find
  20. Ira Leviant and Roi Reichart. 2015. Judgment language matters: Towards judgment language informed vector space modeling. In arXiv preprint arXiv:1508.00106. — [arXiv:1508.00106](https://arxiv.org/abs/1508.00106) find
  21. Minh-Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In Proc. of CoNLL. find our card
  22. Minh-Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2015a. Addressing the rare word problem in neural machine translation. In Proc. of ACL. find our card
  23. Thang Luong, Hieu Pham, and Christopher D Manning. 2015b. Bilingual word representations with monolingual quality in mind. In Proc. of NAACL. find our card
  24. Héctor Martı́nez Alonso, Anders Johannsen, Sussi Olsen, Sanni Nimb, Nicolai Hartvig Sørensen, Anna Braasch, Anders Søgaard, and Bolette Sandford Pedersen. 2015. Supersense tagging for Danish. In Proc. of NODAL-IDA, page 21. find
  25. Héctor Martı́nez Alonso, Anders Johannsen, Sussi Olsen, Sanni Nimb, and Bolette Sandford Pedersen. 2016. An empirically grounded expansion of the supersense inventory. In Proc. of the Global Wordnet Conference. find
  26. Ryan McDonald, Slav Petrov, and Keith Hall. 2011. Multi-source transfer of delexicalized dependency parsers. In Proc. of EMNLP, pages 62–72. find our card
  27. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In Proc. of ICLR. find
  28. Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation. In arXiv preprint arXiv:1309.4168v1. find
  29. George A. Miller, Claudia Leacock, Randee Tengi, and Ross T. Bunker. 1993. A semantic concordance. In Proc. of HLT, pages 303–308. find
  30. Simonetta Montemagni, Francesco Barsotti, Marco Battista, Nicoletta Calzolari, Ornella Corazzari, Alessandro Lenci, Antonio Zampolli, Francesca Fanciulli, Maria Mas-setani, Remo Raffaelli, et al. 2003. Building the italian syntactic-semantic treebank. In Treebanks, pages 189–210. Springer. find
  31. Franz Joseph Och. 1999. An efficient method for determining bilingual word classes. In EACL. find
  32. Slav Petrov, Dipanjan Das, and Ryan McDonald. 2012. A universal part-of-speech tagset. In Proc. of LREC. find
  33. Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In Proc. of ICLR. find our card
  34. Oscar Täckström, Ryan Mc-Donald, and Jakob Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In Proc. of NAACL, pages 477–487. find our card
  35. Yulia Tsvetkov, Leonid Boytsov, Anatole Gershman, Eric Nyberg, and Chris Dyer. 2014. Metaphor detection with cross-lingual model transfer. In Proc. of ACL. find
  36. Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Guillaume Lample, and Chris Dyer. 2015. Evaluation of word vector representations by subspace alignment. In Proc. of EMNLP. find
  37. Daniel Zeman and Philip Resnik. 2008. Cross-language parser adaptation between related languages. In Proc. of IJCNLP, pages 35–42. find
  38. Will Y Zou, Richard Socher, Daniel M Cer, and Christopher D Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proc. of EMNLP, pages 1393–1398. find