Elamur Research⌕
☰
home / papers / 2016 / Cross-lingual Models of Word Embeddings: An Empirical Comparison

Cross-lingual Models of Word Embeddings: An Empirical Comparison

Paper · 2016 · Shyam Upadhyay, Manaal Faruqui, Chris Dyer et al.

Summary

Many methods learn word embeddings across languages, but each relies on a different kind of bilingual supervision, and the literature lacked a systematic, side-by-side comparison of them. Upadhyay et al. fill this gap by comparing four representative models, ordered from most to least costly supervision: BiSkip (word-aligned parallel text), BiCVM (sentence alignments), BiCCA (a bilingual lexicon) and BiVCD (comparable documents). They first show that all four can be expressed as instances of one general optimization scheme. They then train every model on the same parallel data for en-de, en-fr, en-sv and en-zh, using 200-dimensional vectors. Evaluation covers four tasks: English word similarity, dictionary induction, cross-lingual document classification and dependency parsing. Expensive supervision tends to give the best results on cross-lingual semantic tasks. Cheaper models stay close on monolingual similarity, and word-level supervision gives only a slight edge in parsing. This helps practitioners choose a method based on their target task and the resources available for their language pair.

Key points

  • Training data: Europarl v7 was used for en-de and en-sv. For en-fr, Europarl was combined with news-commentary and the UN corpus from WMT 2015. For en-zh, the FBIS news corpus (LDC2003E14) was segmented with the Stanford Chinese Segmenter. Each pair has roughly 1.7–2.0M sentence pairs.
  • Tuning: hyperparameters were chosen with the BestAvg strategy, which picks the setting with the best average across all tasks. BiSkip word alignments came from fast_align.
  • Parsing results: in Table 7, average LAS was 75.9 for BiSkip and 75.6 for BiCCA, against 74.8 for monolingual embeddings. BiCVM (67.6) and BiVCD (66.8) fell below that baseline.
  • Vector-space analysis: PCA plots show that BiSkip and BiCVM place translation pairs almost on top of each other. BiCCA and BiVCD separate antonyms such as peace/war better.
  • Scope: BilBOWA was left out because problems in its public code made it underperform even monolingual vectors. Multilingual models covering more than two languages are not part of the comparison.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓University of Illinois Urbana-ChampaignUrbana · United States108 works · 67,432 citations

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

🧭 Topics 1 Multilingual & cross-lingual NLP · 467 works

Cite

Cite as Shyam Upadhyay, Manaal Faruqui, Chris Dyer and Dan Roth. 2016. Cross-lingual Models of Word Embeddings: An Empirical Comparison. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1157

@inproceedings{upadhyay2016cross,
  title = {{Cross-lingual Models of Word Embeddings: An Empirical Comparison}},
  author = {Shyam Upadhyay, Manaal Faruqui, Chris Dyer and Dan Roth},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1157},
  url = {https://doi.org/10.18653/v1/p16-1157}
}

References

39 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Al-Rfou, Bryan Perozzi, and Steven Skiena. 2013. Polyglot: Distributed word representations for multilingual nlp. In Proc. of CoNLL. our card
  2. Waleed Ammar, George Mulcaire, Yulia Tsvetkov, Guillaume Lample, Chris Dyer, and Noah A Smith. 2016. Massively multilingual word embeddings. arXiv preprint arXiv:1602.01925. our card
  3. Francis Bond and Ryan Foster. 2013. Linking and extending an open multilingual wordnet. In Proc. of ACL. our card
  4. Sabine Buchholz and Erwin Marsi. 2006. Conll-X shared task on multilingual dependency parsing. In Proc. of CoNLL. our card
  5. Sarath Chandar, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas C Raykar, and Amrita Saha. 2014. An autoencoder approach to learning bilingual word representations. In Proc. of NIPS. find
  6. Jocelyn Coulmance, Jean-Marc Marty, Guillaume Wenzek, and Amine Benhalloum. 2015. Trans-gram, fast cross-lingual word-embeddings. In Proc. of EMNLP. find
  7. Chris Dyer, Victor Chahuneau, and Noah A Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proc. of NAACL. find
  8. Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In Proc. of EACL. our card
  9. Lev Finkelstein, Evgeniy Gabrilovich, Yossi Matias, Ehud Rivlin, Zach Solan, Gadi Wolfman, and Eytan Ruppin. 2001. Placing search in context: the concept revisited. In Proc. of WWW. find
  10. Yoav Freund and Robert E Schapire. 1999. Large margin classification using the perceptron algorithm. Machine learning, 37(3):277–296. find
  11. Stephan Gouws and Anders Søgaard. 2015. Simple task-specific bilingual word embeddings. In Proc. of NAACL. find
  12. Stephan Gouws, Yoshua Bengio, and Greg Corrado. 2015. Bilbowa: Fast bilingual distributed representations without word alignments. In Proc. of ICML. find
  13. Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2015. Cross-lingual dependency parsing based on distributed representations. In Proc. of ACL. our card
  14. Jiang Guo, Wanxiang Che, David Yarowsky, Haifeng Wang, and Ting Liu. 2016. A representation learning framework for multi-source transfer parsing. In Proc. of AAAI. find
  15. David R Hardoon, Sandor Szedmak, and John Shawe-Taylor. 2004. Canonical correlation analysis: An overview with application to learning methods. Neural computation, 16(12):2639–2664. find
  16. Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual Models for Compositional Distributional Semantics. In Proc. of ACL. find
  17. Felix Hill, Roi Reichart, and Anna Korhonen. 2014. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. arXiv preprint arXiv:1408.3456. our card
  18. Harold Hotelling. 1936. Relations between two sets of variates. Biometrika, 28(3/4):321–377. find
  19. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proc. of COLING. our card
  20. Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Proc. of MT Summit. find
  21. Ang Lu, Weiran Wang, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2015. Deep multilingual correlation for improved word embeddings. In Proc. of NAACL. find
  22. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Bilingual word representations with monolingual quality in mind. In Proc. of the Workshop on Vector Space Modeling for NLP. our card
  23. Ryan McDonald, Joakim Nivre, Yvonne Quirmbach-Brundage, Yoav Goldberg, Dipanjan Das, Kuzman Ganchev, Keith Hall, Slav Petrov, Hao Zhang, Oscar Täckström, Claudia Bedini, Núria Bertomeu Castelló, and Jungmee Lee. 2013. Universal dependency annotation for multilingual parsing. In Proc. of ACL. our card
  24. Quinn McNemar. 1947. Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2):153–157. find
  25. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. our card
  26. Tomas Mikolov, Quoc V Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation. arXiv preprint arXiv:1309.4168. find
  27. Jerome L. Myers and Arnold D. Well. 1995. Research Design & Statistical Analysis. Routledge. find
  28. Pushpendre Rastogi, Benjamin Van Durme, and Raman Arora. 2015. Multiview LSA: Representation learning via generalized CCA. In Proceedings of NAACL. find
  29. Gerard Salton and Christopher Buckley. 1988. Term-weighting approaches in automatic text retrieval. Information Processing and Management. find
  30. Tobias Schnabel, Igor Labutov, David Mimno, and Thorsten Joachims. 2015. Evaluation methods for unsupervised word embeddings. In Proc. of EMNLP. our card
  31. Anders Søgaard, Željko Agić, Héctor Martı́nez Alonso, Barbara Plank, Bernd Bohnet, and Anders Johannsen. 2015. Inverted indexing for cross-lingual nlp. In Proc. of ACL. find
  32. James H Steiger. 1980. Tests for comparing elements of a correlation matrix. Psychological bulletin, 87(2):245. find
  33. O. Täckström, R. McDonald, and J. Uszkoreit. 2012. Cross-lingual word clusters for direct transfer of linguistic structure. In NAACL. our card
  34. Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky, and Christopher Manning. 2005. A conditional random field word segmenter for sighan bake-off 2005. In Proc. of SIGHAN. find
  35. Yulia Tsvetkov, Manaal Faruqui, Wang Ling, Guillaume Lample, and Chris Dyer. 2015. Evaluation of word vector representations by subspace alignment. In Proc. of EMNLP. find
  36. Ivan Vulić and Marie-Francine Moens. 2013a. Cross-lingual semantic similarity of words as the similarity of their semantic word responses. In Proc. of NAACL. find
  37. Ivan Vulić and Marie-Francine Moens. 2013b. A study on bootstrapping bilingual vector spaces from non-parallel data (and nothing else). In Proc. of EMNLP. find
  38. Ivan Vulić and Marie-Francine Moens. 2015. Bilingual word embeddings from non-parallel document-aligned data applied to bilingual lexicon induction. In Proc. of ACL. find
  39. Will Y. Zou, Richard Socher, Daniel Cer, and Christopher D. Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proc. of EMNLP. our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/upadhyay-2016-cross-lingual-models-word-embeddings/