Elamur Research
☰
home / papers / 2015 / Bilingual Word Representations with Monolingual Quality in Mind

Bilingual Word Representations with Monolingual Quality in Mind

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/W15-1521.pdf

Summary

Earlier work on bilingual word embeddings mostly optimized for cross-lingual tasks, especially cross-lingual document classification (CLDC). This often came at the cost of monolingual quality: neighbourhoods of similar words within each language were poorly preserved. Luong, Pham and Manning show that both goals can be met at once.

They propose BiSkip, a joint objective that learns German and English vectors from scratch. It combines two monolingual skipgram terms with a bilingual term of the same kind. Using word alignments, a word predicts the context words around its translation in the other language, in addition to its own neighbours. Training effectively runs four skipgram directions over one shared vocabulary. It is built on word2vec with negative sampling and trained on Europarl v7.

The resulting embeddings set a new best CLDC score for German→English and remain competitive for English→German. They also beat earlier bilingual embeddings by a wide margin on every monolingual word similarity dataset tested. The authors argue that the key is keeping the bilingual constraint architecturally consistent with the monolingual models.

Key points

  • Results: CLDC accuracy of 80.3 for de→en, a new state of the art, and 90.7 for en→de.
  • Data: Europarl v7 de–en with 1.9M sentence pairs (about 49.7M English and 52.0M German tokens).
    • Preprocessing: digits mapped to 0 and words seen fewer than 5 times replaced with <unk>.
    • Final vocabularies: 40K English and 95K German.
  • Two variants:
    • BiSkip-UnsupAlign uses Berkeley aligner links.
    • BiSkip-MonoAlign assumes monotonic, length-proportional alignment.
    • Both cluster words similarly within each language, but real alignments give better translations.
  • Training setup (SG-NS): learning rate 0.025, 30 negative samples, window size 5, subsampling 1e-4, 10 epochs. Each word's final embedding is obtained by adding together the vector it has as an input word and the vector it has as an output word.
  • Availability: code, data and embeddings are released at stanford.edu/~lmthang/bivec. A t-SNE plot of finance vocabulary shows translation pairs (e.g., managers/managern) and related German compounds placed close together.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

136 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 129 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/w15-1521

@inproceedings{luong2015bilingual,
  title = {{Bilingual Word Representations with Monolingual Quality in Mind}},
  author = {Luong, Thang and Pham, Hieu and Manning, Christopher D.},
  booktitle = {Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing},
  year = {2015},
  doi = {10.3115/v1/w15-1521},
  url = {https://doi.org/10.3115/v1/w15-1521}
}

References

45 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin. 2003. A neural probabilistic language model. JMLR, 3:1137–1155. find
  2. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. JMLR, 3:993– 1022. find our card
  3. Jordan Boyd-Graber and David M. Blei. 2009. Multilingual topic models for unaligned text. In UAI. find
  4. Peter F. Brown, Peter V. deSouza, Robert L. Mercer, Vincent J. Della Pietra, and Jenifer C. Lai. 1992. Class-based n-gram models of natural language. CL, 18(4):467–479. find
  5. Peter F. Brown, Vincent J. Della Pietra, Stephen A. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. CL, 19(2):263–311, June. find
  6. Sarath Chandar A P, Stanislas Lauly, Hugo Larochelle, Mitesh Khapra, Balaraman Ravindran, Vikas C Raykar, and Amrita Saha. 2014. An autoencoder approach to learning bilingual word representations. In NIPS. find
  7. Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In EMNLP. find our card
  8. R. Collobert and J. Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML. find
  9. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. 2011. Natural language processing (almost) from scratch. JMLR, 12:2493–2537. find our card
  10. S. T. Dumais, G. W. Furnas, T. K. Landauer, S. Deerwester, and R. Harshman. 1988. Using latent semantic analysis to improve access to textual information. In CHI. find
  11. Manaal Faruqui and Chris Dyer. 2014. Improving vector space word representations using multilingual correlation. In EACL. find
  12. Stephan Gouws, Yoshua Bengio, and Greg Corrado. 2014. Bilbowa: Fast bilingual distributed representations without word alignments. In NIPS Deep Learning Workshop. find
  13. Spence Green, Nicholas Andrews, Matthew R. Gormley, Mark Dredze, and Christopher D. Manning. 2012. Entity clustering across languages. In NAACL. find
  14. Michael Gutmann and Aapo Hyvärinen. 2012. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. JMLR, 13:307–361. find
  15. Aria Haghighi, Percy Liang, Taylor Berg-Kirkpatrick, and Dan Klein. 2008. Learning bilingual lexicons from monolingual corpora. In ACL. find
  16. Karl Moritz Hermann and Phil Blunsom. 2014. Multilingual Models for Compositional Distributional Semantics. In ACL. find
  17. E. H. Huang, R. Socher, C. D. Manning, and A. Y. Ng. 2012. Improving word representations via global context and multiple word prototypes. In ACL. find
  18. Alexandre Klementiev, Ivan Titov, and Binod Bhattarai. 2012. Inducing crosslingual distributed representations of words. In COLING. find
  19. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In MT Summit. find
  20. Tomáš Kočiský, Karl Moritz Hermann, and Phil Blunsom. 2014. Learning Bilingual Word Representations by Marginalizing Alignments. In ACL. find
  21. P. Liang, B. Taskar, and D. Klein. 2006. Alignment by agreement. In NAACL. find
  22. Minh-Thang Luong, Richard Socher, and Christopher D. Manning. 2013. Better word representations with recursive neural networks for morphology. In CoNLL. find our card
  23. Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In NAACL-HLT. find our card
  24. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Interspeech. find
  25. Tomas Mikolov, Stefan Kombrink, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. 2011. Extensions of recurrent neural network language model. In ICASSP. find
  26. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In ICLR. find
  27. Tomas Mikolov, Quoc V. Le, and Ilya Sutskever. 2013b. Exploiting similarities among languages for machine translation. CoRR, abs/1309.4168. find
  28. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013c. Distributed representations of words and phrases and their compositionality. In NIPS. find
  29. David Mimno, Hanna M. Wallach, Jason Naradowsky, David A. Smith, and Andrew McCallum. 2009. Polylingual topic models. In EMNLP. find
  30. Andriy Mnih and Geoffrey Hinton. 2009. A scalable hierarchical distributed language model. In NIPS. find
  31. Frederic Morin. 2005. Hierarchical probabilistic neural network language model. In AISTATS. find
  32. Xiaochuan Ni, Jian-Tao Sun, Jian Hu, and Zheng Chen. 2009. Mining multilingual topics from wikipedia. In WWW. find
  33. Yves Peirsman and Sebastian Padó. 2010. Cross-lingual induction of selectional preferences with bilingual vector spaces. In NAACL-HLT. find
  34. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP. find
  35. Nick Ruiz and Marcello Federico. 2011. Topic adaptation for lecture translation through bilingual latent semantic models. In WMT. find
  36. Richard Socher, John Bauer, Christopher D. Manning, and Andrew Y. Ng. 2013a. Parsing With Compositional Vector Grammars. In ACL. find
  37. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013b. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP. find
  38. Yik-Cheung Tam and Tanja Schultz. 2007. Bilingual LSA-based translation lexicon adaptation for spoken language translation. In Interspeech. find
  39. J. Turian, L. Ratinov, and Y. Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In ACL. find our card
  40. Ivan Vulić and Marie-Francine Moens. 2013. A study on bootstrapping bilingual vector spaces from non-parallel data (and nothing else). In EMNLP. find
  41. Ivan Vulic, Wim De Smet, Marie-Francine Moens, and KU Leuven. 2011. Identifying word translations from comparable corpora using latent topic models. In ACL-HLT. find
  42. Mengqiu Wang, Wanxiang Che, and Christopher D. Manning. 2013. Joint word alignment and bilingual named entity recognition using dual decomposition. In ACL. find
  43. David Yarowsky and Grace Ngai. 2001. Inducing multilingual pos taggers and np bracketers via robust projection across aligned corpora. In NAACL. find
  44. Bing Zhao and Eric P. Xing. 2007. HM-BiTAM: Bilingual Topic Exploration, Word Alignment, and Translation. In NIPS. find
  45. Will Y. Zou, Richard Socher, Daniel Cer, and Christopher D. Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In EMNLP. find