Elamur Research⌕
☰
home / papers / 2014 / Multilingual Models for Compositional Distributed Semantics

Multilingual Models for Compositional Distributed Semantics

Summary

Most compositional semantic models depend on parse trees, word alignments, or task labels such as sentiment, which limits them to well-resourced languages. Hermann and Blunsom remove these requirements. They learn word embeddings and composed sentence and document vectors using only sentence-aligned parallel text. Each language gets its own compositional vector model (CVM). Training pulls the vectors of translation pairs together and uses a noise-contrastive hinge loss to push randomly sampled non-translations at least a margin apart. The authors test two syntax-free composition functions: ADD, a summed bag of words, and BI, which applies tanh over adjacent word pairs. They also stack a second CVM over sentence vectors to produce document representations. The models are trained on Europarl v7 and on a new multilingual corpus built from TED talks. On cross-lingual document classification with Reuters RCV1/RCV2, they outperform the best previously reported results, those of Klementiev et al. (2012). Qualitative analysis shows they also align languages that never appeared together in training. The method suits low-resource settings where only bitext is available.

Key points

  • The training signal is the squared Euclidean distance between the two sentence encodings, wrapped in a margin-based hinge loss over sampled negative sentences, plus L2 regularisation. No word alignment or syntax is used.
  • The four variants are ADD, BI, DOC/ADD and DOC/BI. The DOC versions add a document-level composition and error signal on top of the sentence-level one.
  • Europarl v7 (English-German and English-French, with the last 100,000 sentences of each pair held out for development) supplied the embeddings for the RCV1/RCV2 classification task.
  • The new TED-based corpus is derived from IWSLT 2013:
    • it has 12,078 parallel documents across 12 language pairs and about 1.68M non-English sentences;
    • talks with id ≥ 1,400 (roughly 10%) form the test set;
    • the 15 most frequent talk keywords serve as multi-label topic classes.
  • In t-SNE plots from the BI+ model trained on Europarl, French and German words and phrases cluster by meaning, even though no French-German parallel data was used. English acted as a pivot.

Interesting ideas

  • I1. Translation as free supervision. Treating a translation pair as two views of the same meaning gives a training signal without annotation. This could help build Ukrainian sentence embeddings from existing Ukrainian-English bitext alone.
  • I2. Contrastive margin against collapse. Minimising the distance between positives alone would let all vectors shrink to one point. Adding sampled negatives with a hinge margin prevents this cheaply, and the trick transfers to any paired-embedding setup.
  • I3. Hierarchical composition for documents. Composing sentence vectors through a second CVM, instead of averaging them, lets document-aligned but sentence-unaligned corpora still train the word vectors. This is useful when only loosely aligned parallel documents exist.
  • I4. Repurposing metadata as a benchmark. Using TED talk keywords as topic labels turned a machine-translation corpus into a many-language classification benchmark. Other parallel collections with tags could be converted the same way for evaluation.
  • I5. Checking pivot effects. Inspecting language pairs that never co-occurred in training (here French-German, linked through English) is a simple test of whether the space is truly shared. This matters when Ukrainian is connected to other languages mainly through English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

107 papers of our library cite it (of the 140884 whose bibliographies we read); 4 of them have a page here, 103 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p14-1006

@inproceedings{hermann2014multilingual,
  title = {{Multilingual Models for Compositional Distributed Semantics}},
  author = {Hermann, Karl Moritz and Blunsom, Phil},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-1006},
  url = {https://doi.org/10.3115/v1/p14-1006}
}

References

41 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Al-Rfou’, B. Perozzi, and S. Skiena. 2013. Polyglot: Distributed word representations for multilingual nlp. In Proceedings of CoNLL. find our card
  2. M. Baroni and R. Zamparelli. 2010. Nouns are vectors, adjectives are matrices: Representing adjective-noun constructions in semantic space. In Proceedings of EMNLP. find
  3. Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155, March. find
  4. D. M. Blei, A. Y. Ng, and M. I. Jordan. 2003. Latent dirichlet allocation. Journal of Machine Learning Research, 3:993–1022. find our card
  5. P. Bloom. 2001. Precis of how children learn the meanings of words. Behavioral and Brain Sciences, 24:1095–1103. find
  6. M. Cettolo, C. Girardi, and M. Federico. 2012. Wit 3 : Web inventory of transcribed and translated talks. In Proceedings of EAMT. find
  7. S. Clark and S. Pulman. 2007. Combining symbolic and distributional models of meaning. In Proceedings of AAAI Spring Symposium on Quantum Interaction. AAAI Press. find
  8. T. Cohn and M. Lapata. 2007. Machine translation by triangulation: Making effective use of multi-parallel corpora. In Proceedings of ACL. find
  9. M. Collins. 2002. Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In Proceedings of ACL-EMNLP. find our card
  10. R. Collobert and J. Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of ICML. find
  11. R. Collobert, J. Weston, L. Bottou, M. Karlen, K. Kavukcuoglu, and P. Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2493–2537. find our card
  12. J. Duchi, E. Hazan, and Y. Singer. 2011. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12:2121–2159, July. find
  13. S. T. Dumais, G. W. Furnas, T. K. Landauer, S. Deerwester, and R. Harshman. 1988. Using latent semantic analysis to improve access to textual information. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. find
  14. C. Dyer, A. Lopez, J. Ganitkevitch, J. Weese, F. Ture, P. Blunsom, H. Setiawan, V. Eidelman, and P. Resnik. 2010. cdec: A Decoder, Alignment, and Learning framework for finite-state and context-free translation models. In Proceedings of ACL. find our card
  15. K. Erk and S. Padó. 2008. A structured vector space model for word meaning in context. Proceedings of EMNLP. find our card
  16. J. R. Firth. 1957. A synopsis of linguistic theory 1930-55. 1952-59:1–32. find
  17. E. Grefenstette and M. Sadrzadeh. 2011. Experimental support for a categorical compositional distributional model of meaning. In Proceedings of EMNLP. find
  18. A. Haghighi, P. Liang, T. Berg-Kirkpatrick, and D. Klein. 2008. Learning bilingual lexicons from monolingual corpora. In Proceedings of ACL-HLT. find our card
  19. K. M. Hermann and P. Blunsom. 2013. The Role of Syntax in Vector Space Models of Compositional Semantics. In Proceedings of ACL. find
  20. K. M. Hermann and P. Blunsom. 2014. Multilingual Distributed Representations without Word Alignment. In Proceedings of ICLR. find
  21. N. Kalchbrenner and P. Blunsom. 2013. Recurrent convolutional neural networks for discourse compositionality. Proceedings of the ACL Workshop on Continuous Vector Space Models and their Compositionality. find our card
  22. A. Klementiev, I. Titov, and B. Bhattarai. 2012. Inducing crosslingual distributed representations of words. In Proceedings of COLING. find our card
  23. P. Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Proceedings of the Machine Translation Summit. find
  24. S. Lauly, A. Boulanger, and H. Larochelle. 2013. Learning multilingual word representations using a bag-of-words autoencoder. In Deep Learning Workshop at NIPS. find
  25. D. D. Lewis, Y. Yang, T. G. Rose, and F. Li. 2004. Rcv1: A new benchmark collection for text categorization research. Journal of Machine Learning Research, 5:361–397, December. find our card
  26. A. K. McCallum. 2002. Mallet: A machine learning for language toolkit. http://mallet.cs.umass.edu. ↗
  27. T. Mikolov, M. Karafiát, L. Burget, J. Černocký, and S. Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of INTER-SPEECH. find
  28. T. Mikolov, K. Chen, G. Corrado, and J. Dean. 2013a. Efficient Estimation of Word Representations in Vector Space. CoRR. find
  29. T. Mikolov, Q. V. Le, and I. Sutskever. 2013b. Exploiting Similarities among Languages for Machine Translation. CoRR. find
  30. J. Mitchell and M. Lapata. 2008. Vector-based models of semantic composition. In In Proceedings of ACL. find
  31. A. Mnih and G. Hinton. 2009. A scalable hierarchical distributed language model. In Proceedings of NIPS. find
  32. J. Ngiam, A. Khosla, M. Kim, J. Nam, H. Lee, and A. Y. Ng. 2011. Multimodal deep learning. In ICML. find
  33. D. Roy. 2003. Grounded spoken language acquisition: Experiments in word learning. IEEE Transactions on Multimedia, 5(2):197–209, June. find
  34. A. P. Sarath Chandar, M. K. Mitesh, B. Ravindran, V. Raykar, and A. Saha. 2013. Multilingual deep learning. In Deep Learning Workshop at NIPS. find
  35. R. Socher, J. Pennington, E. H. Huang, A. Y. Ng, and C. D. Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of EMNLP. find our card
  36. R. Socher, B. Huval, C. D. Manning, and A. Y. Ng. 2012. Semantic compositionality through recursive matrix-vector spaces. In Proceedings of EMNLP-CoNLL, pages 1201–1211. find our card
  37. N. Srivastava and R. Salakhutdinov. 2012. Multimodal learning with deep boltzmann machines. In Proceedings of NIPS. find
  38. J. Turian, L. Ratinov, and Y. Bengio. 2010. Word representations: a simple and general method for semi-supervised learning. In Proceedings of ACL. find our card
  39. P. D. Turney. 2012. Domain and function: A dual-space model of semantic relations and compositions. Journal of Artificial Intelligence Research, 44:533– 585. find
  40. W.-T. Yih, K. Toutanova, J. C. Platt, and C. Meek. 2011. Learning discriminative projections for text similarity measures. In Proceedings of CoNLL. find our card
  41. W. Y. Zou, R. Socher, D. Cer, and C. D. Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceedings of EMNLP. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hermann-2014-multilingual-models-compositional-distributed-semantics/