Elamur Research⌕
☰
home / papers / 2012 / Inducing Crosslingual Distributed Representations of Words

Inducing Crosslingual Distributed Representations of Words

Summary

The paper addresses how to build word representations that work across two languages, so that a model trained with labels in one language can be applied directly to another language that has no annotation. The authors train a neural language model, a variant of Bengio et al. (2003), for each language. They couple the two models through a multitask learning framework adapted from Cavallanti et al. (2010). Each word in either vocabulary is a separate task. Words that tend to be translations of each other, judged from word alignments in parallel data (e.g. produced with GIZA++), are linked as related tasks. This relatedness is encoded in an interaction matrix, so the shared embedding space is aligned while both language models are being trained. The authors test the resulting 40-dimensional embeddings on English–German crosslingual document classification with no adaptation to the target language. Classifiers built on these embeddings clearly beat a machine-translation baseline and a word-glossing baseline. The work matters because it gives a general, task-independent way to transfer supervised NLP models to lower-resourced languages.

Key points

  • Interaction matrix: task relatedness is defined as A = I + L, where L is the Laplacian of a weighted graph whose edges link words likely to be mutual translations. The edge weights come from alignment statistics over bitext.

  • Base model: a feed-forward n-gram neural language model with a shared embedding table, a logistic hidden layer and a softmax output, trained by SGD. Input and output vocabularies are kept separate to speed up training.

  • Results with 1000 labeled training documents:

    DirectionDistribRepsMTGlossedMajority class
    en → de77.6%68.1%65.1%46.8%
    de → en71.1%67.4%68.6%46.8%

  • Small training sets: the gains are largest when labeled data is scarce. About 100 labeled examples already gave a good classifier, and accuracy changed little as more labels were added. The authors attribute this to the low embedding size (d = 40).

  • Positioning: the closest prior work is crosslingual Brown clustering (Täckström et al., 2012). The authors argue that dense vectors can encode several properties of a word at once, which a single atomic cluster label cannot.

Interesting ideas

  • I1. Words as related tasks. Treating each vocabulary item as a task and coupling tasks through a graph turns embedding alignment into a regularizer. Any external similarity source could define that graph, including dictionaries, transliteration pairs or gazetteers, not just bitext.
  • I2. Graph Laplacian as prior knowledge. Using identity plus a Laplacian gives an invertible, positive-definite coupling matrix from any weighted graph. This is a cheap and principled way to inject soft "these items should behave alike" constraints into many joint-learning setups.
  • I3. Align during training, not after. The two monolingual models are aligned as they are trained, rather than trained separately and mapped afterwards. The monolingual models can therefore exploit abundant unlabeled text while the parallel data only steers them.
  • I4. Direct transfer as an intrinsic test. Training a simple classifier in one language and testing it untouched in another is an easy evaluation for any shared multilingual representation. Comparing against MT and glossing baselines makes the result interpretable.
  • I5. Separate input and output vocabularies. Restricting the softmax output vocabulary while keeping a larger input vocabulary is a practical speed trick for training neural language models on limited compute.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

159 papers of our library cite it (of the 140884 whose bibliographies we read); 21 of them have a page here, 138 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{klementiev2012inducing,
  title = {{Inducing Crosslingual Distributed Representations of Words}},
  author = {Klementiev, Alexandre and Titov, Ivan and Bhattarai, Binod},
  booktitle = {Proceedings of COLING 2012},
  year = {2012},
  url = {https://aclanthology.org/C12-1089/}
}

References

28 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agirre, E., Alfonseca, E., Hall, K., Kravalova, J., Pasca, M., and Soroa, A. (2009). A study on similarity and relatedness using distributional and wordnet-based approaches. In Proc. of the Annual Meeting of the North American Association of Computational Linguistics (NAACL), pages 19–27, Boulder, C… find our card
  2. Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. (2003). A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
  3. Callison-Burch, C., Koehn, P., Monz, C., Post, M., Soricut, R., and Specia, L. (2012). Findings of the 2012 workshop on statistical machine translation. In Proc. of the Workshop on Statistical Machine Translation (WMT), pages 10–51, Montréal, Canada. Association for Computational Linguistics. find our card
  4. Cavallanti, G., Cesa-bianchi, N., and Gentile, C. (2010). Linear algorithms for online multitask classification. Journal of Machine Learning Research, 11:2901–2934. find
  5. Collins, M. (2002). Discriminative training methods for hidden markov models: Theory and experiments with perceptron algorithms. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  6. Collobert, R. and Weston, J. (2007). Fast semantic extraction using a novel neural network architecture. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 560–567. find
  7. Fortuna, B. and Shawe-Taylor, J. (2005). The use of machine translation tools for cross-lingual text mining. In Proc. of the Workshop on Learning with Multiple Views, 22nd ICML. find
  8. Fouss, F., Pirotte, A., Renders, J., and Saerens, M. (2007). Random-walk computation of similarities between nodes of a graph with application to collaborative recommendation. Knowledge and Data Engineering, IEEE Transactions on, 19(3):355–369. find
  9. Frederking, R., Hovy, E., and Ide, N. (2001). Special issue on multi-lingual information management. Computer and the Humanities, 35. find
  10. Glorot, X., Bordes, A., and Bengio, Y. (2011). Domain adaptation for large-scale sentiment classification: A deep learning approach. In Getoor, L. and Scheffer, T., editors, Proc. of the International Conference on Machine Learning (ICML), pages 513–520, New York, NY, USA. ACM. find
  11. Huang, E., Socher, R., Manning, C., and Ng, A. (2012). Improving word representations via global context and multiple word prototypes. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL). find our card
  12. Klementiev, A., Irvine, A., Callison-Burch, C., and Yarowsky, D. (2012). Toward statistical machine translation without parallel corpora. In Proc. of the Meeting of the European Association of Computational Linguistics (EACL). find
  13. Koehn, P. (2005). Europarl: A parallel corpus for statistical machine translation. In Proc. of the Machine Translation Summit. find
  14. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source toolkit for statistical machine translation. In Proc. of the ACL-2007 Demo and Poster Sessions. find
  15. Lavrenko, V., Choquette, M., and Croft, W. B. (2002). Cross-lingual relevance models. In Proc. of International Conference on Research and Development in Information Retrieval (SIGIR), pages 175–182. find
  16. Lewis, D., Yang, Y., Rose, T., and Li, F. (2004). RCV1: A new benchmark co . in: Sekine, s. and ranchhod, e. named entities: Recognition, classification and use. special issue of lingvisticæ investigationes. 30(1) pp.135-162. ollection for text categorization research. Journal of Machine Learning Re… find
  17. Li, S. (2009). Fast Algorithms for Sparse Matrix Inversion. PhD thesis, Stanford University. find
  18. Mimno, D., Wallach, H. M., Naradowsky, J., Smith, D. A., and McCallum, A. (2009). Polylingual topic models. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 880–889, Singapore. Association for Computational Linguistics. find
  19. Och, F. and Ney, H. (2003). A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. find
  20. Rapp, R. (1995). Identifying word translations in non-parallel texts. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 320–322. find our card
  21. Shi, L., Mihalcea, R., and Tian, M. (2010). Cross language text classification by model translation and semi-supervised learning. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1057–1067, Cambridge, MA. Association for Computational Linguistics. find
  22. Socher, R., Huang, E. H., Pennin, J., Ng, A. Y., and Manning, C. D. (2011a). Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Proc. of the Conference on Advances in Neural Information Processing Systems (NIPS), pages 801–809. find
  23. Socher, R., Pennington, J., Huang, E. H., Ng, A. Y., and Manning, C. D. (2011b). Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proc. of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 151–161, Stroudsburg, PA, USA. Association for… find our card
  24. Steinberger, R. and Pouliquen, B. (2007). Cross-lingual named entity recognition. Lingvistica Investigationes, 30:135–162. find
  25. Täckström, O., McDonald, R., and Uszkoreit, J. (2012). Cross-lingual word clusters for direct transfer of linguistic structure. In Proc. of the Annual Meeting of the North American Association of Computational Linguistics (NAACL), pages 477–487, Montréal, Canada. find our card
  26. Titov, I. (2011). Domain adaptation by constraining inter-domain variability of latent feature representation. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 62–71, Portland, Oregon, USA. Association for Computational Linguistics. find
  27. Turian, J., Ratinov, L., and Bengio, Y. (2010). Word representations: a simple and general method for semi-supervised learning. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 384–394. Association for Computational Linguistics. van der Plas, L. and Tiedem… find our card
  28. Zhang, D., Mei, Q., and Zhai, C. (2010). Cross-lingual latent topic extraction. In Proc. of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 1128–1137, Uppsala, Sweden. Association for Computational Linguistics. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/klementiev-2012-inducing-crosslingual-distributed-representations-words/