Elamur Research⌕
☰
home / papers / 2013 / Bilingual Word Embeddings for Phrase-Based Machine Translation

Bilingual Word Embeddings for Phrase-Based Machine Translation

Summary

Statistical MT systems struggle to recognise that two phrases mean the same thing when their exact word combinations rarely co-occur in parallel data. Zou et al. address this by learning Chinese and English word embeddings that share one semantic space. Chinese vectors are first initialised as alignment-weighted mixtures of existing English embeddings (Huang et al., 2012), using Berkeley Aligner counts. Training then combines a Collobert-style contrastive context objective, with global document context, and a translation-equivalence penalty. That penalty pulls each language's vectors toward the alignment-weighted average of the other language's vectors. Training uses curriculum learning over a 100k-word vocabulary on Chinese Gigaword. The resulting embeddings beat tf-idf baselines and prior work on SemEval-2012 Chinese word similarity. They also align words better than monolingually trained vectors. Used as a single phrase-pair similarity feature in Stanford Phrasal, they lift NIST08 Chinese-English BLEU from 30.01 to 30.49. The work matters because it shows that cross-lingual distributed semantics can be added cheaply to a conventional MT pipeline, and it released early Mandarin embeddings.

Key points

  • Training objective and data: Chinese embeddings minimise the context objective plus λ·J_TEO, with λ = 50. Vocabulary bands of {5k, 10k, 25k, 50k} were trained with minibatch L-BFGS on Chinese Gigaword (Fifth Edition, LDC2011T13). This took about 19 days on an eight-core machine.

  • SemEval-2012 Task 4 (297 word pairs), Spearman ×100: pruned tf-idf scores 46.7, Align-Init 52.9, Mono-trained 59.3 and Biling-trained 60.8. Biling-trained also has the best Kendall's Tau ×100, at 43.3.

  • OntoNotes v4.0 NER (simple feed-forward network with a 5-word window and 2000 hidden units), F1:

    • Align-Init: 0.41
    • Biling-trained: 0.52
    • Mono-trained: 0.58

    Bilingual training therefore improves on its initialisation but does not beat monolingual training on this task.

  • Vector-matching alignment (cosine matching on 10,000 sentence pairs): AER drops from 0.71 for monolingual training to 0.59 for bilingual training.

  • NIST08 BLEU:

    • Baseline: 30.01
    • Random-Init + mono: 30.09
    • Align-Init: 30.31
    • Align-Init + mono: 30.40
    • Biling-trained: 30.49

    The authors call the gain modest, since it only slightly exceeds a reference standard deviation of 0.29 BLEU.

Interesting ideas

  • I1. Alignment-weighted initialisation: Each low-resource word's vector starts as a smoothed, alignment-count-weighted average of a high-resource language's embeddings. This gives a usable cross-lingual space before any training and could bootstrap embeddings for a language pair with good parallel data, such as Ukrainian-English.
  • I2. Alignment-matrix regulariser: A row-normalised alignment matrix serves as a soft "translation" operator. The penalty ‖V_zh − A·V_en‖² keeps the two vocabularies tied while monolingual training continues. It is a simple, differentiable way to inject dictionary-like supervision into any embedding objective.
  • I3. Embedding-based alignment as an evaluation: Naive word alignment by cosine similarity, scored with AER, works as an intrinsic test of cross-lingual equivalence. It reuses existing alignment tooling and does not require human similarity judgements.
  • I4. Cheap phrase-pair similarity feature: Averaging the word vectors of each side of a phrase pair, ignoring OOV words and using a zero vector if nothing is known, then taking the cosine gives one dense feature a decoder can weigh. This lightweight pattern applies to any phrase table or candidate-ranking system.
  • I5. Averaging over late checkpoints: To handle noise from online minibatch optimisation, the authors evaluate embeddings from several random late checkpoints and average the scores. This is a sensible habit for reporting more stable intrinsic metrics.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

162 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 145 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zou2013bilingual,
  title = {{Bilingual Word Embeddings for Phrase-Based Machine Translation}},
  author = {Zou, Will Y. and Socher, Richard and Cer, Daniel and Manning, Christopher D.},
  booktitle = {Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  doi = {10.18653/v1/d13-1141},
  url = {https://doi.org/10.18653/v1/d13-1141}
}

References

42 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Klementiev, I. Titov and B. Bhattarai. 2012. Inducing Crosslingual Distributed Representation of Words. COLING. find
  2. Y. Bengio, J. Louradour, R. Collobert and J. Weston. 2009. Curriculum Learning. ICML. find our card
  3. Y. Bengio, R. Ducharme, P. Vincent and C. Jauvin. 2003. A Neural Probabilistic Language Model. Journal of Machine Learning Research. find
  4. Y. Bengio and Y. LeCunn. 2007. Scaling learning algorithms towards AI. Large-Scale Kernal Machines. find
  5. J. Boyd-Graber and P. Resnik. 2010. Holistic sentiment analysis across languages: multilingual supervised latent dirichlet allocation. EMNLP. find
  6. D. Cer, M. Galley, D. Jurafsky and C. Manning. 2010. Phrasal: A Toolkit for Statistical Machine Translation with Facilities for Extraction and Incorporation of Arbitrary Model Features. In Proceedings of the North American Association of Computational Linguistics - Demo Session (NAACL-10). find
  7. D. Cer, C. Manning and D. Jurafsky. 2010. The Best Lexical Metric for Phrase-Based Statistical MT System Optimization. NAACL. find
  8. R. Collobert and J. Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. ICML. find
  9. G. Foster and R. Kuhn. 2009. Stabilizing minimum error rate training. Proceedings of the Fourth Workshop on Statistical Machine Translation. find
  10. M. Galley, P. Chang, D. Cer, J. R. Finkel and C. D. Manning. 2008. find
  11. NIST Open Machine Translation 2008 Evaluation: Stanford University’s System Description. Unpublished working notes of the 2008 NIST Open Machine Translation Evaluation Workshop. find
  12. S. Green, S. Wang, D. Cer and C. Manning. 2013. Fast and adaptive online training of feature-rich translation models. ACL. find
  13. G. Hinton, L. Deng, D. Yu, G. Dahl, A. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. Sainath and B. Kingsbury. 2012. Deep Neural Networks for Acoustic Modeling in Speech Recognition. IEEE Signal Processing Magazine. find
  14. E. Hovy, M. Marcus, M. Palmer, L. Ramshaw and R. Weischedel. 2006. OntoNotes: the 90% solution. NAACL-HLT. find
  15. E. H. Huang, R. Socher, C. D. Manning and A. Y. Ng. 2012. Improving Word Representations via Global Context and Multiple Word Prototypes. ACL. find our card
  16. P. Jin and Y. Wu. 2012. SemEval-2012 Task 4: Evaluating Chinese Word Similarity. Proceedings of the First Joint Conference on Lexical and Computational Semantics-Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semanti… find
  17. R. Jones. 2006. Generating query substitutions. In Proceedings of the 15th international conference on World Wide Web. find
  18. P. Koehn, F. J. Och and D. Marcu. 2003. Statistical Phrase-Based Translation. HLT. find
  19. H. Le, A. Allauzen and F. Yvon 2012. Continuous space translation models with neural networks. NAACL. find
  20. P. Liang, B. Taskar and D. Klein. 2006. Alignment by agreement. NAACL. find our card
  21. M. Luong, R. Socher and C. Manning. 2013. Better word representations with recursive neural networks for morphology. CONLL. find our card
  22. L. van der Maaten and G. Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research. find
  23. A. Maas and R. E. Daly and P. T. Pham and D. Huang and A. Y. Ng and C. Potts. 2011. Learning word vectors for sentiment analysis. ACL. find our card
  24. C. Manning and P. Raghavan and H. Schtze. 2008. Introduction to Information Retrieval. Cambridge University Press, New York, NY, USA. find
  25. T. Mikolov, M. Karafiat, L. Burget, J. Cernocky and S. Khudanpur. 2010. Recurrent neural network based language model. INTERSPEECH. find
  26. T. Mikolov, K. Chen, G. Corrado and J. Dean. 2013. Efficient Estimation of Word Representations in Vector Space. arXiv:1301.3781v1. find
  27. A. Mnih and G. Hinton. 2008. A scalable hierarchical distributed language model. NIPS. find
  28. F. Morin and Y. Bengio. 2005. Hierarchical probabilistic neural network language model. AISTATS. find
  29. F. Och. 2003. Minimum error rate training in statistical machine translation. ACL. find
  30. M. Paşca, D. Lin, J. Bigham, A. Lifchits and A. Jain. 2006. Names and similarities on the web: fact extraction in the fast lane. ACL. find
  31. Y. Peirsman and S. Padó. 2010. Cross-lingual induction of selectional preferences with bilingual vector spaces. ACL. find
  32. J. Reisinger and R. J. Mooney. 2010. Multi-prototype vector-space models of word meaning. NAACL. find our card
  33. F. Sebastiani. 2002. Machine learning in automated text categorization. ACM Comput. Surv., 34:1-47, March. find
  34. R. Socher, J. Pennington, E. Huang, A. Y. Ng and C. D. Manning. 2011. Semi-Supervised Recursive Autoencoders for Predicting Sentiment Distributions. EMNLP. find our card
  35. R. Socher, E. H. Huang, J. Pennington, A. Y. Ng, and C. D. Manning. 2011. Dynamic Pooling and Unfolding Recursive Autoencoders for Paraphrase Detection. NIPS. find
  36. E. Sumita. 2000. Lexical transfer using a vector-space model. ACL. find
  37. Y. Tam, I. Lane and T. Schultz. 2007. Bilingual-LSA based LM adaptation for spoken language translation. ACL. find
  38. S. Tellex and B. Katz and J. Lin and A. Fernandes and G. Marton. 2003. Quantitative evaluation of passage retrieval algorithms for question answering. In Proceedings of the 26th Annual International ACM SIGIR Conference on Search and Development in Information Retrieval, pages 41-47. ACM Press. find
  39. J. Turian and L. Ratinov and Y. Bengio. 2010. Word representations: A simple and general method for semi-supervised learning. ACL. find our card
  40. M. Wang, W. Che and C. D. Manning. 2013. Joint Word Alignment and Bilingual Named Entity Recognition Using Dual Decomposition. ACL. find
  41. K. Yamada and K. Knight. 2001. A Syntax-based Statistical Translation Model. ACL. find our card
  42. B. Zhao and E. P. Xing 2006. BiTAM: Bilingual topic AdMixture Models for word alignment. ACL. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zou-2013-bilingual-word-embeddings-phrase-based/