Elamur Research⌕
☰
home / papers / 2014 / Fast and Robust Neural Network Joint Models for Statistical Machine Translation

Fast and Robust Neural Network Joint Models for Statistical Machine Translation

Summary

The paper tackles a long-standing gap in statistical MT. Neural language models helped translation, but they were too slow to run inside the decoder and usually ignored the source sentence. The authors propose a neural network joint model (NNJM): a feed-forward network in the style of Bengio et al. (2003) that predicts each target word from its three preceding target words plus an 11-word source window. The window is centred on an "affiliated" source word, chosen from the word alignment by a simple heuristic. The model is trained on 10–30M words of parallel data, uses only words, and plugs into a Hiero-style decoder as an ordinary feature.

Two tricks make decoding cheap:

  • Self-normalization: the network learns to produce scores that need no softmax.
  • Precomputation: the first hidden layer is precomputed per word and position.

Together they give roughly a 10,000x speedup, putting the cost near that of a back-off LM.

On NIST OpenMT12 Arabic-English, the features add +3.0 BLEU over a strong baseline that already includes an NNLM, and +6.3 BLEU over a basic Hiero system. This shows that bilingual neural features pay off most when used directly in decoding rather than only for rescoring.

Key points

  • Architecture: a 14-word input (n = 4, m = 11), 192-dimensional shared embeddings, and two tanh hidden layers of 512 units. The vocabulary has 16k source and 16k target words on the input side and 32k target words on the output side. Out-of-vocabulary words are replaced by their POS tag.
  • Training: about 12 hours on a single Tesla K10 GPU. Weight updates are clipped to [-0.1, 0.1], and the learning rate is halved whenever validation likelihood worsens.
  • Self-normalization: an α-weighted penalty on log²Z is added to the objective. With α = 0.1, held-out log-likelihood barely changes (−1.83 vs −1.82). The mean |log Z| error drops from 5.02 to 0.68. Lookups become about 15x faster.
  • Hidden-layer precomputation: this gives a further ~1000x speedup, leaving about 3,500 operations per lookup instead of about 35M for the standard model. It requires a single hidden layer, which costs little BLEU.
  • Results: gains of +3.0 BLEU (strong baseline) and +6.3 BLEU (Hiero-like baseline) on OpenMT12 Arabic-English. Improvements also hold for Chinese-English and the DARPA BOLT conditions. A single S2T/L2R NNJM already yields 50–80% of the total gain.

Interesting ideas

  • I1. Self-normalization through a penalty term: adding a squared log-normalizer penalty to training lets the model's raw output score stand in for a probability at test time. The α parameter gives explicit control over the trade-off between accuracy and normalization error. This could help any large-vocabulary scorer that runs inside a search loop.
  • I2. Caching embedding-times-weights products: with a fixed number of input positions, each word's contribution to the first hidden layer can be stored offline in a lookup table. Inference then reduces mostly to additions. The trick suits feed-forward models with a fixed window, such as taggers or scorers for low-resource Ukrainian tools.
  • I3. One-to-one affiliation heuristic: every target word is forced to link to exactly one source word, using the middle of several aligned words or inheriting from a neighbour if unaligned. This turns messy alignments into a clean, fixed-size bilingual context without hand-crafted phrase segmentation.
  • I4. Integrate into decoding rather than rescoring: the authors argue that k-best rescoring is too shallow to exploit a strong joint model. When evaluating a new feature, it is worth testing it inside the search and not only on reranked outputs.
  • I5. Exploit duplicate requests: about 95% of lookups within a sentence repeat earlier ones. Caching results, and sharing n-gram construction with the back-off LM, cuts the real decoding overhead, which is a general lesson for deploying neural scorers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

189 papers of our library cite it (of the 140884 whose bibliographies we read); 14 of them have a page here, 175 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p14-1129

@inproceedings{devlin2014fast,
  title = {{Fast and Robust Neural Network Joint Models for Statistical Machine Translation}},
  author = {Devlin, Jacob and Zbib, Rabih and Huang, Zhongqiang and Lamar, Thomas and Schwartz, Richard and Makhoul, John},
  booktitle = {Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2014},
  doi = {10.3115/v1/p14-1129},
  url = {https://doi.org/10.3115/v1/p14-1129}
}

References

29 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Auli, Michel Galley, Chris Quirk, and Geoffrey Zweig. 2013. Joint language and translation modeling with recurrent neural networks. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1044– 1054, Seattle, Washington, USA, October. Association for… find
  2. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155. find
  3. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent dirichlet allocation. J. Mach. Learn. Res., 3:993–1022, March. find our card
  4. David Chiang, Kevin Knight, and Wei Wang. 2009. 11,001 new features for statistical machine translation. In HLT-NAACL, pages 218–226. find
  5. David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics, 33(2):201–228. find our card
  6. Josep Maria Crego and François Yvon. 2010. Factored bilingual n-gram language models for statistical machine translation. Machine Translation, 24(2):159– 175. find
  7. Jacob Devlin and Spyros Matsoukas. 2012. Trait-based hypothesis selection for machine translation. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT ’12, pages 528–532, Stroudsburg, PA, USA. As… find
  8. Jacob Devlin. 2009. Lexical features for statistical machine translation. Master’s thesis, University of Maryland. find
  9. Nizar Habash, Ryan Roth, Owen Rambow, Ramy Eskander, and Nadi Tomeh. 2013. Morphological analysis and disambiguation for dialectal arabic. In HLT-NAACL, pages 426–432. find
  10. Zhongqiang Huang, Jacob Devlin, and Rabih Zbib. 2013. Factored soft source syntactic constraints for hierarchical machine translation. In EMNLP, pages 556–566. find
  11. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. find our card
  12. Reinhard Kneser and Hermann Ney. 1995. Improved backing-off for m-gram language modeling. In Acoustics, Speech, and Signal Processing, 1995. ICASSP-95., 1995 International Conference on, volume 1, pages 181–184. IEEE. find
  13. Hai-Son Le, Alexandre Allauzen, and François Yvon. 2012. Continuous space translation models with neural networks. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT ’12, pages 39– 48, Stroudsb… find
  14. Yann LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. 1998. Efficient backprop. In Neural networks: Tricks of the trade, pages 9–50. Springer. find
  15. José B Marino, Rafael E Banchs, Josep M Crego, Adrià De Gispert, Patrik Lambert, José AR Fonollosa, and Marta R Costa-Jussà. 2006. N-gram-based machine translation. Computational Linguistics, 32(4):527– 549. find our card
  16. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In IN-TERSPEECH, pages 1045–1048. find
  17. Tomas Mikolov, Wen tau Yih, and Geoffrey Zweig. 2013. Linguistic regularities in continuous space word representations. In HLT-NAACL, pages 746– 751. find our card
  18. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. find
  19. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063. find
  20. Jason Riesa, Ann Irvine, and Daniel Marcu. 2011. Feature-rich language-independent syntax-based alignment for statistical machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’11, pages 497–507, Stroudsburg, PA, USA. Association for Computa… find
  21. Ronald Rosenfeld. 1996. A maximum entropy approach to adaptive statistical language modeling. Computer, Speech and Language, 10:187–228. find
  22. Antti Rosti, Bing Zhang, Spyros Matsoukas, and Rich Schwartz. 2010. BBN system description for WMT10 system combination task. In WMT/MetricsMATR, pages 321–326. find
  23. Holger Schwenk. 2010. Continuous-space language models for statistical machine translation. Prague Bull. Math. Linguistics, 93:137–146. find
  24. Holger Schwenk. 2012. Continuous space translation models for phrase-based statistical machine translation. In COLING (Posters), pages 1071–1080. find
  25. Libin Shen, Jinxi Xu, and Ralph Weischedel. 2010. String-to-dependency statistical machine translation. Computational Linguistics, 36(4):649–671, December. find
  26. Matthew Snover, Bonnie Dorr, and Richard Schwartz. 2008. Language and translation model adaptation using comparable corpora. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 857–866, Stroudsburg, PA, USA. Association for Computational Linguistics… find
  27. Makoto Tanaka, Yasuhara Toru, Jun-ya Yamamoto, and Mikio Norimatsu. 2013. An efficient language model using double-array structures. find
  28. Ashish Vaswani, Yinggong Zhao, Victoria Fossum, and David Chiang. 2013. Decoding with large-scale neural language models improves translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1387–1392, Seattle, Washington, USA, October. Association fo… find our card
  29. Will Y Zou, Richard Socher, Daniel Cer, and Christopher D Manning. 2013. Bilingual word embeddings for phrase-based machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1393–1398. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/devlin-2014-fast-robust-neural-network-joint/