Elamur Research⌕
☰
home / papers / 2016 / Incorporating Structural Alignment Biases into an Attentional Neural Translation Model

Incorporating Structural Alignment Biases into an Attentional Neural Translation Model

Paper · 2016 · Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vymolova et al.

Summary

The paper addresses a gap in attentional neural machine translation (NMT). The attention mechanism of Bahdanau et al. (2015) plays a role similar to word alignment, but it lacks the structural priors that classic alignment models such as the IBM models and Vogel's HMM aligner (as in GIZA++) build in explicitly. The authors extend a bidirectional LSTM attentional encoder-decoder with four such biases:

  • Positional bias: position features are fed into the attention scorer.
  • Markov conditioning: a window of the previous step's attention weights informs the current attention.
  • Fertility: handled both through local features and through a global term that models each source word's total attention with a learned normal distribution.
  • Agreement: attention in the two translation directions is encouraged to match via joint symmetric training.

Experiments cover low-resource translation between English and Chinese, Russian, Estonian and Romanian, in both directions. The enriched model consistently lowers perplexity and mostly improves greedy-decoding BLEU over vanilla attention. Greedy neural output still trails a phrase-based system on such small data. However, the model's scores work well as reranking features for phrase-based candidates. The work shows that linguistic knowledge from SMT alignment can make NMT more data-efficient.

Key points

  • Perplexity drops in all 8 directions compared with both the encoder-decoder and the attentional baseline. For example, Zh-En goes from 4.77 to 4.31, and En-Et from 19.4 to 17.0.
  • With greedy decoding, BLEU gains over the attentional model are mostly statistically significant. Zh-En rises from 29.9 to 31.56 and En-Ro from 28.1 to 30.07.
  • En-Ru is the exception: adding global fertility reduced BLEU from 5.94 to 5.26. The authors attribute this to a noisy web-crawled Russian-English corpus.
  • In reranking phrase-based n-best lists, the model beats the phrase-based baseline by at least 1 BLEU for every language pair into English. For Zh-En the gain is nearly 3 BLEU (40.63 to 43.50).
  • Fertility is modelled with far fewer parameters than in IBM Models 3–5, because it operates over source word embeddings rather than per-word tables.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓The University of MelbourneMelbourne · Australia60 works · 21,199 citations🎓Monash UniversityMelbourne · Australia31 works · 54,201 citations🏢Microsoft Research (United Kingdom)Cambridge · United Kingdom6 works · 2,344 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vymolova, Kaisheng Yao, Chris Dyer and Gholamreza Haffari. 2016. Incorporating Structural Alignment Biases into an Attentional Neural Translation Model. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1102

@inproceedings{cohn2016incorporating,
  title = {{Incorporating Structural Alignment Biases into an Attentional Neural Translation Model}},
  author = {Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vymolova, Kaisheng Yao, Chris Dyer and Gholamreza Haffari},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1102},
  url = {https://doi.org/10.18653/v1/n16-1102}
}

References

26 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexandra Antonova and Alexey Misyurev. 2011. Building a web-based parallel corpus and filtering out machine-translated text. In Proceedings of the 4th Workshop on Building and Using Comparable Corpora: Comparable Corpora and the Web, pages 136– 144. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations (ICLR), San Diego, CA.
  3. Peter E. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2). find
  4. Yong Cheng, Shiqi Shen, Zhongjun Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2015. Agreement-based joint training for bidirectional attention-based neural machine translation. In arXiv: 1512.04650 [cs.CL]. — [arXiv:1512.04650](https://arxiv.org/abs/1512.04650) find
  5. K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio. 2014. On the properties of neural machine translation. In arXiv:1409.1259 [cs.CL].
  6. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia,… find
  7. S. Feng, S. Liu, M. Li, and M. Zhou. 2016. Implicit Distortion and Fertility Models for Attention-based Encoder-Decoder NMT Model. ArXiv e-prints, January. Kuzman Ganchev, João V. Graça, and Ben Taskar. 2008. Better alignments = better translations? In Proceedings of ACL-08: HLT, pages 986–993, Co… find
  8. Kenneth Heafield. 2011. KenLM: faster and smaller language model queries. In Proceedings of the EMNLP 2011 Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland, United Kingdom, July.
  9. S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9:1735–1780. find
  10. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, October.
  11. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of North American Chapter of the Association for Computational Linguistics on Human Language Technology, pages 48–54. find
  12. Philipp Koehn, Amittai Axelrod, Alexandra Birch, Chris Callison-Burch, Miles Osborne, David Talbot, and Michael White. 2005. Edinburgh system description for the 2005 IWSLT speech translation evaluation. In IWSLT, pages 68–75.
  13. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proc. ACL Interactive Poster and Demonstration Sessions, pages… find
  14. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Dekang Lin and Dekai Wu, editors, Proceedings of EMNLP 2004, pages 388–395, Barcelona, Spain, July. Association for Computational Linguistics.
  15. Philipp Koehn. 2005. Europarl: A Parallel Corpus for Statistical Machine Translation. In Conference Proceedings: the tenth Machine Translation Summit, pages 79–86. find
  16. AAMT. Tomer Levinboim, Ashish Vaswani, and David Chiang. 2015. Model invertibility regularization: Sequence alignment with or without parallel data. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 609– 618, Denver, CO. find
  17. Percy Liang, Ben Taskar, and Dan Klein. 2006. Alignment by agreement. In Proceedings of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 104–111, New York, NY.
  18. Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal, September. Association for Computational Li…
  19. Graham Neubig, Makoto Morishita, and Satoshi Nakamura. 2015. Neural reranking improves subjective quality of machine translation: NAIST at WAT2015. In Proceedings of the 2nd Workshop on Asian Translation (WAT2015), Kyoto, Japan, October. find
  20. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. find
  21. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics - Volume 1, ACL. find
  22. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL.
  23. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Neural Information Processing Systems (NIPS), pages 3104–3112, Montréal. find
  24. Z. Tu, Z. Lu, Y. Liu, X. Liu, and H. Li. 2016. Modeling Coverage for Neural Machine Translation. ArXiv eprints, January.
  25. Stephan Vogel, Hermann Ney, and Christoph Tillmann. 1996. HMM-based word alignment in statistical translation. In Proceedings of the International Conference on Computational Linguistics (COLING), pages 836– 841. find
  26. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pages 2048–2057.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cohn-2016-incorporating-structural-alignment-biases-into/