Elamur Research⌕
☰
home / papers / 2016 / Supervised Attentions for Neural Machine Translation

Supervised Attentions for Neural Machine Translation

Paper · 2016 · Haitao Mi, Zhiguo Wang and Abe Ittycheriah

Summary

In attention-based NMT, the attention weights often fail to match real word correspondences, so the model's alignment quality is poor. Mi, Wang and Ittycheriah use alignments of the training pairs as supervision for the attention model. Each reference alignment matrix comes from one of three sources: GIZA++ in one direction, the GDFA merge, or a MaxEnt aligner. It is converted into row-wise probability distributions, and unaligned target words are attached to the source end-of-sentence token. An optional variant first spreads each link with a Gaussian window and then normalizes. The Euclidean distance between this matrix and the model's attention matrix is subtracted from the usual log-likelihood objective.

The authors trained on roughly 5 million Chinese-English sentence pairs from DARPA BOLT, building on a strong coverage-based large-vocabulary baseline (Cov. LVNMT). Joint training with smoothed MaxEnt supervision gave the best single-system translation quality and even outperformed a tree-to-string syntax-based SMT system. It also raised attention alignment F1 by about 5 points. The method offers a simple, inexpensive way to inject alignment knowledge into neural translation.

Key points

  • Test sets and metric: evaluation used NIST MT06, MT08 news and MT08 web, scored with (TER-BLEU)/2, where lower is better.
  • Translation results: the best system (MaxEnt alignments, joint training, Gaussian smoothing) averaged 13.04. This compares with 14.24 for Cov. LVNMT and 13.36 for tree-to-string. BLEU gains over the baseline were significant at p < 0.01 on all three test sets.
  • Optimization strategy: training the alignment and translation parts separately did not help. The A → T setting was much worse, averaging 20.40, so joint optimization (J) is needed.
  • Alignment results: on 447 hand-aligned test sentences, attention F1 rose from 45.76 to 51.94. The MaxEnt aligner itself, trained on 67k hand-aligned sentences, reaches 75.96.
  • Other observations:
    • The choice of alignment source (Zh → En, GDFA or MaxEnt) changed translation quality only slightly.
    • The brevity penalty consistently moved closer to the reference length once the alignment cost was added.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IBMArmonk · United States85 works · 66,118 citations🔬IBM Research - Thomas J. Watson Research CenterYorktown Heights · United States41 works · 46,197 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Haitao Mi, Zhiguo Wang and Abe Ittycheriah. 2016. Supervised Attentions for Neural Machine Translation. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1249

@inproceedings{mi2016supervised,
  title = {{Supervised Attentions for Neural Machine Translation}},
  author = {Haitao Mi, Zhiguo Wang and Abe Ittycheriah},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1249},
  url = {https://doi.org/10.18653/v1/d16-1249}
}

References

20 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sasa Hasan Arne Mauser and Hermann Ney. 2008. Automatic evaluation measures for statistical machine translation system optimization. In Proceedings of LREC 2008, Marrakech, Morocco, may. find
  2. D. Bahdanau, K. Cho, and Y. Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. ArXiv e-prints, September.
  3. Yong Cheng, Shiqi Shen, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Agreement-based joint training for bidirectional attention-based neural machine translation. In Proceedings of IJCAI, New York, USA, July. find
  4. Martin Cmejrek, Haitao Mi, and Bowen Zhou. 2013. Flexible and efficient hypergraph interactions for joint hierarchical and forest-to-string decoding. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 545–555, Seattle, Washington, USA, October. Associati… find
  5. Michael Collins, Philipp Koehn, and Ivona Kucerova. 2005. Clause restructuring for statistical machine translation. In Proceedings of ACL, pages 531–540, Ann Arbor, Michigan, June.
  6. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of ibm model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia,… find
  7. Mark Hopkins and Jonathan May. 2011. Tuning as ranking. In Proceedings of EMNLP.
  8. Abraham Ittycheriah and Salim Roukos. 2005. A maximum entropy word aligner for arabic-english machine translation. In HLT ’05: Proceedings of the HLT and EMNLP, pages 89–96. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural ma… find
  9. Xiaoqing Li, Jiajun Zhang, and Chengqing Zong. 2016. Towards zero unknown word in neural machine translation. In Proceedings of IJCAI 2016, pages 2852– 2858, New York, NY, USA, July. find
  10. P. Liang, B. Taskar, and D. Klein. 2006. Alignment by agreement. In North American Association for Computational Linguistics (NAACL), pages 104–111.
  11. P. Liang, D. Klein, and M. I. Jordan. 2008. Agreement-based learning. In Advances in Neural Information Processing Systems (NIPS). find
  12. Yang Liu, Haitao Mi, Yang Feng, and Qun Liu. 2009. Joint decoding with multiple translation models. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 2 - Volume 2, ACL ’09, pages… find
  13. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal, September. Association for Computational Linguis…
  14. Haitao Mi, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. 2016a. A coverage embedding model for neural machine translation. ArXiv e-prints. find
  15. Haitao Mi, Zhiguo Wang, and Abe Ittycheriah. 2016b. Vocabulary manipulation for neural machine translation. In Proceedings of ACL, Berlin, Germany, August. find
  16. Franz Josef Och and Hermann Ney. 2000. Improved statistical alignment models. In Proceedings of the 38th Annual Meeting on Association for Computational Linguistics, ACL ’00, pages 440–447, Stroudsburg, PA, USA. Association for Computational Linguistics.
  17. Z. Tu, Z. Lu, Y. Liu, X. Liu, and H. Li. 2016. Coverage-based Neural Machine Translation. ArXiv e-prints, January. find
  18. Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method.
  19. CoRR. Jiajun Zhang. 2016. Exploiting source-side monolingual data in neural machine translation. In Proceedings of EMNLP 2016, Austin, Texas, USA, November.
  20. Bing Zhao and Yaser Al-onaizan. 2008. Generalizing local and non-local word-reordering patterns for syntax-based machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, EMNLP ’08, pages 572–581, Stroudsburg, PA, USA. Association for Computational Li… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mi-2016-supervised-attentions-neural-machine-translation/