Elamur Research
☰
home / papers / 2017 / Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder

Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P17-1177.pdf

Summary

Standard attentional NMT models treat the source sentence as a flat word sequence and ignore its syntax. This can lead to errors such as wrong attachment and to translations that split up source phrases. Chen et al. propose a syntax-aware model for Chinese-English translation that uses precomputed, binarized phrase-structure trees in the style of the Penn Chinese Treebank. The model uses only the tree shape, not the node labels.

  • Encoder: a bidirectional tree encoder built from Tree-GRU units. A bottom-up pass is followed by a top-down pass, so every node, including each word, carries both sequential and full-tree context.
  • Decoder: a tree-coverage model extends word-level coverage so that attention can account for parent–child relations. This favors phrasal cohesion and discourages translating overlapping nodes twice.

The full system beats a plain attentional baseline and a stronger baseline that combines a Tree-LSTM encoder with word coverage. The largest gains appear on long sentences. This was presented as the first NMT work to inject source syntax into both the encoder and the decoder.

Key points

  • The best model (bidirectional tree encoder plus tree-coverage) improves on the conventional attention-based NMT baseline by +3.54 BLEU, and it surpasses the combination of the Tree-LSTM encoder (Eriguchi et al., 2016) with coverage (Tu et al., 2016) by +1.90 BLEU.
  • With a two-phase training strategy, the bidirectional encoder outperforms the unidirectional Tree-GRU by 0.79 BLEU.
  • With 512 hidden units, the bidirectional encoder averages 32.48 BLEU over MT02–MT06. This beats Tree-GRU (31.93) and the head-lexicalized variant of Teng and Zhang (2016) (32.03).
  • A coverage variant that also looks at neighboring words gave no benefit and lowered BLEU by about 0.2.
  • Gains increase for sentences longer than 20 words and peak above 50 words.
  • The code is released at github.com/howardchenhd/Syntax-awared-NMT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Machine translation

Cited by

53 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 51 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1177 · arXiv 1707.05436

@inproceedings{chen2017improved,
  title = {{Improved Neural Machine Translation with a Syntax-Aware Encoder and Decoder}},
  author = {Chen, Huadong and Huang, Shujian and Chiang, David and Chen, Jiajun},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1177},
  eprint = {1707.05436},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p17-1177}
}

References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR 2015. http://arxiv.org/abs/1409.0473. ↗ our card
  2. David Chiang. 2007. Hierarchical phrase-based translation. Compututational Linguistics 33(2):201–228. https://doi.org/10.1162/coli.2007.33.2.201. ↗ our card
  3. Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the properties of neural machine translation: Encoder–decoder approaches. In Proc. Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation. pages 103–111. http://www.aclweb.org/anthology/W14-4012. ↗ our card
  4. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014b. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proc. EMNLP. pages 1724–1734. http://www.aclweb.org/anthology/D14-1179. ↗
  5. Trevor Cohn, Cong Duy Vu Hoang, Ekaterina Vy-molova, Kaisheng Yao, Chris Dyer, and Gholamreza Haffari. 2016. Incorporating structural alignment biases into an attentional neural translation model. In Proc. NAACL HLT. pages 876–885. http://www.aclweb.org/anthology/N16-1102. ↗
  6. Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-sequence attentional neural machine translation. In Proc. ACL. pages 823– 833. http://www.aclweb.org/anthology/P16-1078. ↗
  7. Shi Feng, Shujie Liu, Nan Yang, Mu Li, Ming Zhou, and Kenny Q. Zhu. 2016. Improving attention modeling with implicit distortion and fertility for machine translation. In Proc. COLING. pages 3082– 3092. http://aclweb.org/anthology/C16-1290. ↗
  8. Heidi J. Fox. 2002. Phrasal cohesion and statistical machine translation. In Proc. EMNLP. pages 304– 3111. https://doi.org/10.3115/1118693.1118732. ↗
  9. Yoon Kim, Carl Denton, Luong Hoang, and Alexander M. Rush. 2017. Structured attention networks. In Proc. ICLR. http://arxiv.org/abs/1702.00887. ↗
  10. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proc. NAACL HLT. pages 48–54. https://doi.org/10.3115/1073445.1073462. ↗
  11. Filippos Kokkinos and Alexandros Potamianos. 2017. Structural attention neural networks for improved sentiment analysis. In Proc. EACL. pages 586–591. http://www.aclweb.org/anthology/E17-2093. ↗
  12. Jiwei Li, Thang Luong, Dan Jurafsky, and Eduard Hovy. 2015. When are tree structures necessary for deep learning of representations? In Proc. EMNLP. pages 2304–2314. http://aclweb.org/anthology/D15-1278. ↗
  13. Yang Liu, Qun Liu, and Shouxun Lin. 2006. Tree-to-string alignment template for statistical machine translation. In Proc. ACL. pages 609–616. https://doi.org/10.3115/1220175.1220252. ↗ our card
  14. Haitao Mi, Baskaran Sankaran, Zhiguo Wang, and Abe Ittycheriah. 2016. Coverage embedding models for neural machine translation. In Proc. EMNLP. pages 955–960. https://aclweb.org/anthology/D16-1096. ↗
  15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. ACL. pages 311–318. https://doi.org/10.3115/1073083.1073135. ↗ our card
  16. Slav Petrov and Dan Klein. 2007. Improved inference for unlexicalized parsing. In Proc. NAACL HLT. pages 404–411. http://www.aclweb.org/anthology/N/N07/N07-1051. ↗
  17. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural MT learn source syntax? In Proc. EMNLP. pages 1526–1534. https://aclweb.org/anthology/D16-1159. ↗ our card
  18. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27, pages 3104– 3112. http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks. ↗
  19. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proc. ACL-IJCNLP. pages 1556–1566. http://www.aclweb.org/anthology/P15-1150. ↗
  20. Zhiyang Teng and Yue Zhang. 2016. Bidirectional tree-structured LSTM with head lexicalization. arXiv:1611.06788. http://arxiv.org/abs/1611.06788. ↗
  21. Zhaopeng Tu, Zhengdong Lu, Yang Liu, Xiaohua Liu, and Hang Li. 2016. Modeling coverage for neural machine translation. In Proc. ACL. pages 76–85. http://www.aclweb.org/anthology/P16-1008. ↗ our card
  22. Naiwen Xue, Fei Xia, Fu-dong Chiou, and Marta Palmer. 2005. The penn chinese treebank: Phrase structure annotation of a large corpus. Nat. Lang. Eng. 11(2):207–238. https://doi.org/10.1017/S135132490400364X. ↗
  23. Kenji Yamada and Kevin Knight. 2001. A syntax-based statistical translation model. In Proc. ACL. pages 523–530. https://doi.org/10.3115/1073012.1073079. ↗ our card
  24. Matthew D. Zeiler. 2012. ADADELTA: an adaptive learning rate method. CoRR abs/1212.5701. http://arxiv.org/abs/1212.5701. ↗ our card
  25. Yue Zhang and Stephen Clark. 2009. Transition-based parsing of the Chinese Treebank using a global discriminative model. In Proc. IWPT. pages 162–171. http://www.aclweb.org/anthology/W09-3825. ↗