Elamur Research
☰
home / papers / 2017 / Towards String-To-Tree Neural Machine Translation

Towards String-To-Tree Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P17-2021.pdf

Summary

The authors investigate whether knowledge of target-language syntax helps neural machine translation (NMT). Such systems usually do not take syntax into account explicitly. Instead of a plain English sentence, the model is trained to output that sentence's linearized, lexicalized constituency tree. In this tree, brackets mark syntactic nodes and the leaves are the words themselves, without POS tags. As a result, the architecture stays standard: attention-based seq2seq as implemented in NEMATUS. The English sentences were parsed automatically with the BLLIP parser, and words were split into subwords using BPE. The syntactic model bpe2tree was compared with the baseline bpe2bpe, which has almost the same number of parameters. Data: the large WMT16 German-English corpus and the low-resource DE/RU/CS–EN pairs from News Commentary v8. bpe2tree mostly achieves higher BLEU, especially when data is scarce. It also reorders words more often and produces more subordinate constructions, and human evaluators preferred its translations slightly more often. Thus, target-language syntax can be added to NMT without changing the architecture.

Key points

  • On WMT16 DE-EN (4.5 million sentence pairs, newstest2016), BLEU rose from 31.19 to 32.13 for single models and from 32.38 to 33.24 for ensembles. On newstest2015 the results are nearly identical: 27.33 vs. 27.36.
  • On News Commentary v8, bpe2tree outperforms the baseline for all three languages. The largest gain is for DE-EN on newstest2016: from 14.16 to 16.13 for single models and from 15.07 to 17.21 for ensembles.
  • The model generated well-formed trees for 5970 of the 6003 sentences in the dev set. From these translations, 233,657 GHKM rules were extracted, of which 22,914 (9.8%) involve word reordering.
  • To compare the models, the authors introduced a distortion score computed from the attention alignments. According to it, bpe2tree reorders words more often. In addition, the first opening bracket usually attends to the main verb of the source sentence.
  • Evaluators on Mechanical Turk reviewed 500 sentences. bpe2tree was chosen in 186 cases (37.2%), bpe2bpe in 154 (30.8%).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Computing & GPUs

Topics: Machine translation

Cited by

73 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 67 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-2021 · arXiv 1704.04743

@inproceedings{aharoni2017string,
  title = {{Towards String-To-Tree Neural Machine Translation}},
  author = {Aharoni, Roee and Goldberg, Yoav},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-2021},
  eprint = {1704.04743},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p17-2021}
}

References

33 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. tion. In Proceedings of the 3rd Workshop on Asian Translation (WAT2016). pages 175–183. find
  2. David Alvarez-Melis and Tommi S. Jaakkola. 2017. Tree-structured decoding with doubly recurrent neural networks. International Conference on Learning Representations (ICLR) . find
  3. Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016b. Tree-to-sequence attentional neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 823–833. http://www.aclweb.org/anthology/P16-1078. ↗
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 . ↗ our card
  5. Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Simaan. 2017. Graph convolutional encoders for syntax-aware neural machine translation. arXiv preprint arXiv:1704.04675 . ↗ our card
  6. Akiko Eriguchi, Yoshimasa Tsuruoka, and Kyunghyun Cho. 2017. Learning to parse and translate improves neural machine translation. arXiv preprint arXiv:1702.03525 http://arxiv.org/abs/1702.03525. ↗ our card
  7. Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer. 2006. Scalable inference and training of context-rich syntactic translation models. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pages 961–968. find our card
  8. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 131–198. find our card
  9. Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What’s in a translation rule? In Daniel Marcu Susan Dumais and Salim Roukos, editors, HLT-NAACL 2004: Main Proceedings. Association for Computational Linguistics, Boston, Massachusetts, USA, pages 273–280. find our card
  10. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and maxent discriminative reranking. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, pages 173–180. find our card
  11. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, pages 1700–1709. http://www.aclweb.org/anthology/D13-1176. ↗
  12. David Chiang. 2005. A hierarchical phrase-based model for statistical machine translation. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, pages 263–270. find our card
  13. Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press, New York, NY, USA, 1st edition. find
  14. David Chiang. 2007. Hierarchical phrase-based translation. computational linguistics 33(2):201–228. find our card
  15. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interactive poster and demonstration sessions. Association for Computational Linguistics, pages 177– 180. find
  16. Do Kook Choe and Eugene Charniak. 2016. Parsing as language modeling. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 2331–2336. https://aclweb.org/anthology/D16-1257. ↗
  17. Chris Dyer, Adhiguna Kuncoro, Miguel Ballesteros, and Noah A. Smith. 2016. Recurrent neural network grammars. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, San Diego, California, pages 199–209. http://www.aclweb.org/anthology/N16-1024. ↗
  18. Minh-Thang Luong, Quoc V Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2015. Multi-task sequence to sequence learning. arXiv preprint arXiv:1511.06114 . ↗ our card
  19. Maria Nadejde, Siva Reddy, Rico Sennrich, Tomasz Dwojak, Marcin Junczys-Dowmunt, Philipp Koehn, and Alexandra Birch. 2017. Syntax-aware neural machine translation using CCG. arXiv preprint arXiv:1702.01147 . ↗
  20. Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016a. Character-based decoding in tree-to-sequence attention-based neural machine transla-Rico Sennrich. 2016. How grammatical is character-level neural machine translation? assessing mt quality with contrastive translation pairs. arXiv preprint arXiv:1612.04629 . Conference on Natural Language Processing (Volume 1: Long Papers). Association for Computational Linguistics, Beijing, China, pages 1556– 1566. http://www.aclweb.org/anthology/P15-1150. ↗ our card
  21. Rico Sennrich, Orhan Firat, Kyunghyun Cho, Alexandra Birch, Barry Haddow, Julian Hitschler, Marcin Junczys-Dowmunt, Samuel L”aubli, Antonio Valerio Miceli Barone, Jozef Mokry, and Maria Nadejde. 2017. Nematus: a Toolkit for Neural Machine Translation. In Proceedings of the Demonstrations at the 15th Conference of the European Chapter of the Association for Computational Linguistics. Valencia, Spain. find our card
  22. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Advances in Neural Information Processing Systems. pages 2773–2781. find
  23. P. Williams, M. Gertz, and M. Post. 2016. Syntax-Based Statistical Machine Translation. Morgan & Claypool publishing. find
  24. Kenji Yamada and Kevin Knight. 2001. A syntax-based statistical translation model. In Proceedings of the 39th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, pages 523–530. find our card
  25. Rico Sennrich and Barry Haddow. 2016. Linguistic input features improve neural machine translation. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 83–91. http://www.aclweb.org/anthology/W16-2209. ↗ our card
  26. Kenji Yamada and Kevin Knight. 2002. A decoder for syntax-based statistical mt. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. Association for Computational Linguistics, pages 303–310. find
  27. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for wmt 16. In Proceedings of the First Conference on Machine Translation. Association for Computational Linguistics, Berlin, Germany, pages 371– 376. http://www.aclweb.org/anthology/W16-2323. ↗ our card
  28. Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 . ↗ our card
  29. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 1715–1725. http://www.aclweb.org/anthology/P16-1162. ↗
  30. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural mt learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 1526–1534. https://aclweb.org/anthology/D16-1159. ↗ our card
  31. Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne. 2016. Syntactically guided neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Berlin, Germany, pages 299–305. http://anthology.aclweb.org/P16-2049. ↗
  32. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. arXiv preprint arXiv:1409.3215 . ↗
  33. Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint find