Elamur Research
☰
home / papers / 2017 / Learning to Parse and Translate Improves Neural Machine Translation

Learning to Parse and Translate Improves Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P17-2012.pdf

Summary

Most neural machine translation (NMT) systems use no linguistic knowledge. Earlier attempts to add syntax mostly targeted the source side and needed external parsers at test time. Eriguchi, Tsuruoka and Cho address this with a multi-task model that injects target-side syntax only during training. They propose NMT+RNNG, which merges an attention-based NMT decoder with a recurrent neural network grammar (RNNG). The translation decoder replaces the RNNG's buffer and shares parameters such as word vectors with it. The model is trained jointly to predict the target words and a sequence of dependency transition actions. Those actions come from automatic SyntaxNet parses of the English side, which the authors frame as a form of knowledge distillation or distant supervision. At inference, the stack and action components are discarded, so translation needs no parser and no extra computation. On four X→English pairs, the method improved RIBES scores everywhere and BLEU on three pairs. This suggests that a syntactic training signal alone can improve translation quality at no deployment cost.

Key points

  • Language pairs and data: Jp-En used the ASPEC corpus from WAT'16, with the first 100K sentence pairs shorter than 50 tokens. Cs-En, De-En and Ru-En used News Commentary v8, with roughly 131K–166K training pairs each.
  • Target-side parses: English sentences were parsed with an unmodified SyntaxNet (Andor et al., 2016). The resulting dependency trees were converted into SHIFT, REDUCE-L and REDUCE-R actions, with each REDUCE carrying a dependency label (about 80–82 action types).
  • BLEU results (test set):
    • Jp-En: 17.88 → 18.84
    • Cs-En: 11.22 → 12.06
    • Ru-En: 12.03 → 12.46
    • These gains were significant at p < 0.005 (bootstrap resampling).
    • De-En BLEU fell slightly, from 16.61 to 16.41.
  • RIBES results: scores rose significantly on all four pairs, for example De-En from 73.75 to 75.03.
  • Ablation (Jp-En dev BLEU): the full model reached 18.60, compared with 17.75 for the baseline. Removing the stack hurt most (17.58), while removing the action component gave 17.94 and removing the buffer gave 18.02. Code is released at github.com/tempra28/nmtrnng.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Machine translation

Cited by

70 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 66 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-2012 · arXiv 1702.03525

@inproceedings{eriguchi2017learning,
  title = {{Learning to Parse and Translate Improves Neural Machine Translation}},
  author = {Eriguchi, Akiko and Tsuruoka, Yoshimasa and Cho, Kyunghyun},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-2012},
  eprint = {1702.03525},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p17-2012}
}

References

33 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Roee Aharoni and Yoav Goldberg. 2017. Towards string-to-tree neural machine translation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. to appear. find our card
  2. Akiko Eriguchi, Kazuma Hashimoto, and Yoshimasa Tsuruoka. 2016. Tree-to-sequence attentional neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. pages 823–833. find
  3. Héctor Martı́nez Alonso, Djamé Seddah, and Benoı̂t Sagot. 2016. From noisy questions to minecraft texts: Annotation challenges in extreme syntax scenario. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT). pages 13–23. find
  4. Kazuma Hashimoto and Yoshimasa Tsuruoka. 2017. Neural Machine Translation with Source-Side Latent Graph Parsing. arXiv preprint arXiv:1702.02265 . ↗
  5. David Alvarez-Melis and Tommi S. Jaakkola. 2017. Tree-structured decoding with doubly-recurrent neural networks. In Proceedings of International Conference on Learning Representations 2017. find
  6. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 . ↗ our card
  7. Daniel Andor, Chris Alberti, David Weiss, Aliaksei Severyn, Alessandro Presta, Kuzman Ganchev, Slav Petrov, and Michael Collins. 2016. Globally normalized transition-based neural networks. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. pages 2442– 2452. find our card
  8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput. 9(8):1735– 1780. find
  9. Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. Automatic evaluation of translation quality for distant language pairs. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. pages 944–952. find our card
  10. Rich Caruana. 1998. Multitask learning. In Learning to learn, Springer, pages 95–133. find
  11. Kyunghyun Cho, Aaron Courville, and Yoshua Bengio. 2015. Describing multimedia content using attention-based encoder-decoder networks. IEEE Transactions on Multimedia 17(11):1875–1886. find
  12. Shihao Ji, S. V. N. Vishwanathan, Nadathur Satish, Michael J. Anderson, and Pradeep Dubey. 2015. Blackout: Speeding up recurrent neural network language models with very large vocabularies. Proceedings of International Conference on Learning Representations 2015 . find
  13. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder–decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. pages 1724–1734. find
  14. Rafal Józefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An empirical exploration of recurrent network architectures. In Proceedings of the 32nd International Conference on Machine Learning. pages 2342–2350. find
  15. Junyoung Chung, Kyunghyun Cho, and Yoshua Bengio. 2016. A character-level decoder without explicit segmentation for neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, pages 1693– 1703. find
  16. Philipp Koehn. 2004. Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing. pages 388–395. find
  17. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the Association for Computational Linguistics Companion Volume Proceedings of the Demo and Poster Sessions. pages 177–180. find
  18. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research 12:2493–2537. find our card
  19. Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and A. Noah Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In Proceedings of the 53rd Annual Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does string-based neural mt learn source syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pages 1526– 1534. find our card
  20. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, Graham Neubig, and Noah A. Smith. 2017. What do recurrent neural network grammars learn about syntax? In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. pages 1249–1258. find
  21. Felix Stahlberg, Eva Hasler, Aurelien Waite, and Bill Byrne. 2016. Syntactically guided neural machine translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. pages 299–305. find
  22. Jason Lee, Kyunghyun Cho, and Thomas Hofmann. 2016. Fully character-level neural machine translation without explicit segmentation. arXiv preprint arXiv:1610.03017 . ↗ our card
  23. WAT. 2016. http://lotus.kuee.kyoto-u.ac.jp/WAT/baseline/ dataPreparationJE.html. ↗
  24. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. pages 1412–1421. find
  25. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 . ↗
  26. Tomáš Mikolov, Martin Karafiát, Lukáš Burget, Jan Černocký, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In Proceedings of the 11th Annual Conference of the International Speech Communication Association (INTER-SPEECH 2010). International Speech Communication Association, pages 1045–1048. find
  27. Mike Mintz, Steven Bills, Rion Snow, and Dan Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP. pages 1003–1011. find
  28. Barret Zoph, Ashish Vaswani, Jonathan May, and Kevin Knight. 2016. Simple, Fast Noise-Contrastive Estimation for Large RNN Vocabularies. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pages 1217–1222. find
  29. Graham Neubig, Yosuke Nakata, and Shinsuke Mori. 2011. Pointwise prediction for robust, adaptable japanese morphological analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. pages 529–533. find
  30. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. pages 311–318. find our card
  31. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. Understanding the exploding gradient problem. arXiv preprint arXiv:1211.5063 abs/1211.5063. ↗
  32. Rico Sennrich and Barry Haddow. 2016. Linguistic input features improve neural machine translation. In Proceedings of the First Conference on Machine Translation. pages 83–91. find our card
  33. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. pages 1715–1725. find