Elamur Research
☰
home / papers / 2016 / Does String-Based Neural MT Learn Source Syntax?

Does String-Based Neural MT Learn Source Syntax?

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D16-1159.pdf

Summary

Encoder-decoder NMT models see only string pairs, so it is unclear whether they pick up source-language grammar that syntax-based MT systems encode explicitly. The authors probe two-layer, 1000-unit non-attentional LSTM models trained on WMT2014 English→French (E2F) and English→German (E2G) data. They compare them with a neural parser (E2P) as an upper bound and with two autoencoders (E2E, and PE2PE on permuted word order) as lower bounds. First, logistic regression reads five labels — voice, tense, top-level syntactic sequence (TSS), POS, and smallest phrase constituent (SPC) — from the cell states, and recursive feature elimination isolates the units that matter. Second, a new decoder is trained to emit linearized constituency trees from each model's encoding vector, and the parses are evaluated on WSJ Section 23. Translation encoders hold far more syntax than autoencoders at both sentence and word level, and word-level labels are better predicted from the lower layer. Still, the extracted trees contain many attachment errors. The work gives early evidence that NMT learns implicit source syntax and shows where that knowledge falls short.

Key points

  • Voice can be predicted from the E2F encoding vector with 92.8% accuracy, against a majority baseline of 82.8%. The E2E autoencoder reaches only 82.7%, so copying a sentence does not require storing its voice. For voice and tense overall, the NMT models and E2P reach about 95%.
  • The E2P upper-bound parser was trained on 8,162K sentence pairs, called the CJ corpus. It combines about 162K gold-treebank sentences (WSJ Penn Treebank sections 0–22, Ontonotes v5, English Web Treebank) with 8M WMT sentences parsed by the Charniak-Johnson parser. It scores 89.6 F1 on WSJ 23, close to the previously published 90.5.
  • Trees decoded from NMT encoders get about 80 labeled F1 (E2G2P 80.34, E2F2P 79.27) and an average tree edit distance (TED) of about 17. Trees from the autoencoders score below 60 F1 (E2E2P 59.35, PE2PE2P 46.64) with TED above 30.
  • The authors also compare the 569 short sentences that every model parsed into valid trees. On this set, NMT-derived parses tag about 86% of POS correctly (85.30 and 87.09), versus 96.00 for E2P and 54–68 for the autoencoders.
  • The Berkeley Parser Analyzer shows that NMT-derived parses make roughly twice as many bracketing errors as E2P for most error types. For "Sense Confusion" (a phrase's head word gets the wrong POS, causing a wrong attachment), they make more than 16 times as many.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Machine translation

Cited by

192 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 173 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d16-1159

@inproceedings{shi2016does,
  title = {{Does String-Based Neural MT Learn Source Syntax?}},
  author = {Shi, Xing and Padhi, Inkit and Knight, Kevin},
  booktitle = {Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing},
  year = {2016},
  doi = {10.18653/v1/d16-1159},
  url = {https://doi.org/10.18653/v1/d16-1159}
}

References

28 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. “beyond” is predicted as “RB” instead of “IN”, resulting in a Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Matous Machacek, Christof Monz, Pavel Pecina, Matt Post, Herv Saint-Amand, Radu Soricut, and Lucia Specia, editors. 2014. Proc. Ninth Workshop on Statistical Machine Translation. missing prepositional phrase. find
  2. Eugene Charniak and Mark Johnson. 2005. Coarse-to-fine n-best parsing and MaxEnt discriminative reranking. In Proc. ACL. find our card
  3. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proc. EMNLP. find
  4. Brooke Cowan, Ivona Kučerová, and Michael Collins. 2006. A discriminative model for tree-to-tree translation. In Proc. EMNLP. find
  5. Steve DeNeefe, Kevin Knight, Wei Wang, and Daniel Marcu. 2007. What can syntax-based MT learn from phrase-based MT? In Proc. EMNLP-CoNLL. find
  6. Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What ’ s in a translation rule ? Information Sciences, 2004. find our card
  7. Michel Galley, Jonathan Graehl, Kevin Knight, Daniel Marcu, Steve DeNeefe, Wei Wang, and Ignacio Thayer. 2006. Scalable inference and training of context-rich syntactic translation models. In Proc. ACL. find our card
  8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8). find
  9. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proc. ACL. find our card
  10. Andrej Karpathy, Justin Johnson, and Li Fei-Fei. 2016. Visualizing and understanding recurrent networks. In Proc. ICLR. find our card
  11. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proc. EMNLP. find our card
  12. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Proc. NIPS. find
  13. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proc. NAACL. find
  14. Jonathan K. Kummerfeld, David Hall, James R. Curran, and Dan Klein. 2012. Parser showdown at the Wall Street Corral: An empirical investigation of error types in parser output. In Proc. EMNLP-CoNLL. find our card
  15. Jonathan K. Kummerfeld, Daniel Tse, James R Curran, and Dan Klein. 2013. An empirical examination of challenges in Chinese parsing. In Proc. ACL. find
  16. Qv Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proc. ICML. find our card
  17. Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016. Visualizing and understanding neural models in nlp. In Proc. NAACL. find our card
  18. Yang Liu, Qun Liu, and Yajuan Lü. 2011. Adjoining tree-to-string translation. In Proc. ACL. find
  19. Mitchell P Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2). find our card
  20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proc. ACL. find our card
  21. Slav Petrov and Ryan McDonald. 2012. Overview of the 2012 shared task on Parsing the Web. In Notes of the First Workshop on Syntactic Analysis of Non-Canonical Language (SANCL). find
  22. Sameer S Pradhan and Nianwen Xue. 2009. Ontonotes: the 90% solution. In Proc. NAACL. find
  23. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. EMNLP. find
  24. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proc. NIPS. find
  25. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-SNE. Journal of Machine Learning Research, 9. find
  26. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015. Grammar as a foreign language. In Proc. NIPS. find
  27. Kaizhong Zhang and Dennis Shasha. 1989. Simple fast algorithms for the editing distance between trees and related problems. SIAM Journal on Computing, 18(6). find
  28. Han Zhao, Zhengdong Lu, and Pascal Poupart. 2015. Self-adaptive hierarchical sentence model. In Proc. IJ-CAI. find