Elamur Research
☰
home / papers / 2017 / What do Neural Machine Translation Models Learn about Morphology?

What do Neural Machine Translation Models Learn about Morphology?

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P17-1080.pdf

Summary

Neural machine translation (NMT) systems are trained end to end, so it is not obvious what linguistic knowledge their hidden states hold. Belinkov and colleagues test how much word structure these models learn. They train attentional LSTM encoder-decoders on TED-talk parallel data (WIT3, released for IWSLT 2016). The language pairs are Arabic, German, French and Czech with English, plus Arabic–Hebrew and Arabic–German. They then freeze each model, take its per-word hidden states and train a small feed-forward classifier to predict part-of-speech (POS) and full morphological tags. Tagging accuracy serves as a proxy for how good the representations are. The experiments compare word embeddings with a character CNN, different encoder layers, different target languages, and the encoder with the decoder. Character-aware encoders capture morphology much better, especially for rare and unseen words, and they also translate better. Lower layers carry more morphological information than higher ones. Source representations get richer when the target language has simpler morphology, and the decoder learns little about morphology. The paper became a standard early reference for probing classifiers in NLP.

Key points

  • Probing method: The NMT model is frozen. A classifier with one hidden ReLU layer is trained on its encoder or decoder states. A linear probe showed the same trends with lower scores. Code is released as boknilev/nmt-repr-analysis on GitHub.
  • Character vs. word input: With the Arabic–English encoder, POS accuracy rises from 89.62 (word) to 95.35 (char), and BLEU from 24.7 to 28.4. With the Arabic–Hebrew encoder, Arabic morphological tagging improves from 65.2 to 79.66.
  • Rare and unseen words: Using Arabic–Hebrew encoder features, the character model gains about 14 points overall on both POS and morphology. On out-of-vocabulary words the gains are +37.6 for POS and +32.7 for morphology. The gap shrinks as word frequency grows.
  • Decoder: POS accuracy from decoder states is only about 44%, and character input barely helps there. In English→Arabic, BLEU does not improve (13.37 vs. 13.00). The authors suggest the attention mechanism takes over much of the decoder's representational work.
  • Layers and target language: Lower encoder layers are better for morphology, while deeper models give better translations. Translating into a morphologically poorer language, such as English, produces better source-side morphology, and this only partly tracks BLEU.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

224 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 204 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p17-1080 · arXiv 1704.03471

@inproceedings{belinkov2017what,
  title = {{What do Neural Machine Translation Models Learn about Morphology?}},
  author = {Belinkov, Yonatan and Durrani, Nadir and Dalvi, Fahim and Sajjad, Hassan and Glass, James},
  booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2017},
  doi = {10.18653/v1/p17-1080},
  eprint = {1704.03471},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/p17-1080}
}

References

45 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. arXiv preprint arXiv:1608.04207 . find our card
  2. Ibrahim Badr, Rabih Zbib, and James Glass. 2008. Segmentation for English-to-Arabic Statistical Machine Translation. In Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies: Short Papers. Columbus, Ohio, HLT-Short ’08, pages 153–156.… find
  3. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv preprint arXiv:1409.0473 . find our card
  4. Yonatan Belinkov and James Glass. 2016. Large-Scale Machine Translation between Arabic and Hebrew: Available Corpora and Initial Results. In Proceedings of the Workshop on Semitic Machine Translation. Association for Computational Linguistics, Austin, Texas, pages 7–12. find our card
  5. Luisa Bentivogli, Arianna Bisazza, Mauro Cettolo, and Marcello Federico. 2016. Neural versus Phrase-Based Machine Translation Quality: a Case Study. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pa… find
  6. Mauro Cettolo. 2016. An Arabic-Hebrew parallel corpus of TED talks. In Proceedings of the Workshop on Semitic Machine Translation. Association for Computational Linguistics, Austin, Texas, pages 1–6. find our card
  7. Mauro Cettolo, Christian Girardi, and Marcello Federico. 2012. WIT 3 : Web Inventory of Transcribed and Translated Talks. In Proceedings of the 16 th Conference of the European Association for Machine Translation (EAMT). Trento, Italy, pages 261– 268. find
  8. Marta R. Costa-jussà and José A. R. Fonollosa. 2016. Character-based Neural Machine Translation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Association for Computational Linguistics, Berlin, Germany, pages 357–361. http://an… find our card
  9. Nadir Durrani, Philipp Koehn, Helmut Schmid, and Alexander Fraser. 2014. Investigating the Usefulness of Generalized Word Representations in SMT. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers. Dublin City University and Association fo… find
  10. Nadir Durrani, Hassan Sajjad, Alexander Fraser, and Helmut Schmid. 2010. Hindi-to-Urdu Machine Translation through Transliteration. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics. Association for Computational Linguistics, Uppsala, Sweden, pages 465– 474.… find
  11. Jeffrey L Elman. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine learning 7(2-3):195–225. find
  12. Lieke Gelderloos and Grzegorz Chrupała. 2016. From phonemes to images: levels of representation in a recurrent neural model of visually-grounded language learning. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. find
  13. The COLING 2016 Organizing Committee, Osaka, Japan, pages 1309– 1319. http://aclweb.org/anthology/C16-1124. ↗
  14. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation 9(8):1735–1780. find
  15. Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2016. Google’s Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation. arXiv preprint arXiv:1611.04558 . find
  16. Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the Limits of Language Modeling. arXiv preprint arXiv:1602.02410 . find
  17. Ákos Kádár, Grzegorz Chrupała, and Afra Alishahi. 2016. Representation of linguistic form and function in recurrent neural networks. arXiv preprint arXiv:1602.08952 . find
  18. Andrej Karpathy, Justin Johnson, and Fei-Fei Li. 2015. Visualizing and Understanding Recurrent Networks. arXiv preprint arXiv:1506.02078 . find our card
  19. Yoon Kim. 2016. Seq2seq-attn. https:// github.com/harvardnlp/seq2seq-attn. find
  20. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2015. Character-aware Neural Language Models. arXiv preprint arXiv:1508.06615 . find our card
  21. Diederik Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980 . find
  22. Philipp Koehn and Hieu Hoang. 2007. Factored Translation Models. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL). Association for Computational Linguistics, Prague, Czech Republic, pages 868–876… find
  23. Minh-Thang Luong and Christopher D. Manning. 2015. Stanford Neural Machine Translation Systems for Spoken Language Domains. In Proceedings of the International Workshop on Spoken Language Translation. Da Nang, Vietnam. find
  24. Minh-Thang Luong and D. Christopher Manning. 2016. Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Models. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pa… find our card
  25. Minh-Thang Luong, Preslav Nakov, and Min-Yen Kan. 2010. A Hybrid Morpheme-Word Representation for Machine Translation of Morphologically Rich Languages. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pages 148–157… find
  26. Thomas Mueller, Helmut Schmid, and Hinrich Schütze. 2013. Efficient Higher-Order CRFs for Morphological Tagging. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Seattle, Washington, USA, pages 322– 332. http://www… find
  27. Preslav Nakov and Jörg Tiedemann. 2012. Combining Word-Level and Character-Level Models for Machine Translation Between Closely-Related Languages. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Jeju, Korea, ACL ’12, pages 301–305… find
  28. Sonja Nieflen and Hermann Ney. 2000. Improving SMT quality with morpho-syntactic analysis. In COLING 2000 Volume 2: The 18th International Conference on Computational Linguistics. http://www.aclweb.org/anthology/C00-2162. ↗
  29. Philipp Koehn and Kevin Knight. 2003. Empirical Methods for Compound Splitting. In 10th Conference of the European Chapter of the Association for Computational Linguistics. pages 187–194. http://www.aclweb.org/anthology/E03-1076. ↗
  30. Arfath Pasha, Mohamed Al-Badrashiny, Mona Diab, Ahmed El Kholy, Ramy Eskander, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. MADAMIRA: A Fast, Comprehensive Tool for Morphological Analysis and Disambiguation of Arabic. In Proceedings of the Ninth International Conference on Languag… find our card
  31. Arne Köhn. 2015. What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Lisbon, Portugal, pages 2067–2073. http://aclweb.org/anthology/D1… ↗
  32. Peng Qian, Xipeng Qiu, and Xuanjing Huang. 2016a. Analyzing Linguistic Knowledge in Sequential Model of Sentence. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 826–835. https://aclweb.org/ant… ↗
  33. Peng Qian, Xipeng Qiu, and Xuanjing Huang. 2016b. Investigating Language Universal and Specific Properties in Word Embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, German… find
  34. Adwait Ratnaparkhi. 1998. Maximum Entropy Models for Natural Language Ambiguity Resolution. Ph.D. thesis, University of Pennsylvania, Philadelphia, PA. find
  35. Hassan Sajjad, Fahim Dalvi, Nadir Durrani, Ahmed Abdelali, Yonatan Belinkov, and Stephan Vogel. 2017. Challenging Language-Dependent Segmentation for Arabic: An Application to Machine Translation and Part-of-Speech Tagging. In Proceedings of the 55th Annual Meeting of the Association for Computation… find our card
  36. Ekaterina Vylomova, Trevor Cohn, Xuanli He, and Gholamreza Haffari. 2016. Word Representation Models for Morphologically Rich Languages in Neural Machine Translation. arXiv preprint arXiv:1606.04217 . find
  37. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s Neural Machine Translation System: Bridging the Gap between Human and Machine Translation. arXiv preprint arXiv:1609.08144 . find
  38. Jie Zhou, Ying Cao, Xuguang Wang, Peng Li, and Wei Xu. 2016. Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation. Transactions of the Association for Computational Linguistics 4:371–383. https://transacl.org/ojs/index.php/tacl/article/view/863. ↗
  39. Helmut Schmid. 1994. Part-of-Speech Tagging with Neural Networks. In Proceedings of the 15th International Conference on Computational Linguistics (Coling 1994). find
  40. Coling 1994 Organizing Committee, Kyoto, Japan, pages 172–176. find
  41. Helmut Schmid. 2000. LoPar: Design and Implementation. Bericht des Sonderforschungsbereiches “Sprachtheoretische Grundlagen fr die Computerlinguistik” 149, Institute for Computational Linguistics, University of Stuttgart. find
  42. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, Berlin, Germany, pages 1… find
  43. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does String-Based Neural MT Learn Source Syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 1526–1534. https://aclweb.org/anthology/D16-1159. ↗ our card
  44. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in neural information processing systems. pages 3104–3112. find
  45. Antonio Toral and Vı́ctor M. Sánchez-Cartagena. 2017. A Multifaceted Evaluation of Neural versus Phrase-Based Machine Translation for 9 Language Directions. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers. Associa… find