Elamur Research
☰
home / papers / 2018 / Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks

Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1801.07772

Summary

This study asks what neural machine translation encoders learn about meaning, beyond the morphology and syntax examined in earlier probing work. The authors trained 4-layer LSTM attentional encoder-decoder models (seq2seq-attn toolkit, 500 dimensions) that translate English into Arabic, Chinese, French, Spanish and Russian, plus an English-to-English autoencoder. Training used 2 million sentences from the multi-parallel United Nations corpus. With the translation models frozen, they took hidden states from each encoder layer and used them as input to a simple feed-forward classifier. The classifier predicted Penn Treebank POS tags and semantic (SEM) tags from the Bjerva et al. (2016) dataset. Classifier accuracy served as an indicator of how much each layer encodes about the task. POS information is strongest in lower layers, whereas semantic distinctions benefit from deeper ones, especially distinctions that depend on discourse context. The choice of target language matters little. These findings clarify how linguistic knowledge is organized inside NMT encoders and may help guide efforts to add semantic signals to translation systems.

Key points

  • Data: SEM tagging used the silver portion of the Bjerva et al. dataset, which has 66 fine-grained tags in 13 coarse groups. POS tagging used the Penn Treebank, with sections 2-21/22/23 for train/dev/test and 34 tags.
  • Baselines (POS/SEM accuracy): the most-frequent-tag baseline scored 91.95/82.00, Skip-gram embeddings scored 87.06/81.11, and a Word2Tag upper bound reached 95.55/91.41.
  • Layer trends: with unidirectional encoders, POS accuracy peaks around layer 1 at roughly 92%. SEM accuracy is best at layer 4 at about 88-88.6%, with gains concentrated on tags such as discourse functions (coordination, apposition) and noun concepts.
  • Target language: all five foreign target languages gave similar results. The English autoencoder produced clearly weaker features, for example 82.1 SEM and 86.9 POS at layer 4.
  • Architecture variants: bidirectional encoders improved BLEU by 1-2 points and SEM accuracy by 3-4%, reaching 92-93%, close to state of the art. Residual connections added about +0.9 BLEU and up to +0.6% tagging accuracy. The layer trends held across 2-, 3- and 4-layer models. Code is available at github.com/boknilev/nmt-repr-analysis.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation

Topics: Part-of-speech tagging · Machine translation

Cited by

94 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 87 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1801.07772 · arXiv 1801.07772

@misc{belinkov2018evaluating,
  title = {{Evaluating Layers of Representation in Neural Machine Translation on Part-of-Speech and Semantic Tagging Tasks}},
  author = {Belinkov, Yonatan and Marquez, Lluis and Sajjad, Hassan and Durrani, Nadir and Dalvi, Fahim and Glass, James},
  year = {2018},
  doi = {10.48550/arxiv.1801.07772},
  eprint = {1801.07772},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1801.07772}
}

References

38 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Lasha Abzianidze, Johannes Bjerva, Kilian Evang, Hessel Haagsma, Rik van Noord, Pierre Ludmann, Duc-Duy Nguyen, and Johan Bos. 2017. The Parallel Meaning Bank: Towards a Multilingual Corpus of Translations Annotated with Compositional Meaning Representations. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 242–247. Association for Computational Linguistics. find our card
  2. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. 2016. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. arXiv preprint arXiv:1608.04207. ↗ our card
  3. Afra Alishahi, Marie Barking, and Grzegorz Chrupała. 2017. Encoding of phonology in a recurrent neural model of grounded speech. In Proceedings of the SIGNLL Conference on Computational Natural Language Learning, Vancouver, Canada. Association for Computational Linguistics. find
  4. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. arXiv preprint arXiv:1409.0473. ↗ our card
  5. Marzieh Bazrafshan and Daniel Gildea. 2013. Semantic Roles for String to Tree Machine Translation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 419–423, Sofia, Bulgaria. Association for Computational Linguistics. find
  6. Marzieh Bazrafshan and Daniel Gildea. 2014. Comparing Representations of Semantic Roles for String-To-Tree Decoding. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1786–1791, Doha, Qatar. Association for Computational Linguistics. find
  7. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do Neural Machine Translation Models Learn about Morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861–872. Association for Computational Linguistics. find our card
  8. Yonatan Belinkov and James Glass. 2017. Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems. In Advances in Neural Information Processing Systems (NIPS). find
  9. Johannes Bjerva, Barbara Plank, and Johan Bos. 2016. Semantic Tagging with Deep Residual Networks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3531–3541, Osaka, Japan. find
  10. The COLING 2016 Organizing Committee. find
  11. Denny Britz, Anna Goldie, Thang Luong, and Quoc Le. 2017. Massive Exploration of Neural Machine Translation Architectures. ArXiv e-prints. find
  12. Seng Yee Chan, Tou Hwee Ng, and David Chiang. 2007. Word Sense Disambiguation Improves Statistical Machine Translation. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 33–40. Association for Computational Linguistics. find
  13. Fahim Dalvi, Nadir Durrani, Hassan Sajjad, Yonatan Belinkov, and Stephan Vogel. 2017. Understanding and Improving Morphological Learning in the Neural Machine Translation Decoder. In Proceedings of the 8th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Taipei, Taiwan. Association for Computational Linguistics. find
  14. Jeffrey L Elman. 1991. Distributed representations, simple recurrent networks, and grammatical structure. Machine learning, 7(2-3):195–225. find
  15. Qin Gao and Stephan Vogel. 2011. Utilizing Target-Side Semantic Role Labels to Assist Hierarchical Phrase-based Machine Translation. In Proceedings of Fifth Workshop on Syntax, Semantics and Structure in Statistical Translation, pages 107–115. Association for Computational Linguistics. find
  16. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735–1780. ↗
  17. Bevan Jones, Jacob Andreas, Daniel Bauer, Moritz Karl Hermann, and Kevin Knight. 2012. Semantics-Based Machine Translation with Hyperedge Replacement Grammars. In Proceedings of COLING 2012, pages 1359–1376. find our card
  18. The COLING 2012 Organizing Committee. find
  19. Ákos Kádár, Grzegorz Chrupała, and Afra Alishahi. 2016. Representation of linguistic form and function in recurrent neural networks. arXiv preprint arXiv:1602.08952. ↗
  20. Andrej Karpathy, Justin Johnson, and Fei-Fei Li. 2015. Visualizing and Understanding Recurrent Networks. arXiv preprint arXiv:1506.02078. ↗ our card
  21. Yoon Kim. 2016. Seq2seq-attn. https://github.com/harvardnlp/seq2seq-attn. ↗
  22. Diederik Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980. ↗
  23. Arne Köhn. 2015. What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2067–2073, Lisbon, Portugal. Association for Computational Linguistics. find
  24. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies. Transactions of the Association for Computational Linguistics, 4:521– 535. find our card
  25. Ding Liu and Daniel Gildea. 2010. Semantic Role Features for Machine Translation. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 716–724. Coling 2010 Organizing Committee. find
  26. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed Representations of Words and Phrases and their Compositionality. In Advances in Neural Information Processing Systems, pages 3111–3119. find
  27. Sebastian Padó. 2006. User’s guide to sigf: Significance testing by approximate randomisation. https://www.nlpado.de/˜sebastian/software/sigf.shtml. ↗
  28. Juan Antonio Pérez-Ortiz and Mikel L. Forcada. 2001. Part-of-Speech Tagging with Recurrent Neural Networks. In Neural Networks, 2001. Proceedings. IJCNN ’01. International Joint Conference on, volume 3, pages 1588–1592. find
  29. Peng Qian, Xipeng Qiu, and Xuanjing Huang. 2016a. Analyzing Linguistic Knowledge in Sequential Model of Sentence. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 826–835, Austin, Texas. Association for Computational Linguistics. find
  30. Peng Qian, Xipeng Qiu, and Xuanjing Huang. 2016b. Investigating Language Universal and Specific Properties in Word Embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1478–1488, Berlin, Germany. Association for Computational Linguistics. find
  31. Rico Sennrich. 2017. How Grammatical is Character-level Neural Machine Translation? Assessing MT Quality with Contrastive Translation Pairs. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 376–382. Association for Computational Linguistics. find our card
  32. Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does String-Based Neural MT Learn Source Syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1526–1534, Austin, Texas. Association for Computational Linguistics. find our card
  33. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems, pages 3104–3112. find
  34. Ekaterina Vylomova, Trevor Cohn, Xuanli He, and Gholamreza Haffari. 2016. Word Representation Models for Morphologically Rich Languages in Neural Machine Translation. arXiv preprint arXiv:1606.04217. ↗
  35. Yu-Hsuan Wang, Cheng-Tao Chung, and Hung-yi Lee. 2017. Gate Activation Signal Analysis for Gated Recurrent Neural Networks and Its Correlation with Phoneme Boundaries. arXiv preprint arXiv:1703.07588. ↗
  36. Dekai Wu, Pascale N Fung, Marine Carpuat, Chi-kiu Lo, Yongsheng Yang, and Zhaojun Wu. 2011. Lexical Semantics for Statistical Machine Translation. In Handbook of Natural Language Processing and Machine Translation: DARPA Global Autonomous Language Exploitation. find
  37. Zhizheng Wu and Simon King. 2016. Investigating Gated Recurrent Networks for Speech Synthesis. In Acoustics, Speech and Signal Processing (ICASSP), 2016 IEEE International Conference on, pages 5140–5144. IEEE. find
  38. Micha Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016. The United Nations Parallel Corpus v1.0. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC 2016), Paris, France. European Language Resources Association (ELRA). find our card