Elamur Research⌕
☰
home / papers / 2018 / Identifying and Controlling Important Neurons in Neural Machine Translation

Identifying and Controlling Important Neurons in Neural Machine Translation

Paper · 2018 · Anthony Bau, Yonatan Belinkov, Hassan Sajjad et al.

Summary

Earlier probing work showed that NMT encoders hold morphological, syntactic and semantic knowledge. However, it examined whole vectors and depended on labeled data. This paper asks three questions: do single neurons matter for translation, are they interpretable, and can they be used to steer the output? The authors rank top-layer encoder neurons without supervision. Their premise is that independently trained models converge on similar important units, so they compare neurons across models using MaxCorr, MinCorr, linear regression (LinReg) and SVCCA. The experiments use 18 LSTM attention models with charCNN inputs, trained on the UN parallel corpus. Zeroing out top-ranked neurons at test time damages translation far more than removing low-ranked ones, which confirms that shared neurons carry key information. Visualizations and inexpensive supervised checks reveal neurons that track position, punctuation, noun phrases, plurality and Spanish noun gender. Shifting the activations of chosen neurons gives partial control over tense, number and gender in translations. The results suggest that some NMT knowledge is localized rather than fully distributed. The methods are task-independent and could help reduce bias.

Key points

  • Data and models: The models translate English into Arabic, Chinese, French, Russian and Spanish, plus an English-English auto-encoder, all trained on the UN corpus (Ziemski et al., 2016). There are 3 models per target, each trained on a different 500K-sentence slice, for 18 models in total. Each is a 500-dimensional 2-layer LSTM with attention.
  • Ranking methods:
    • MaxCorr takes a neuron's single best Pearson match in any other model.
    • MinCorr takes the weakest of its best matches across the other models.
    • LinReg uses the mean squared error when predicting the neuron from another model's full representation.
    • SVCCA keeps PCA components covering 99% of the variance and ranks directions rather than individual neurons.
  • Verification: Neurons are erased from the top or bottom of each ranking before decoding. Supervised checks use GMM classifiers and conditional variance computed over Spacy annotations.
  • Interpretable units (English-Spanish model): One neuron detects plural nouns (F1 0.87), and another segments noun phrases (IOB accuracy 0.80). Others respond to parentheses, conjunctions, dates and Spanish noun gender. Many highly ranked units mainly encode word position.
  • Control: Adjusting neuron activations changes tense, number and gender with varying success. For example, a Spanish verb moves from past (apoyó) to imperfect (apoyaba) to present (apoya). Chinese needed much larger shifts, which fits the fact that it rarely marks tense.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,105 worksInterpretability · 149 works

Cite

Cite as Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi and James Glass. 2018. Identifying and Controlling Important Neurons in Neural Machine Translation. arXiv:1811.01157.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1811.01157 · arXiv 1811.01157

@misc{bau2018identifying,
  title = {{Identifying and Controlling Important Neurons in Neural Machine Translation}},
  author = {Anthony Bau, Yonatan Belinkov, Hassan Sajjad, Nadir Durrani, Fahim Dalvi and James Glass},
  year = {2018},
  doi = {10.48550/arxiv.1811.01157},
  eprint = {1811.01157},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.01157}
}

References

33 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks. arXiv preprint arXiv:1608.04207, 2016.
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473, 2014.
  3. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Computer Vision and Pattern Recognition, 2017. find
  4. Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. What do Neural Machine Translation Models Learn about Morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 861–872. Association for Computation…
  5. La parte interesada Cuestion interesada Gran útil sing. sing. sing. α Translation Gen -1 -0.5, 0, 0.5 Temas relativos a la información Cuestiones relativas a la información ms. fm. Translation Tense past/present past/present past/impf./present past/present untensed/present Gino Brunner, Yuyi Wang… — [arXiv:1801.06024](https://arxiv.org/abs/1801.06024) find
  6. Chris Dyer, Victor Chahuneau, and Noah A. Smith. A Simple, Fast, and Effective Reparameterization of IBM Model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 644–648. Association for Computatio… find
  7. Jeffrey L. Elman. Distributed representations, simple recurrent networks, and grammatical structure. Machine learning, 7(2-3):195–225, 1991. find
  8. J. Ganesh, Manish Gupta, and Vasudeva Varma. Interpretation of Semantic Tweet Representations. In Proceedings of the 2017 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining 2017, ASONAM ’17, pp. 95–102, New York, NY, USA, 2017. ACM. ISBN 978-1-4503-4993-2. doi: 10.1… — [doi:10.1145/3110025.3110083](https://doi.org/10.1145/3110025.3110083) find
  9. Ross W. Gayler and Simon D. Levy. Compositional connectionism in cognitive science ii: the localist/distributed dimension. Connection Science, 23(2):85–89, 2011. doi: 10.1080/09540091.2011.587505. URL https://doi.org/10.1080/09540091.2011.587505. — [doi:10.1080/09540091.2011.587505](https://doi.org/10.1080/09540091.2011.587505) ↗
  10. Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional Sequence to Sequence Learning. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1243–…
  11. Felix Hill, Kyunghyun Cho, Sébastien Jean, and Yoshua Bengio. The representational geometry of word meanings acquired by neural machine translation models. Machine Translation, 31(1):3– 18, Jun 2017. ISSN 1573-0573. doi: 10.1007/s10590-017-9194-2. URL https://doi.org/10.1007/s10590-017-9194-2. ↗
  12. Ákos Kádár, Grzegorz Chrupała, and Afra Alishahi. Representation of linguistic form and function in recurrent neural networks. arXiv preprint arXiv:1602.08952, 2016.
  13. Andrej Karpathy, Justin Johnson, and Li Fei-Fei. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078, 2015.
  14. Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. Character-aware Neural Language Models. arXiv preprint arXiv:1508.06615, 2015.
  15. Arne Köhn. What’s in an Embedding? Analyzing Word Embeddings through Multilingual Evaluation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2067–2073, Lisbon, Portugal, September 2015. Association for Computational Linguistics. URL http://aclweb.org/… ↗
  16. Jiwei Li, Will Monroe, and Dan Jurafsky. Understanding Neural Networks through Representation Erasure. arXiv preprint arXiv:1612.08220, 2016a. find
  17. Yixuan Li, Jason Yosinski, Jeff Clune, Hod Lipson, and John Hopcroft. Convergent Learning: Do different neural networks learn the same representations? In International Conference for Learning Representations (ICLR), 2016b. find
  18. Zachary C Lipton. The Mythos of Model Interpretability. In ICML Workshop on Human Interpretability in Machine Learning (WHI), 2016. find
  19. Ari S. Morcos, David G.T. Barrett, Neil C. Rabinowitz, and Matthew Botvinick. On the importance of single directions for generalization. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=r1iuQjxCZ. ↗
  20. Peng Qian, Xipeng Qiu, and Xuanjing Huang. Analyzing Linguistic Knowledge in Sequential Model of Sentence. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 826–835, Austin, Texas, November 2016a. Association for Computational Linguistics. URL https://acl… find
  21. Peng Qian, Xipeng Qiu, and Xuanjing Huang. Investigating Language Universal and Specific Properties in Word Embeddings. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1478–1488, Berlin, Germany, August 2016b. Association for Co… find
  22. Alec Radford, Rafal Jozefowicz, and Ilya Sutskever. Learning to generate reviews and discovering sentiment. arXiv preprint arXiv:1704.01444, 2017.
  23. Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Inf… find
  24. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715–1725. Association for Computational Linguistics, 2016. doi: 10.1…
  25. Xing Shi, Kevin Knight, and Deniz Yuret. Why Neural Translations are the Right Length. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2278–2282. Association for Computational Linguistics, 2016a. doi: 10.18653/v1/D16-1248. URL http://www.aclweb.org/anth… ↗
  26. Xing Shi, Inkit Padhi, and Kevin Knight. Does String-Based Neural MT Learn Source Syntax? In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1526–1534, Austin, Texas, November 2016b. Association for Computational Linguistics. URL https://aclweb.org/antholo… ↗
  27. Jinsong Su, Shan Wu, Deyi Xiong, Yaojie Lu, Xianpei Han, and Biao Zhang. Variational Recurrent Neural Machine Translation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI-18), 2018. find
  28. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Sys…
  29. Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818–833. Springer, 2014.
  30. Biao Zhang, Deyi Xiong, jinsong su, Hong Duan, and Min Zhang. Variational Neural Machine Translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 521–530. Association for Computational Linguistics, 2016. doi: 10.18653/v1/D16-1050. URL http://www.acl… ↗
  31. B. Zhou, A. Khosla, Lapedriza. A., A. Oliva, and A. Torralba. Learning Deep Features for Discriminative Localization. CVPR, 2016. find
  32. Bolei Zhou, Yiyou Sun, David Bau, and Antonio Torralba. Revisiting the Importance of Individual Units in CNNs via Ablation. arXiv preprint arXiv:1806.02891, 2018. — [arXiv:1806.02891](https://arxiv.org/abs/1806.02891) find
  33. Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. The United Nations Parallel Corpus v1.0. In Nicoletta Calzolari (Conference Chair), Khalid Choukri, Thierry Declerck, Sara Goggi, Marko Grobelnik, Bente Maegaard, Joseph Mariani, Helene Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperid…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bau-2018-identifying-controlling-important-neurons-neural/