Elamur Research⌕
☰
home / papers / 2019 / Visualizing and Understanding the Effectiveness of BERT

Visualizing and Understanding the Effectiveness of BERT

Paper · 2019 · Yaru Hao, Li Dong, Furu Wei et al.

Summary

Pre-training followed by fine-tuning works well across NLP tasks, but it has not been clear why it trains more easily and generalizes better. Hao et al. study this using BERT-large. They compare fine-tuning the pre-trained weights with training the same architecture from random initialization on four GLUE datasets: MNLI, RTE, SST-2 and MRPC.

They use three visualizations:

  • 1D loss curves, interpolated between the starting and final parameters.
  • 2D loss surfaces, whose second axis is the fine-tuning direction from a different dataset, rescaled to the same norm.
  • Optimization trajectories, with each epoch projected onto that 2D plane.

Pre-trained initialization gives wider, flatter optima and smoother descent paths, so convergence is faster. Fine-tuning also resists overfitting even though BERT is heavily over-parameterized. The training loss surface closely tracks the generalization error surface, which helps explain BERT's strong results on unseen data. A layer-wise analysis shows that layers near the input change little during fine-tuning and carry transferable knowledge, while upper layers absorb task-specific information. The findings point toward fine-tuning methods that deliberately seek flat minima.

Key points

  • Setup: BERT-large with the cased tokenizer and the Adam optimizer, fine-tuned for 3–5 epochs. Training from scratch took longer, running 8 epochs on SST-2 and 16 on each of the remaining datasets. Data splits follow GLUE (Wang et al., 2019), and accuracy is the metric.
  • Loss landscapes: both interpolation coefficients range over [−4, 4], with 40 sampled points per axis. Task-specific layers are frozen, so only the BERT encoder parameters are perturbed.
  • Fine-tuned baselines: dev accuracy is 86.54 on MNLI, 75.45 on RTE, 94.04 on SST-2 and 90.20 on MRPC.
  • Layer rollback (low layers): resetting layers 0–7 to their pre-trained values costs little and slightly improves MNLI (+0.19).
  • Layer rollback (high layers): resetting layers 16–23 is damaging, for example −53.88 on MNLI and −34.75 on SST-2.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States240 works · 123,537 citations🎓Beihang UniversityBeijing · China27 works · 8,089 citations

📚 Shelves 5 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works

Cite

Cite as Yaru Hao, Li Dong, Furu Wei and Ke Xu. 2019. Visualizing and Understanding the Effectiveness of BERT. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d19-1424

@inproceedings{hao2019visualizing,
  title = {{Visualizing and Understanding the Effectiveness of BERT}},
  author = {Yaru Hao, Li Dong, Furu Wei and Ke Xu},
  booktitle = {Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},
  year = {2019},
  doi = {10.18653/v1/d19-1424},
  url = {https://doi.org/10.18653/v1/d19-1424}
}

References

35 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alexei Baevski, Sergey Edunov, Yinhan Liu, Luke Zettlemoyer, and Michael Auli. 2019. Cloze-driven pretraining of self-attention networks. arXiv preprint arXiv:1903.07785. find
  2. Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, and Danilo Giampiccolo. 2006. The second PASCAL recognising textual entailment challenge. In Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment. find
  3. Luisa Bentivogli, Ido Dagan, Hoa Trang Dang, Danilo Giampiccolo, and Bernardo Magnini. 2009. The fifth PASCAL recognizing textual entailment challenge. In In Proc Text Analysis Conference (TAC09. find
  4. Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. 2017. Entropy-SGD: Biasing gradient descent into wide valleys. find
  5. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW’05,… find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. our card
  7. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). our card
  8. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. arXiv preprint arXiv:1905.03197. our card
  9. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1–9, Prague. Association for Computational Linguistics. find
  10. Yoav Goldberg. 2019. Assessing BERT’s syntactic abilities. CoRR, abs/1901.05287. our card
  11. Ian J. Goodfellow and Oriol Vinyals. 2015. Qualitatively characterizing neural network optimization problems. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. find
  12. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Flat minima. Neural Comput., 9(1):1–42. find
  13. Jeremy Howard and Sebastian Ruder. 2018a. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328–339, Melbourne, Australia. Association for Computational Linguistics. find
  14. Jeremy Howard and Sebastian Ruder. 2018b. Universal language model fine-tuning for text classification. In ACL. Association for Computational Linguistics. find
  15. Daniel Jiwoong Im, Michael Tao, and Kristin Branson. 2016. An empirical analysis of deep network loss surfaces. CoRR, abs/1612.04010. — [arXiv:1612.04010](https://arxiv.org/abs/1612.04010) find
  16. Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407. our card
  17. Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. 2016. On large-batch training for deep learning: Generalization gap and sharp minima. CoRR, abs/1609.04836. find
  18. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, San Diego, CA. find
  19. Adhiguna Kuncoro, Chris Dyer, John Hale, Dani Yogatama, Stephen Clark, and Phil Blunsom. 2018. LSTMs can learn syntax-sensitive dependencies well, but modeling structure makes them better. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape… our card
  20. Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. 2018. Visualizing the loss landscape of neural nets. In Neural Information Processing Systems. find
  21. Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of LSTMs to learn syntax-sensitive dependencies. CoRR, abs/1611.01368. our card
  22. Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. CoRR, abs/1903.08855. our card
  23. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b. Multi-task deep neural networks for natural language understanding. CoRR, abs/1901.11504. our card
  24. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6297–6308. our card
  25. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog… our card
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30, pages 5998–6008. Curran Associates, Inc. our card
  27. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations. our card
  28. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
  29. Chiyuan Zhang, Samy Bengio, and Yoram Singer. 2019. Are all layers created equal? arXiv preprint arXiv:1902.01996. — [arXiv:1902.01996](https://arxiv.org/abs/1902.01996) find
  30. Matthew Peters, Sebastian Ruder, and Noah A. Smith. 2019. To tune or not to tune? Adapting pretrained representations to diverse tasks. find
  31. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. find
  32. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  33. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–… find
  34. Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. BERT rediscovers the classical NLP pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. our card
  35. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019b. What do you learn from context? Probing for sentence structure in contextualized word representations. In International Conference… our card

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hao-2019-visualizing-understanding-effectiveness-bert/