Elamur Research
☰
home / papers / 2019 / Revealing the Dark Secrets of BERT

Revealing the Dark Secrets of BERT

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1908.08593

Summary

BERT tops many NLP benchmarks, but it is unclear how it achieves this. Kovaleva et al. look at self-attention, the core component, to see what individual heads encode. They use bert-base-uncased (12 layers, 12 heads per layer, 110M parameters), both pre-trained and fine-tuned on seven GLUE tasks: MRPC, STS-B, SST-2, QQP, RTE, QNLI and MNLI. For each input they extract every head's attention matrix. They then sort these matrices into recurring visual types using manual labels and a CNN classifier. They also test whether some heads track FrameNet relations between predicates and core frame elements. Finally, they "switch off" heads by giving every token equal attention and measure how task scores change.

Only a few attention patterns appear, and they repeat across heads and tasks, which points to heavy redundancy. Removing single heads, or even whole layers, often raises accuracy rather than lowering it. This makes the work an early argument for pruning BERT and searching for leaner sub-architectures.

Key points

  • Five attention types: Vertical (focus on [CLS]/[SEP]), Diagonal (neighbouring tokens), Vertical+Diagonal, Block (within one sentence of a pair) and Heterogeneous.
    • About 400 hand-labelled maps trained an 8-layer CNN, which reached F1 0.86.
    • At most 32% (MRPC) to 61% (QQP) of heads fall into the potentially informative Heterogeneous class.
  • How heads are disabled: each token's attention weights in a head are replaced with the uniform value 1/L, where L is sequence length.
  • Effect of disabling one head: the best gain per task ranges from 0.1% (STS-B) to 1.2% (MRPC). For MRPC and RTE, removing a random head improves results on average.
  • Effect of disabling a whole layer: removing layer 1 boosts RTE by 3.2% absolute. QNLI and MNLI lose up to 0.2%.
  • FrameNet heads: two of the 144 heads appear to align with FrameNet relations, but neither matters for GLUE performance.
    • STS-B and RTE instead depend on two heads (layer 4 head 1, layer 2 head 12).
    • These heads emphasise words that occur in both input sentences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Semantic role labeling

Cited by

242 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 235 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1908.08593 · arXiv 1908.08593

@misc{kovaleva2019revealing,
  title = {{Revealing the Dark Secrets of BERT}},
  author = {Kovaleva, Olga and Romanov, Alexey and Rogers, Anna and Rumshisky, Anna},
  year = {2019},
  doi = {10.48550/arxiv.1908.08593},
  eprint = {1908.08593},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1908.08593}
}

References

24 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin. 2019. Rethinking complex neural network architectures for document classification. In Proceedings of NAACL 2019: Conference of the North American Chapter of the Association for Computational Linguistics. find
  2. Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In Proceedings of the 17th international conference on Computational linguistics-Volume 1, pages 86–90. Association for Computational Linguistics. find our card
  3. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1–14. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  5. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005). find our card
  6. Jonathan Frankle and Michael Carbin. 2018. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635. find
  7. Yoav Goldberg. 2019. Assessing bert’s syntactic abilities. arXiv preprint arXiv:1901.05287. find our card
  8. Ganesh Jawahar, Benoı̂t Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In 57th Annual Meeting of the Association for Computational Linguistics (ACL), Florence, Italy. find our card
  9. Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew Peters, and Noah A Smith. 2019. Linguistic knowledge and transferability of contextual representations. arXiv preprint arXiv:1903.08855. find our card
  10. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? arXiv preprint arXiv:1905.10650. find our card
  11. Alec Radford, Karthik Narasimhan, Time Salimans, and Ilya Sutskever. 2018. Improving language understanding with unsupervised learning. Technical report, Technical report, OpenAI. find
  12. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250. find our card
  13. Alexey Romanov and Chaitanya Shivade. 2018. Lessons from natural language inference in the clinical domain. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1586–1596. find our card
  14. Carson T. Schutze. 1996. The Empirical Base of Linguistics: Grammaticality Judgments and Linguistic Methodology. University of Chicago Press. find
  15. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631–1… find
  16. Gongbo Tang, Mathias Müller, Annette Rios, and Rico Sennrich. 2018. Why self-attention? a targeted evaluation of neural machine translation architectures. arXiv preprint arXiv:1808.08946. find
  17. Ke Tran, Arianna Bisazza, and Christof Monz. 2018. The importance of being recurrent for modeling hierarchical structure. arXiv preprint arXiv:1803.03585. find
  18. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  19. Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418. find our card
  20. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. arXiv:1905.00537 [cs]. find our card
  21. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461. find our card
  22. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find
  23. Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901.10430. find our card
  24. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Advances in neural information processing systems, pages 3320–3328. find