Elamur Research
☰
home / papers / 2019 / How to Fine-Tune BERT for Text Classification?

How to Fine-Tune BERT for Text Classification?

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1905.05583

Summary

The paper asks how to get the most out of a pre-trained BERT model for text classification. At the time, BERT's potential beyond plain fine-tuning had not been studied systematically. The authors ran a broad set of controlled experiments with uncased BERT-base and Chinese BERT-base. They used seven English benchmarks and one Chinese news corpus, covering sentiment, question and topic classification.

They tested choices within fine-tuning: handling inputs longer than 512 tokens, picking which layer to use, applying layer-wise decaying learning rates and avoiding catastrophic forgetting. They also tested two extra stages. One is additional masked-LM and next-sentence pre-training on within-task, in-domain or cross-domain text. The other is multi-task fine-tuning, where tasks share the encoder but each has its own classifier.

From these results they derive a three-step recipe: further pre-train, optionally fine-tune jointly on related tasks, then fine-tune on the target task. The recipe produced new state-of-the-art error rates on all eight datasets and gave large gains when labeled data is scarce. This makes the paper a practical guide for adapting BERT-style models to new domains and languages.

Key points

  • Datasets: The experiments use the Chinese Sogou News corpus together with seven English sets: DBPedia, AG's News, Yahoo! Answers, the TREC question set, the full and polarity versions of Yelp (Yelp F. and Yelp P.), and IMDb. The code is released as BERT4doc-Classification on GitHub.
  • Further pre-training: This was the strongest single improvement over standard fine-tuning (BERT-FiT). In-domain pre-training (BERT-IDPT-FiT) cut error by 18.57% on average, within-task by 16.07% and cross-domain by 14.38%. Using BERT only as a feature extractor was 9.22% worse.
  • Fine-tuning findings: The top encoder layer works best for classification, and a learning rate that decreases toward lower layers helps prevent catastrophic forgetting. Multi-task fine-tuning (BERT-MFiT-FiT) gave small gains on IMDb, Yelp P., AG and DBP. It brought little or nothing on top of cross-domain pre-training.
  • Few-shot setting: With only 0.4% of the IMDb training data, within-task pre-training lowered the error rate from 17.26% to 9.23%.
  • BERT-LARGE: With within-task pre-training, BERT-LARGE reached the best reported errors: IMDb 4.21%, Yelp P. 1.81%, Yelp F. 28.62%, AG 4.66% and DBP 0.61%. These beat ULMFiT, which outperforms plain BERT-base fine-tuning on most of these tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

311 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 301 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1905.05583 · arXiv 1905.05583

@misc{sun2019how,
  title = {{How to Fine-Tune BERT for Text Classification?}},
  author = {Sun, Chi and Qiu, Xipeng and Xu, Yige and Huang, Xuanjing},
  year = {2019},
  doi = {10.48550/arxiv.1905.05583},
  eprint = {1905.05583},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1905.05583}
}

References

38 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rich Caruana. 1993. Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the Tenth International Conference on Machine Learning. find
  2. Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. 2017. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. arXiv preprint arXiv:1711.02257. — [arXiv:1711.02257](https://arxiv.org/abs/1711.02257) find
  3. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, pages 160–167. ACM. find
  4. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364. find our card
  5. Alexis Conneau, Holger Schwenk, Loıc Barrault, and Yann Lecun. 2016. Very deep convolutional networks for natural language processing. arXiv preprint arXiv:1606.01781, 2. — [arXiv:1606.01781](https://arxiv.org/abs/1606.01781) find
  6. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in neural information processing systems, pages 3079–3087. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805. find
  8. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. arXiv preprint arXiv:1801.06146. find
  9. Rie Johnson and Tong Zhang. 2017. Deep pyramid convolutional neural networks for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 562–570. find
  10. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. arXiv preprint arXiv:1404.2188. find our card
  11. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302. find
  12. Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International conference on machine learning, pages 1188– 1196. find our card
  13. Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130. find our card
  14. Liyuan Liu, Jingbo Shang, Xiang Ren, Frank Fangzheng Xu, Huan Gui, Jian Peng, and Jiawei Han. 2018. Empower sequence labeling with task-aware neural language model. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  15. Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Recurrent neural network for text classification with multi-task learning. arXiv preprint arXiv:1605.05101. — [arXiv:1605.05101](https://arxiv.org/abs/1605.05101) find
  16. Xiaodong Liu, Jianfeng Gao, Xiaodong He, Li Deng, Kevin Duh, and Ye-Yi Wang. 2015. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. find
  17. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504. find our card
  18. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. arXiv preprint arXiv:1803.02893. find our card
  19. Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies-volume 1, pages 142–150. Association… find our card
  20. Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. 2017. Learned in translation: Contextualized word vectors. In Advances in Neural Information Processing Systems, pages 6294–6305. find our card
  21. Michael McCloskey and Neal J Cohen. 1989. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pages 109– 165. Elsevier. find
  22. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  23. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543. find
  24. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. arXiv preprint arXiv:1802.05365. find
  25. Chao Qiao, Bo Huang, Guocheng Niu, Daren Li, Daxiang Dong, Wei He, Dianhai Yu, and Hua Wu. 2018. Anew method of region embedding for text classification. In International Conference on Learning Representations. find
  26. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/languageunsupervised/language understanding paper. pdf. find
  27. Marek Rei. 2017. Semi-supervised multitask learning for sequence labeling. arXiv preprint arXiv:1704.07156. find
  28. Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Neural speed reading via skim-rnn. arXiv preprint arXiv:1711.02085. find
  29. Dinghan Shen, Yizhe Zhang, Ricardo Henao, Qinliang Su, and Lawrence Carin. 2018. Deconvolutional latent-variable model for text sequence matching. In Thirty-Second AAAI Conference on Artificial Intelligence. find
  30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008. find
  31. Ellen M Voorhees and Dawn M Tice. 1999. The trec-8 question answering track evaluation. In TREC, volume 1999, page 82. Citeseer. find
  32. Canhui Wang, Min Zhang, Shaoping Ma, and Liyun Ru. 2008. Automatic online news issue construction in web environment. In Proceedings of the 17th international conference on World Wide Web, pages 457–466. ACM. find
  33. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google’s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144. find
  34. Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 14… find our card
  35. Dani Yogatama, Chris Dyer, Wang Ling, and Phil Blunsom. 2017. Generative and discriminative text classification with recurrent neural networks. arXiv preprint arXiv:1703.01898. — [arXiv:1703.01898](https://arxiv.org/abs/1703.01898) find
  36. Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Advances in neural information processing systems, pages 3320–3328. find
  37. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649–657. find our card
  38. Yizhe Zhang, Dinghan Shen, Guoyin Wang, Zhe Gan, Ricardo Henao, and Lawrence Carin. 2017. Deconvolutional paragraph representation learning. In Advances in Neural Information Processing Systems, pages 4169–4179. find