Elamur Research⌕
☰
home / papers / 2019 / BAM! Born-Again Multi-Task Networks for Natural Language Understanding

BAM! Born-Again Multi-Task Networks for Natural Language Understanding

Summary

Multi-task neural models in NLP often fail to match the accuracy of separately trained single-task models. This paper introduces Born-Again Multi-tasking (BAM), which uses knowledge distillation to address that gap. First, a separate BERT-Large model is fine-tuned for each task. These models then act as teachers for one shared multi-task student of identical architecture. The authors also add teacher annealing: the training target begins as the teacher's predicted distribution and shifts linearly toward the gold label. This lets the student learn from the richer soft signal early on and still exceed its teachers later.

The method is evaluated on the GLUE benchmark, and dev results are reported as medians over at least 20 random seeds. Single→Multi distillation matches or beats standard single-task and multi-task training on every task except STS-B, the only regression task. The result matters because a single model that handles all tasks reliably is simpler and cheaper to deploy than a collection of per-task models.

Key points

  • Dev results: average GLUE dev scores (excluding WNLI) were 84.0 for Single, 85.5 for Multi, and 86.0 for Single→Multi. Single→Multi also outperformed Single→Single and Multi→Multi distillation.
  • Test results: BERT-Large + BAM reached 82.3 on the GLUE test leaderboard, compared with 80.5 for BERT-Large. This was achieved without ensembling and without single-task fine-tuning, which added only +0.1 on top of BAM.
  • Ablations: dropping teacher annealing cost 0.5 points, and pure distillation (λ = 0) did no better than plain multi-task training. Removing task sampling (proportional to |D|^0.75) cost 0.4 points, and removing layer-wise learning rates cost 0.3.
  • RTE task combinations: RTE alone scored 70.0. Adding unrelated tasks (QQP, CoLA, SST) raised it to 75.1, while adding MNLI raised it to 83.4. This points to both regularization and transfer between related tasks as sources of gain.
  • Parameter sharing: under 0.01% of the parameters are task-specific, and repeating the distillation for extra rounds brought no significant improvement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

71 papers of our library cite it (of the 140884 whose bibliographies we read); 11 of them have a page here, 60 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1595

@inproceedings{clark2019bam,
  title = {{BAM! Born-Again Multi-Task Networks for Natural Language Understanding}},
  author = {Clark, Kevin and Luong, Minh-Thang and Khandelwal, Urvashi and Manning, Christopher D. and Le, Quoc V.},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1595},
  url = {https://doi.org/10.18653/v1/p19-1595}
}

References

44 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Ba and Rich Caruana. 2014. Do deep nets really need to be deep? In NIPS. find
  2. Joachim Bingel and Anders Søgaard. 2017. Identifying beneficial task relations for multi-task learning in deep neural networks. In EACL. find our card
  3. Samuel R Bowman, Ellie Pavlick, Edouard Grave, Benjamin Van Durme, Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R Thomas McCoy, Roma Patel, et al. 2018. Looking for ELMo’s friends: Sentence-level pretraining beyond language modeling. arXiv preprint arXiv:1812.10860. — [arXiv:1812.10860](https://arxiv.org/abs/1812.10860) find
  4. Cristian Bucilu, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In SIGKDD. find
  5. Rich Caruana. 1997. Multitask learning. Machine Learning. find
  6. Daniel M. Cer, Mona T. Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval@ACL. find
  7. Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In ICML. find
  8. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In NIPS. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT. find
  10. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In IWP@IJCNLP. find our card
  11. Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. 2018. Born again neural networks. In ICML. find
  12. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B. Dolan. 2007. The third pascal recognizing textual entailment challenge. In ACL-PASCAL@ACL. find
  13. Braden Hancock, Clara McCreery, Ines Chami, Vincent Chen, Sen Wu, Jared Dunnmon, Paroma Varma, Max Lam, and Chris R. 2019. Massive multi-task learning with snorkel metal: Bringing more supervision to bear. find
  14. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. 2017. A joint many-task model: Growing a neural network for multiple NLP tasks. In EMNLP. find our card
  15. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531. find our card
  16. Sture Holm. 1979. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics. find
  17. Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In ACL. find
  18. Shankar Iyer, Nikhil Dandekar, and Kornl Csernai. 2017. First quora dataset release: Question pairs. find
  19. Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Unifying question answering and text classification via span extraction. arXiv preprint arXiv:1904.09286. — [arXiv:1904.09286](https://arxiv.org/abs/1904.09286) find
  20. Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In EMNLP. find our card
  21. Adhiguna Kuncoro, Miguel Ballesteros, Lingpeng Kong, Chris Dyer, and Noah A. Smith. 2016. Distilling an ensemble of greedy dependency parsers into one mst parser. In EMNLP. find
  22. Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2017. Adversarial multi-task learning for text classification. In ACL. find our card
  23. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Improving multi-task deep neural networks via knowledge distillation for natural language understanding. arXiv preprint arXiv:1904.09482. — [arXiv:1904.09482](https://arxiv.org/abs/1904.09482) find
  24. Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019b. Multi-task deep neural networks for natural language understanding. arXiv preprint arXiv:1901.11504. find our card
  25. Minh-Thang Luong, Quoc V. Le, Ilya Sutskever, Oriol Vinyals, and Lukasz Kaiser. 2016. Multi-task sequence to sequence learning. In ICLR. find our card
  26. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730. find our card
  27. Lili Mou, Ran Jia, Yan Xu, Ge Li, Lu Zhang, and Zhi Jin. 2016. Distilling word embeddings: An encoding approach. In CIKM. find
  28. Emilio Parisotto, Jimmy Ba, and Ruslan Salakhutdinov. 2016. Actor-mimic: Deep multitask and transfer reinforcement learning. In ICLR. find
  29. Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In NAACL-HLT. find
  30. Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. find our card
  31. Barbara Plank and Héctor Martı́nez Alonso. 2017. When is multitask learning effective? Semantic sequence prediction under varying data conditions. In EACL. find
  32. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy S. Liang. 2016. Squad: 100, 000+ questions for machine comprehension of text. In EMNLP. find our card
  33. Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098. find our card
  34. Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard. 2019. Latent multi-task architecture learning. In AAAI. find
  35. Victor Sanh, Thomas Wolf, and Sebastian Ruder. 2019. A hierarchical multi-task approach for learning embeddings from semantic tasks. In AAAI. find
  36. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In ACL. find
  37. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP. find
  38. Anders Søgaard and Yoav Goldberg. 2016. Deep multi-task learning with low level tasks supervised at lower layers. In ACL. find our card
  39. Xu Tan, Yi Ren, Di He, Tao Qin, and Tie-Yan Liu. 2019. Multilingual neural machine translation with knowledge distillation. In ICLR. find
  40. Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. 2017. Distral: Robust multitask reinforcement learning. In NIPS. find
  41. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS. find
  42. Alex Wang, Amapreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR. find our card
  43. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2018. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471. find
  44. Adina Williams, Nikita Nangia, and Samuel R. Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/clark-2019-bam-born-again-multi-task/