Elamur Research
☰
home / papers / 2020 / Gradient Surgery for Multi-Task Learning

Gradient Surgery for Multi-Task Learning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2001.06782

Summary

Training one network on many tasks should save data, but joint optimization often does worse than separately trained models, and the reason is unclear. Yu et al. argue that trouble appears when three factors coincide, a "tragic triad": task gradients with negative cosine similarity, a large magnitude gap that lets one task dominate the update, and strong positive curvature that makes the optimizer overrate gains for the dominant task and underrate losses for the weaker one. To remove the first factor, they propose PCGrad (projecting conflicting gradients). When two task gradients conflict, each loses its component along the other. Gradients that agree are left unchanged. Because only the gradient step changes, PCGrad works with any architecture and can be stacked on existing multi-task models. The authors prove local conditions under which it beats plain multi-task gradient descent. They test it on a 2D toy landscape, multi-task CIFAR classification, multi-objective scene understanding, multi-task RL and goal-conditioned RL, and report gains in data efficiency, speed and final performance. The result is a cheap, drop-in remedy for interference between tasks.

Key points

  • Conflict is defined as cos φ_ij < 0 between task gradients. Magnitude imbalance is measured by a gradient magnitude similarity Φ, which equals 1 for equal norms and approaches 0 as the norms diverge. Curvature is the averaged Hessian term along the multi-task gradient.
  • On a 2D two-valley example, Adam stalls exactly where all three triad conditions hold. Adam combined with PCGrad continues to the optimum.
  • The paper reports a more than 30% absolute improvement on multi-task reinforcement learning problems compared with prior approaches.
  • PCGrad is model-agnostic. In supervised multi-task settings it adds further gains when combined with earlier state-of-the-art architectures.
  • Experiments (Section 5.3) suggest the triad also occurs in deep networks. The conditions hold frequently until the learner starts solving the harder task. The work appeared at NeurIPS 2020 (Stanford, UC Berkeley, Robotics at Google).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

122 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 121 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2001.06782 · arXiv 2001.06782

@misc{yu2020gradient,
  title = {{Gradient Surgery for Multi-Task Learning}},
  author = {Yu, Tianhe and Kumar, Saurabh and Gupta, Abhishek and Levine, Sergey and Hausman, Karol and Finn, Chelsea},
  year = {2020},
  doi = {10.48550/arxiv.2001.06782},
  eprint = {2001.06782},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2001.06782}
}

References

63 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. Segnet: A deep convolutional encoder-decoder architecture for image segmentation. IEEE transactions on pattern analysis and machine intelligence, 39(12), 2017. find
  2. Bart Bakker and Tom Heskes. Task clustering and gating for bayesian multitask learning. J. Mach. Learn. Res., 2003. find
  3. Hakan Bilen and Andrea Vedaldi. Integrated perception with recurrent multi-task neural networks. In Advances in Neural Information Processing Systems, 2016. find
  4. Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004. find
  5. Paul H Calamai and Jorge J Moré. Projected gradient methods for linearly constrained problems. Mathematical programming, 39(1), 1987. find
  6. Rich Caruana. Multitask learning. Machine Learning, 1997. find
  7. Arslan Chaudhry, Marc’Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. arXiv:1812.00420, 2018. find
  8. Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. arXiv:1711.02257, 2017. — [arXiv:1711.02257](https://arxiv.org/abs/1711.02257) find
  9. Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, 2018. find
  10. Felipe Codevilla, Matthias Miiller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-to-end driving via conditional imitation learning. In International Conference on Robotics and Automation (ICRA), 2018. find
  11. Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In International Conference on Machine Learning, 2008. find
  12. Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–322… find
  13. Wojciech M. Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant M. Jayakumar, Grzegorz Swirszcz, and Max Jaderberg. Distilling policy distillation. In International Conference on Artificial Intelligence and Statistics, AISTATS, 2019. find
  14. Coline Devin, Abhishek Gupta, Trevor Darrell, Pieter Abbeel, and Sergey Levine. Learning modular neural network policies for multi-task and multi-robot transfer. CoRR, abs/1609.07088, 2016. — [arXiv:1609.07088](https://arxiv.org/abs/1609.07088) find
  15. Daxiang Dong, Hua Wu, Wei He, Dianhai Yu, and Haifeng Wang. Multi-task learning for multiple language translation. In Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), 2015. find our card
  16. Yunshu Du, Wojciech M. Czarnecki, Siddhant M. Jayakumar, Razvan Pascanu, and Balaji Lakshminarayanan. Adapting auxiliary losses using gradient similarity. CoRR, abs/1812.02224, 2018. — [arXiv:1812.02224](https://arxiv.org/abs/1812.02224) find
  17. Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference on Machin… find
  18. Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal gradient descent for continual learning. arXiv preprint arXiv:1910.07104, 2019. find
  19. Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. CoRR, abs/1701.08734, 2017. URL http://arxiv.org/abs/1701.08734. ↗
  20. William Fulton. Eigenvalues, invariant factors, highest weights, and schubert calculus. Bulletin of the American Mathematical Society, 37(3):209–249, 2000. find
  21. Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide-and-conquer reinforcement learning. CoRR, abs/1711.09874, 2017. — [arXiv:1711.09874](https://arxiv.org/abs/1711.09874) find
  22. Ian J Goodfellow, Oriol Vinyals, and Andrew M Saxe. Qualitatively characterizing neural network optimization problems. arXiv:1412.6544, 2014. — [arXiv:1412.6544](https://arxiv.org/abs/1412.6544) find
  23. Yunhui Guo, Mingrui Liu, Tianbao Yang, and Tajana Rosing. Improved schemes for episodic memory-based lifelong learning, 2020. find
  24. Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. International Conference on Machine Learning, 2018. find
  25. Karol Hausman, Jost Tobias Springenberg, Ziyu Wang, Nicolas Heess, and Martin Riedmiller. Learning an embedding space for transferable robot skills. 2018. find
  26. Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, 2019. find
  27. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv:1503.02531, 2015. find our card
  28. Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Computer Vision and Pattern Recognition, 2018. find
  29. Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, 2018. find
  30. Diederik P Kingma and Jimmy Ba. arXiv:1412.6980, 2014. find
  31. Iasonas Kokkinos. Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In Computer Vision and Pattern Recognition, 2017. find
  32. Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1), 2016. find
  33. Shikun Liu, Edward Johns, and Andrew J. Davison. End-to-end multi-task learning with attention. CoRR, abs/1803.10704, 2018. — [arXiv:1803.10704](https://arxiv.org/abs/1803.10704) find
  34. Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015. find
  35. Mingsheng Long and Jianmin Wang. Learning multiple tasks with deep relationship networks. arXiv:1506.02117, 2, 2015. — [arXiv:1506.02117](https://arxiv.org/abs/1506.02117) find
  36. David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, 2017. find
  37. Kevis-Kokitsi Maninis, Ilija Radosavovic, and Iasonas Kokkinos. Attentive single-tasking of multiple tasks. CoRR, abs/1904.08918, 2019. — [arXiv:1904.08918](https://arxiv.org/abs/1904.08918) find
  38. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv:1806.08730, 2018. find our card
  39. Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Computer Vision and Pattern Recognition, 2016. find
  40. Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Conference on Computer Vision and Pattern Recognition, CVPR, 2016. find
  41. Angelia Nedic and Asuman Ozdaglar. Distributed subgradient methods for multi-agent optimization. IEEE Transactions on Automatic Control, 54(1), 2009. find
  42. Emilio Parisotto, Jimmy Lei Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv:1511.06342, 2015. — [arXiv:1511.06342](https://arxiv.org/abs/1511.06342) find
  43. Boris T Polyak. Some methods of speeding up the convergence of iteration methods. USSR Computational Mathematics and Mathematical Physics, 4(5):1–17, 1964. find
  44. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI Blog, 1(8), 2019. find
  45. Martin Riedmiller, Roland Hafner, Thomas Lampe, Michael Neunert, Jonas Degrave, Tom Van de Wiele, Volodymyr Mnih, Nicolas Heess, and Jost Tobias Springenberg. Learning by playing-solving sparse reward tasks from scratch. arXiv:1802.10567, 2018. — [arXiv:1802.10567](https://arxiv.org/abs/1802.10567) find
  46. Clemens Rosenbaum, Tim Klinger, and Matthew Riemer. Routing networks: Adaptive selection of non-linear functions for multi-task learning. International Conference on Learning Representations (ICLR), 2018. find
  47. Clemens Rosenbaum, Ignacio Cases, Matthew Riemer, and Tim Klinger. Routing networks and the challenges of modular and compositional computation. arXiv:1904.12774, 2019. Adam: A method for stochastic optimization. — [arXiv:1904.12774](https://arxiv.org/abs/1904.12774) find
  48. Kevin Roth, Aurelien Lucchi, Sebastian Nowozin, and Thomas Hofmann. Stabilizing training of generative adversarial networks through regularization. In Advances in Neural Information Processing Systems, 2017. find
  49. Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv:1706.05098, 2017. find our card
  50. Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülçehre, Guillaume Desjardins, James Kirkpatrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy distillation. In International Conference on Learning Representations, ICLR, 2016. find
  51. Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv:1606.04671, 2016. find
  52. Tom Schaul, Diana Borsa, Joseph Modayil, and Razvan Pascanu. Ray interference: a source of plateaus in deep reinforcement learning. arXiv:1904.11455, 2019. — [arXiv:1904.11455](https://arxiv.org/abs/1904.11455) find
  53. Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In Advances in Neural Information Processing Systems, 2018. find
  54. Yuekai Sun. Notes on first-order methods for minimizing smooth functions, 2015. https://web.stanford.edu/class/msande318/notes/notes-first-order-smooth.pdf. ↗
  55. Mihai Suteu and Yike Guo. Regularizing deep multi-task networks using orthogonal gradients. arXiv preprint arXiv:1912.06844, 2019. — [arXiv:1912.06844](https://arxiv.org/abs/1912.06844) find
  56. Yee Teh, Victor Bapst, Wojciech M Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, 2017. find
  57. Simon Vandenhende, Bert De Brabandere, and Luc Van Gool. Branched multi-task networks: Deciding what layers to share. CoRR, abs/1904.02920, 2019. — [arXiv:1904.02920](https://arxiv.org/abs/1904.02920) find
  58. Aaron Wilson, Alan Fern, Soumya Ray, and Prasad Tadepalli. Multi-task reinforcement learning: a hierarchical bayesian approach. In International Conference on Machine Learning, 2007. find
  59. Markus Wulfmeier, Abbas Abdolmaleki, Roland Hafner, Jost Tobias Springenberg, Michael Neunert, Tim Hertweck, Thomas Lampe, Noah Siegel, Nicolas Heess, and Martin Riedmiller. Regularized hierarchical policies for compositional transfer in robotics. arXiv:1906.11228, 2019. — [arXiv:1906.11228](https://arxiv.org/abs/1906.11228) find
  60. Yongxin Yang and Timothy M Hospedales. Trace norm regularised deep multi-task learning. arXiv:1606.04038, 2016. — [arXiv:1606.04038](https://arxiv.org/abs/1606.04038) find
  61. Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: A benchmark and evaluation for multi-task and meta-reinforcement learning. arXiv:1910.10897, 2019. — [arXiv:1910.10897](https://arxiv.org/abs/1910.10897) find
  62. Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Computer Vision and Pattern Recognition, 2018. find
  63. Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In European conference on computer vision. Springer, 2014. find