Elamur Research
☰
home / papers / 2017 / An Overview of Multi-Task Learning in Deep Neural Networks

An Overview of Multi-Task Learning in Deep Neural Networks

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1706.05098

Summary

Optimizing a model for a single objective ignores useful signal from related tasks. This survey by Sebastian Ruder, first published as a blog post, explains how multi-task learning (MTL) uses that signal in deep neural networks. It offers practical guidance and introduces no new data, models or experiments. It compares the two dominant designs. In hard parameter sharing, tasks share hidden layers but keep separate output heads. In soft parameter sharing, each task has its own network, and the distance between their parameters is penalized. The survey then outlines the mechanisms believed to make MTL effective. It reviews earlier MTL research on linear, kernel and Bayesian models, focusing on cross-task sparsity and modelling task relations. It also covers newer deep approaches and catalogues kinds of auxiliary tasks. The main conclusion is that the roughly two-decade-old hard sharing scheme still dominates, while methods that learn what to share look promising. A principled theory of task relatedness is still missing, and that gap is the main open problem when selecting auxiliary tasks.

Key points

  • Hard parameter sharing goes back to Caruana (1993). Baxter (1997) showed that the overfitting risk for shared parameters is smaller by an order of N (the number of tasks) than for task-specific output layers.
  • Soft parameter sharing regularizes the distance between per-task models. Examples include ℓ2 distance (Duong et al., 2015) and the trace norm (Yang and Hospedales, 2017).
  • Five proposed mechanisms explain why MTL works: implicit data augmentation, attention focusing, eavesdropping (e.g., via hints), representation bias, and regularization (lower Rademacher complexity).
  • Non-neural MTL relies on block-sparse ℓ1/ℓq norms. Illustrations are group lasso, as used in the work of Yuan and Lin (2006) and of Argyriou and Pontil (2007), as well as Jalali et al. (2010), who split the parameter matrix as A = B + S to combine block-wise and element-wise sparsity.
  • Auxiliary task types include:
    • predicting lane markings or facial landmarks;
    • smoother, less quantized labels;
    • features used as outputs rather than inputs;
    • "future" information available only after prediction time;
    • language modelling or autoencoder objectives.
  • Choosing auxiliary tasks: in NLP sequence tagging, auxiliary tasks with compact, uniform label distributions tend to help (Alonso and Plank, 2017). Gains are also more likely when the main task plateaus quickly and the auxiliary task does not (Bingel and Søgaard, 2017).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

241 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 237 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1706.05098 · arXiv 1706.05098

@misc{ruder2017overview,
  title = {{An Overview of Multi-Task Learning in Deep Neural Networks}},
  author = {Ruder, Sebastian},
  year = {2017},
  doi = {10.48550/arxiv.1706.05098},
  eprint = {1706.05098},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1706.05098}
}

References

56 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abu-Mostafa, Y. S. (1990). Learning from hints in neural networks. Journal of Complexity, 6(2):192–198. find
  2. Alonso, H. M. and Plank, B. (2017). When is multitask learning effective? Multitask learning for semantic sequence prediction under varying data conditions. In EACL. find
  3. Ando, R. K. and Tong, Z. (2005). A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data. Journal of Machine Learning Research, 6:1817–1853. find
  4. Argyriou, A. and Pontil, M. (2007). Multi-Task Feature Learning. In Advances in Neural Information Processing Systems. find
  5. Arık, S. Ö., Chrzanowski, M., Coates, A., Diamos, G., Gibiansky, A., Kang, Y., Li, X., Miller, J., Raiman, J., Sengupta, S., and Shoeybi, M. (2017). Deep Voice: Real-time Neural Text-to-Speech. In ICML 2017. find
  6. Bakker, B. and Heskes, T. (2003). Task Clustering and Gating for BayesianMultitask Learning. Journal of Machine Learning Research, 1(1):83–99. find
  7. Baxter, J. (1997). A Bayesian/information theoretic model of learning to learn via multiple task sampling. Machine Learning, 28:7–39. find
  8. Baxter, J. (2000). A Model of Inductive Bias Learning. Journal of Artificial Intelligence Research, 12:149–198. find
  9. Ben-David, S. and Schuller, R. (2003). Exploiting task relatedness for multiple task learning. Learning Theory and Kernel Machines, pages 567–580. find
  10. Bingel, J. and Søgaard, A. (2017). Identifying beneficial task relations for multi-task learning in deep neural networks. In EACL. find
  11. Caruana, R. (1993). Multitask learning: A knowledge-based source of inductive bias. In Proceedings of the Tenth International Conference on Machine Learning. find
  12. Caruana, R. (1998). Multitask Learning. Autonomous Agents and Multi-Agent Systems, 27(1):95–133. find
  13. Caruana, R. and de Sa, V. R. (1997). Promoting poor features to supervisors: Some inputs work better as outputs. Advances in Neural Information Processing Systems 9: Proceedings of The 1996 Conference, 9:389. find
  14. Cavallanti, G., Cesa-Bianchi, N., and Gentile, C. (2010). Linear Algorithms for Online Multitask Classification. Journal of Machine Learning Research, 11:2901–2934. find
  15. Chen, X., Kim, S., Lin, Q., Carbonell, J. G., and Xing, E. P. (2010). Graph-Structured Multi-task Regression and an Efficient Optimization Method for General Fused Lasso. pages 1–21. find
  16. Cheng, H., Fang, H., and Ostendorf, M. (2015). Open-Domain Name Error Detection using a Multi-Task RNN. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 737–746. find
  17. Collobert, R. and Weston, J. (2008). A unified architecture for natural language processing. Proceedings of the 25th international conference on Machine learning - ICML ’08, 20(1):160–167. find
  18. Crammer, K. and Mansour, Y. (2012). Learning Multiple Tasks Using Shared Hypotheses. Neural Information Processing Systems (NIPS), pages 1484–1492. find
  19. Daumé III, H. (2009). Bayesian multitask learning with latent hierarchies. pages 135–142. find
  20. Deng, L., Hinton, G. E., and Kingsbury, B. (2013). New types of deep neural network learning for speech recognition and related applications: An overview. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8599–8603. find
  21. Duong, L., Cohn, T., Bird, S., and Cook, P. (2015). Low Resource Dependency Parsing: Cross-lingual Parameter Sharing in a Neural Network Parser. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Proc… find
  22. Evgeniou, T., Micchelli, C. A., and Pontil, M. (2005). Learning multiple tasks with kernel methods. Journal of Machine Learning Research, 6:615–637. find
  23. Evgeniou, T. and Pontil, M. (2004). Regularized multi-task learning. International Conference on Knowledge Discovery and Data Mining, page 109. find
  24. Ganin, Y. and Lempitsky, V. (2015). Unsupervised Domain Adaptation by Backpropagation. In Proceedings of the 32nd International Conference on Machine Learning., volume 37. find
  25. Girshick, R. (2015). Fast R-CNN. In Proceedings of the IEEE International Conference on Computer Vision, pages 1440–1448. find
  26. Hashimoto, K., Xiong, C., Tsuruoka, Y., and Socher, R. (2016). A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks. find our card
  27. Heskes, T. (2000). Empirical Bayes for Learning to Learn. Proceedings of the Seventeenth International Conference on Machine Learning, pages 367–364. find
  28. Jacob, L., Vert, J.-p., Bach, F. R., and Vert, J.-p. (2009). Clustered Multi-Task Learning: A Convex Formulation. Advances in Neural Information Processing Systems 21, pages 745–752. find
  29. Jalali, A., Ravikumar, P., Sanghavi, S., and Ruan, C. (2010). A Dirty Model for Multi-task Learning. Advances in Neural Information Processing Systems. find
  30. Kang, Z., Grauman, K., and Sha, F. (2011). Learning with whom to share in multi-task feature learning. Proceedings of the 28th International Conference on Machine Learning, (4):4–5. find
  31. Kendall, A., Gal, Y., and Cipolla, R. (2017). Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics. find
  32. Kim, S. and Xing, E. P. (2010). Tree-Guided Group Lasso for Multi-Task Regression with Structured Sparsity. 27th International Conference on Machine Learning, pages 1–14. find
  33. Kumar, A. and Daumé III, H. (2012). Learning Task Grouping and Overlap in Multi-task Learning. Proceedings of the 29th International Conference on Machine Learning, pages 1383–1390. find
  34. Lawrence, N. D. and Platt, J. C. (2004). Learning to learn with the informative vector machine. Twenty-first international conference on Machine learning - ICML ’04, page 65. find
  35. Liu, S., Pan, S. J., and Ho, Q. (2016). Distributed Multi-task Relationship Learning. In Proceedings of the 19th International Conference on Artificial Intelligence and Statistics (AISTATS), pages 751–760. find
  36. Liu, X., Gao, J., He, X., Deng, L., Duh, K., and Wang, Y.-Y. (2015). Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval. NAACL-2015, pages 912–921. find
  37. Long, M. and Wang, J. (2015). Learning Multiple Tasks with Deep Relationship Networks. arXiv preprint arXiv:1506.02117. — [arXiv:1506.02117](https://arxiv.org/abs/1506.02117) find
  38. Lounici, K., Pontil, M., Tsybakov, A. B., and van de Geer, S. (2009). Taking Advantage of Sparsity in Multi-Task Learning. Stat, (1). find
  39. Lu, Y., Kumar, A., Zhai, S., Cheng, Y., Javidi, T., and Feris, R. (2016). Fully-adaptive Feature Sharing in Multi-Task Networks with Applications in Person Attribute Classification. find
  40. Misra, I., Shrivastava, A., Gupta, A., and Hebert, M. (2016). Cross-stitch Networks for Multi-task Learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. find
  41. Negahban, S. and Wainwright, M. J. (2008). Joint support recovery under high-dimensional scaling: Benefits and perils of $ell\_{1,infty}$-regularization. Advances in Neural Information Processing Systems, pages 1161–1168. find
  42. Ramsundar, B., Kearnes, S., Riley, P., Webster, D., Konerding, D., and Pande, V. (2015). Massively Multitask Networks for Drug Discovery. find
  43. Rei, M. (2017). Semi-supervised Multitask Learning for Sequence Labeling. In Proceedings of ACL 2017. find
  44. Ruder, S., Bingel, J., Augenstein, I., and Søgaard, A. (2017). Sluice networks: Learning what to share between loosely related tasks. find
  45. Saha, A., Rai, P., Daumé, H., and Venkatasubramanian, S. (2011). Online learning of multiple tasks and their relationships. Journal of Machine Learning Research, 15:643–651. find
  46. Søgaard, A. and Goldberg, Y. (2016). Deep multi-task learning with low level tasks supervised at lower layers. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 231–235. find our card
  47. Thrun, S. and O’Sullivan, J. (1996). Discovering Structure in Multiple Learning Tasks: The TC Algorithm. Proceedings of the Thirteenth International Conference on Machine Learning, 28(1):5–5. find
  48. Xue, Y., Liao, X., Carin, L., and Krishnapuram, B. (2007). Multi-Task Learning for Classification with Dirichlet Process Priors. Journal of Machine Learning Research, 8:35–63. find
  49. Yang, Y. and Hospedales, T. (2017a). Deep Multi-task Representation Learning: A Tensor Factorisation Approach. In Proceedings of ICLR 2017. find
  50. Yang, Y. and Hospedales, T. M. (2017b). Trace Norm Regularised Deep Multi-Task Learning. In Workshop track - ICLR 2017. find
  51. Yu, J. and Jiang, J. (2016). Learning Sentence Embeddings with Auxiliary Tasks for Cross-Domain Sentiment Classification. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP2016), pages 236–246. find
  52. Yu, K., Tresp, V., and Schwaighofer, A. (2005). Learning Gaussian processes from multiple tasks. Proceedings of the International Conference on Machine Learning (ICML), 22:1012–1019. find
  53. Yuan, M. and Lin, Y. (2006). Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1):49–67. find
  54. Zhang, C. H. and Huang, J. (2008). The sparsity and bias of the lasso selection in high-dimensional linear regression. Annals of Statistics, 36(4):1567–1594. find
  55. Zhang, Y. and Yeung, D.-y. (2010). A Convex Formulation for Learning Task Relationships in Multi-Task Learning. Uai, pages 733–442. find
  56. Zhang, Z., Luo, P., Loy, C. C., and Tang, X. (2014). Facial Landmark Detection by Deep Multi-task Learning. In European Conference on Computer Vision, pages 94–108. find