Elamur Research
☰
home / papers / 2017 / Learning multiple visual domains with residual adapters

Learning multiple visual domains with residual adapters

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1705.08045

Summary

The paper asks whether one deep network can serve many visually unrelated image domains, such as handwritten characters, traffic signs and flower species, without keeping a separate full model for each. The authors start from ResNet and insert small residual adapter modules. These are lightweight domain-specific layers placed beside large shared filters, and they hold less than 10% of the parameters. They show that schemes in which one network predicts another network's weights reduce to this kind of adapter layer.

Training proceeds as follows:

  • The network and adapters are first pretrained on ImageNet.
  • The shared weights are then frozen, and only the adapters are trained for each new domain, so earlier domains are not forgotten.
  • Optionally, all ten tasks are trained jointly end-to-end.

To evaluate the approach, they introduce the Visual Decathlon Challenge, a ten-dataset benchmark. Its score rewards models that do well on every domain rather than just on average. With weight decay tuned per domain, adapters match or beat full fine-tuning while using about twice the capacity of a single ResNet. This makes compact multi-domain models practical.

Key points

  • The Visual Decathlon benchmark covers ten datasets: Aircraft, CIFAR-100, Daimler Pedestrians, Describable Textures (DTD), German Traffic Signs, ILSVRC 2012 (ImageNet), VGG-Flowers, OmniGlot, SVHN and UCF101 Dynamic Images. The maximum score is 10,000 points. Ten separately fine-tuned networks score 2,500.
  • Tuned residual adapters score 2643 points at 2× the size of one ResNet. With a single shared weight decay, average accuracy is still high (73.88%), but the score falls to 2118 because DTD and Flowers perform poorly.
  • Weight decay should depend on dataset size. Higher decay (0.002 or 0.005, versus the default 0.0005) helps small datasets like Flowers, while larger datasets prefer the lower value.
  • Controls perform worse. Fine-tuning only the last two convolutional layers reaches 64.7% mean accuracy, versus 73.9% for adapters. Adapters without the residual connection reach 70.3% and converge more slowly.
  • Domain labels are not required at test time. A small ResNet predicts the domain with 99.8% accuracy, causing only a marginal drop. A larger adapter model built on ResNet28 (12× capacity) beats fine-tuning by about 600 points and exceeds the prior CIFAR-100 state of the art (81.2%) by 3.5 points.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

156 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 147 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1705.08045 · arXiv 1705.08045

@misc{rebuffi2017learning,
  title = {{Learning multiple visual domains with residual adapters}},
  author = {Rebuffi, Sylvestre-Alvise and Bilen, Hakan and Vedaldi, Andrea},
  year = {2017},
  doi = {10.48550/arxiv.1705.08045},
  eprint = {1705.08045},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1705.08045}
}

References

42 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Argyriou, T. Evgeniou, and M Pontil. Multi-task feature learning. In Proc. NIPS, volume 19, page 41. MIT; 1998, 2007. find
  2. L. Bertinetto, J. F. Henriques, J. Valmadre, P. Torr, and A. Vedaldi. Learning feed-forward one-shot learners. In Proc. NIPS, pages 523–531, 2016. find
  3. H. Bilen, B. Fernando, E. Gavves, A. Vedaldi, and S. Gould. Dynamic image networks for action recognition. In Proc. CVPR, 2016. find
  4. H. Bilen and A. Vedaldi. Integrated perception with recurrent multi-task neural networks. In Proc. NIPS, 2016. find
  5. H. Bilen and A. Vedaldi. Universal representations: The missing link between faces, text, planktons, and cat breeds. arXiv preprint arXiv:1701.07275, 2017. — [arXiv:1701.07275](https://arxiv.org/abs/1701.07275) find
  6. R. Caruana. Multitask learning. Machine Learning, 28, 1997. find
  7. M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proc. CVPR, 2014. find
  8. R. Collobert and J. Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In icml, pages 160–167. ACM, 2008. find
  9. H. Daumé III. Frustratingly easy domain adaptation. ACL 2007, page 256, 2007. find
  10. T. Evgeniou and M. Pontil. Regularized multi–task learning. In SIGKDD, pages 109–117. ACM, 2004. find
  11. R. M. French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128–135, 1999. find
  12. Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. Proc. ICML, 2015. find
  13. K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In Proc. ECCV, pages 630–645. Springer, 2016. find
  14. J. T. Huang, J. Li, D. Yu, L. Deng, and Y. Gong. Cross-language knowledge transfer using multilingual deep neural network with shared hidden layers. In ICASSP, pages 7304–7308, 2013. find
  15. S. Ioffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR, 2015. find
  16. X. Jia, B. De Brabandere, T. Tuytelaars, and L. Gool. Dynamic filter networks. In Proc. NIPS, pages 667–675, 2016. find
  17. J. Kirkpatrick, E. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. National Academy of Sciences, 2017. find
  18. I. Kokkinos. Ubernet: Training auniversal’convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. Proc. CVPR, 2017. find
  19. A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. 2009. find
  20. B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338, 2015. find
  21. Z. Li and D. Hoiem. Learning without forgetting. In Proc. ECCV, pages 614–629, 2016. find
  22. X. Liu, J. Gao, X. He, L. Deng, K. Duh, and Y. Wang. Representation learning using multi-task deep neural networks for semantic classification and information retrieval. In HLT-NAACL, pages 912–921, 2015. find
  23. M. Long, H. Zhu, J. Wang, and M. I. Jordan. Unsupervised Domain Adaptation with Residual Transfer Networks. In Proc. NIPS, pages 136–144, 2016. find
  24. S. Maji, J. Kannala, E. Rahtu, M. Blaschko, and A. Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013. find
  25. T. Mitchell. Never-ending learning. Technical report, DTIC Document, 2010. find
  26. S. Munder and D. M. Gavrila. An experimental study on pedestrian classification. PAMI, 28(11):1863–1868, 2006. find
  27. Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011. find
  28. M-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In ICCVGIP, Dec 2008. find
  29. Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. CNN features off-the-shelf: an astounding baseline for recognition. In CVPR DeepVision Workshop, 2014. find
  30. S. A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert. icarl: Incremental classifier and representation learning. In Proc. CVPR, 2017. find
  31. Amir Rosenfeld and John K Tsotsos. Incremental learning through deep adaptation. arXiv preprint arXiv:1705.04228, 2017. — [arXiv:1705.04228](https://arxiv.org/abs/1705.04228) find
  32. O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and K. Fei-Fei. Imagenet large scale visual recognition challenge, 2014. find our card
  33. A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, and R. Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016. find
  34. J. Schmidhuber. Learning to control fast-weight memories: An alternative to dynamic recurrent networks. Neural Computation, 4(1):131–139, 1992. find
  35. K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402, 2012. — [arXiv:1212.0402](https://arxiv.org/abs/1212.0402) find
  36. J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, 32(0):323–332, 2012. find
  37. A. V. Terekhov, G. Montone, and J. K. O’Regan. Knowledge transfer in deep block-modular neural networks. In Biomimetic and Biohybrid Systems, pages 268–279, 2015. find
  38. S. Thrun. Lifelong learning algorithms. In Learning to learn, pages 181–209. Springer, 1998. find
  39. E. Tzeng, J. Hoffman, T. Darrell, and K. Saenko. Simultaneous deep transfer across domains and tasks. In Proc. CVPR, pages 4068–4076, 2015. find
  40. Sergey Zagoruyko and Nikos Komodakis. arXiv:1605.07146, 2016. — [arXiv:1605.07146](https://arxiv.org/abs/1605.07146) find
  41. T. Zhang, B. Ghanem, S. Liu, and N. Ahuja. Robust visual tracking via structured multi-task sparse learning. IJCV, 101(2):367–383, 2013. find
  42. Z. Zhang, P. Luo, C. C. Loy, and X. Tang. Facial landmark detection by deep multi-task learning. In Proc. ECCV, 2014. Wide residual networks. arXiv preprint find