Elamur Research⌕
☰
home / papers / 2018 / Progress & Compress: A scalable framework for continual learning

Progress & Compress: A scalable framework for continual learning

Paper · 2018 · Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki et al.

Summary

Neural networks that learn tasks one after another tend to lose earlier skills, a problem called catastrophic forgetting. Existing fixes each have a cost. Finetuning forgets. EWC can become too rigid to learn new tasks. Progressive Networks grow quadratically in size as tasks accumulate. Schwarz et al. (DeepMind/Oxford, ICML 2018) propose Progress & Compress (P&C), which uses two fixed-size columns. In the "progress" phase, an active column learns the new task without restrictions and reuses features from a frozen knowledge base through lateral adaptor connections. In the "compress" phase, the active column's behaviour is distilled into the knowledge base. A new online EWC variant protects older skills during this step, and its penalty does not grow with the number of tasks. The method needs no stored data, no task-specific parameters and no growing architecture. The authors test it on sequential Omniglot alphabet classification, Atari games and 3D maze navigation. P&C achieves the best accuracy among the scalable methods while keeping the parameter count small. This makes continual learning feasible for long task sequences.

Key points

  • Omniglot results after 5 passes: P&C reaches 82.84 ± 1.4 test accuracy. This beats online EWC (79.15), EWC (75.96), LwF (66.48) and finetuning (60.74). The less scalable baselines score higher: Progressive Nets reach 86.50 and one model per task reaches 88.34.
  • Model size on Omniglot:
    • P&C uses 659K parameters.
    • Progressive Nets use 108,000K.
    • Separate per-task models use 5,680K.
    • Original EWC uses 11,100K, because its regularisers accumulate.
  • Atari setup: six games (Space Invaders, Krull, Beamrider, Hero, Stargunner, Ms. Pac-man) are each visited 5 times, with 50m frames per visit. P&C clearly improves on several games and roughly matches EWC baselines on the others.
  • Adaptors: these are small MLPs, implemented as 1×1 convolutions in CNNs, with trainable scaling vectors. Distillation minimises the KL divergence between the active column's outputs and the knowledge base's outputs.
  • When P&C helps most: it gains most where tasks allow positive transfer. On Omniglot, online EWC alone is already competitive.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom122 works · 160,277 citations🏢Google DeepMindLondon · United Kingdom72 works · 84,326 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu and Raia Hadsell. 2018. Progress & Compress: A scalable framework for continual learning. arXiv:1805.06370.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1805.06370 · arXiv 1805.06370

@misc{schwarz2018progress,
  title = {{Progress \& Compress: A scalable framework for continual learning}},
  author = {Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki, Agnieszka Grabska-Barwinska, Yee Whye Teh, Razvan Pascanu and Raia Hadsell},
  year = {2018},
  doi = {10.48550/arxiv.1805.06370},
  eprint = {1805.06370},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1805.06370}
}

References

42 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aljundi, R., Chakravarty, P., and Tuytelaars, T. Expert gate: Lifelong learning with a network of experts. arXiv preprint arXiv:1611.06194, 2016. find
  2. Bellemare, M. G., Veness, J., and Bowling, M. Investigating contingency awareness using atari 2600 games. In AAAI, 2012. find
  3. Bengio, Y. Deep learning of representations for unsupervised and transfer learning. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning, pp. 17–36, 2012. find
  4. Berseth, G., Xie, C., Cernek, P., and Van de Panne, M. Progressive reinforcement learning with distillation for multi-skilled motion control. ICLR, 2018. find
  5. Caruana, R. Multitask learning. In Learning to learn, pp. 95–133. Springer, 1998. find
  6. Chaudhry, A., Dokania, P. K., Ajanthan, T., and Torr, P. H. Riemannian walk for incremental learning: Understanding forgetting and intransigence. arXiv preprint arXiv:1801.10112, 2018. find
  7. Furlanello, T., Zhao, J., Saxe, A. M., Itti, L., and Tjan, B. S. Active long term memory networks. CoRR, abs/1606.02355, 2016. — [arXiv:1606.02355](https://arxiv.org/abs/1606.02355) find
  8. Ghosh, D., Singh, A., Rajeswaran, A., Kumar, V., and Levine, S. Divide-and-conquer reinforcement learning. CoRR, abs/1711.09874, 2017. — [arXiv:1711.09874](https://arxiv.org/abs/1711.09874) find
  9. Graves, A., Wayne, G., Reynolds, M., Harley, T., Danihelka, I., Grabska-Barwińska, A., Colmenarejo, S. G., Grefenstette, E., Ramalho, T., Agapiou, J., et al. Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626):471–476, 2016. find
  10. He, X. and Jaeger, H. Overcoming catastrophic interference by conceptors. ICLR, 2018. find
  11. Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  12. Huszár, F. On quadratic penalties in elastic weight consolidation. arXiv preprint arXiv:1712.03847, 2017. — [arXiv:1712.03847](https://arxiv.org/abs/1712.03847) find
  13. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A. A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, pp. 201611835, 2017. find
  14. Koch, G., Zemel, R., and Salakhutdinov, R. Siamese neural networks for one-shot image recognition. In ICML Deep Learning Workshop, volume 2, 2015. find
  15. Lake, B. M., Salakhutdinov, R., and Tenenbaum, J. B. Human-level concept learning through probabilistic program induction. Science, 350(6266):1332–1338, 2015. find
  16. Li, Y., Hernández-Lobato, J. M., and Turner, R. E. Stochastic expectation propagation. In Advances in Neural Information Processing Systems, pp. 2323–2331, 2015. find
  17. Li, Z. and Hoiem, D. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017. find
  18. Lopez-Paz, D. et al. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6470–6479, 2017. find
  19. MacKay, D. J. Information theory, inference and learning algorithms. Cambridge university press, 2003. find
  20. Minka, T. P. Expectation propagation for approximate Bayesian inference. In UAI, pp. 362–369, 2001. find
  21. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013. find
  22. Nguyen, C. V., Li, Y., Bui, T. D., and Turner, R. E. Variational continual learning. arXiv preprint arXiv:1710.10628, 2017. find
  23. Parisotto, E., Ba, J. L., and Salakhutdinov, R. Actor-mimic: Deep multitask and transfer reinforcement learning. arXiv preprint arXiv:1511.06342, 2015. find
  24. Rebuffi, S.-A., Kolesnikov, A., and Lampert, C. H. icarl: Incremental classifier and representation learning. arXiv preprint arXiv:1611.07725, 2016. find
  25. Ring, M. B. Continual Learning in Reinforcement Environments. R. Oldenbourg Verlag, 1995. find
  26. Robins, A. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2):123–146, 1995. find
  27. Rozantsev, A., Salzmann, M., and Fua, P. Beyond sharing weights for deep domain adaptation. arXiv preprint arXiv:1603.06432, 2016. — [arXiv:1603.06432](https://arxiv.org/abs/1603.06432) find
  28. Rusu, A. A., Colmenarejo, S. G., Gulcehre, C., Desjardins, G., Kirkpatrick, J., Pascanu, R., Mnih, V., Kavukcuoglu, K., and Hadsell, R. Policy distillation. arXiv preprint arXiv:1511.06295, 2015. find
  29. Rusu, A. A., Rabinowitz, N. C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., and Hadsell, R. Progressive neural networks. arXiv preprint arXiv:1606.04671, 2016a. find
  30. Rusu, A. A., Vecerik, M., Rothörl, T., Heess, N., Pascanu, R., and Hadsell, R. Sim-to-real robot learning from pixels with progressive nets. arXiv preprint arXiv:1610.04286, 2016b. — [arXiv:1610.04286](https://arxiv.org/abs/1610.04286) find
  31. Schmidhuber, J. Powerplay: Training an increasingly general problem solver by continually searching for the simplest still unsolvable problem. Frontiers in psychology, 4: 313, 2013. find
  32. Schmidhuber, J. One big net for everything. arXiv preprint arXiv:1802.08864, 2018. — [arXiv:1802.08864](https://arxiv.org/abs/1802.08864) find
  33. Shin, H., Lee, J. K., Kim, J., and Kim, J. Continual learning with deep generative replay. arXiv preprint arXiv:1705.08690, 2017. find
  34. Shu, T., Xiong, C., and Socher, R. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. CoRR, abs/1712.07294, 2017. find
  35. Silver, D. L., Yang, Q., and Li, L. Lifelong machine learning systems: Beyond learning algorithms. In AAAI Spring Symposium: Lifelong Machine Learning, volume 13, pp. 05, 2013. find
  36. Sutton, R. S. and Barto, A. G. Reinforcement learning: An introduction, volume 1. MIT press Cambridge, 1998. find
  37. Taylor, M. E. and Stone, P. An introduction to inter-task transfer for reinforcement learning. AI Magazine, 32(1): 15–34, 2011. find
  38. Teh, Y., Bapst, V., Czarnecki, W. M., Quan, J., Kirkpatrick, J., Hadsell, R., Heess, N., and Pascanu, R. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pp. 4499–4509, 2017. find
  39. Thrun, S. Explanation-based neural network learning: A lifelong learning approach, volume 357. Springer Science & Business Media, 1996. find
  40. Vinyals, O., Blundell, C., Lillicrap, T., Wierstra, D., et al. Matching networks for one shot learning. In Advances in Neural Information Processing Systems, pp. 3630–3638, 2016.
  41. Zenke, F., Poole, B., and Ganguli, S. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987–3995, 2017. find
  42. Progress & Compress: A scalable framework for continual learning. Supplementary material Train B-E 1.0 Train A 0.8 Test Accuracy on Task A 0.6 0.4 EWC online EWC Finetuning 0.2 0.0 Training steps Figure 5. Performance retention on permuted MNIST. Shown is the test accuracy on an initial permutation… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schwarz-2018-progress-compress-scalable-framework-continual/