Elamur Research⌕
☰
home / papers / 2017 / Memory Aware Synapses: Learning what (not) to forget

Memory Aware Synapses: Learning what (not) to forget

Paper · 2017 · Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny et al.

Summary

The paper addresses lifelong learning (LLL), where a single neural network learns tasks one after another and cannot access data from earlier tasks. As a result, it tends to suffer from catastrophic forgetting. The authors, from KU Leuven and Facebook AI Research, note that a model has finite capacity, so it cannot retain everything. The model should itself decide what is worth remembering and what can be let go. To this end, they propose Memory Aware Synapses (MAS). For each parameter, MAS accumulates an importance weight: how strongly the network's output function changes when the parameter is slightly perturbed. Loss values are not used, so no labels are needed, and importance can be updated online on unlabeled data from the real environment. When a new task is learned, a regularizer penalizes changes to important parameters. The method was tested on sequences of object recognition tasks and on learning embeddings for <subject, predicate, object> triplets. The authors report state-of-the-art results. The paper's value lies in letting the model focus its "memory" on what actually comes up during use.

Key points

  • MAS is compared with data-based methods (LwF, EBLL) and model-based methods (EWC, IMM, SI). Unlike EWC (the diagonal of the Fisher matrix) and SI (changes in the loss during training), MAS estimates the sensitivity of the output function itself. In this way, the method avoids near-zero gradients at a local minimum of the loss.
  • The method requires a constant amount of memory, is not tied to a particular type of task (not only classification), works with pretrained models, and adapts to test conditions.
  • A local variant of MAS is theoretically connected to Hebb's rule: connections between neurons that fire together are strengthened.
  • Importance was estimated separately on training data and on test data. For the 1000 most important parameters, the two estimates almost coincide, which means the important parameters can be identified without labels.
  • In an experiment with four fact learning tasks, MAS was adapted to a subset of sports facts from the first task that occurred frequently at test time. The adaptive variant best preserved both accuracy and the embedding structure for this subset. Plain finetuning, by contrast, eventually "collapsed" all of these examples into a single point.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MetaMenlo Park · United States214 works · 241,492 citations🎓KU LeuvenLeuven · Belgium46 works · 7,833 citations🤝IMECLeuven · Belgium4 works · 1,823 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach and Tinne Tuytelaars. 2017. Memory Aware Synapses: Learning what (not) to forget. arXiv:1711.09601.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1711.09601 · arXiv 1711.09601

@misc{aljundi2017memory,
  title = {{Memory Aware Synapses: Learning what (not) to forget}},
  author = {Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach and Tinne Tuytelaars},
  year = {2017},
  doi = {10.48550/arxiv.1711.09601},
  eprint = {1711.09601},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1711.09601}
}

References

39 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aljundi, R., Chakravarty, P., Tuytelaars, T.: Expert gate: Lifelong learning with a network of experts. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016) find
  2. Andrychowicz, M., Denil, M., Gómez, S., Hoffman, M.W., Pfau, D., Schaul, T., de Freitas, N.: Learning to learn by gradient descent by gradient descent. In: Lee, D.D., Sugiyama, M., Luxburg, U.V., Guyon, I., Garnett, R. (eds.) Advances in Neural Information Processing Systems 29, pp. 3981–3989. Curr… find
  3. de Campos, T.E., Babu, B.R., Varma, M.: Character recognition in natural images. In: Proceedings of the International Conference on Computer Vision Theory and Applications, Lisbon, Portugal (February 2009) find
  4. Elhoseiny, M., Cohen, S., Chang, W., Price, B.L., Elgammal, A.M.: Sherlock: Scalable fact learning in images. In: AAAI. pp. 4016–4024 (2017) find
  5. Everingham, M., Van Gool, L., Williams, C.K.I., Winn, J., Zisserman, A.: The PAS-CAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal-network.org/challenges/VOC/voc2012/workshop/index.html ↗
  6. Fernando, C., Banarse, D., Blundell, C., Zwols, Y., Ha, D., Rusu, A.A., Pritzel, A., Wierstra, D.: Pathnet: Evolution channels gradient descent in super neural networks. arXiv preprint arXiv:1701.08734 (2017)
  7. Finn, C., Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: Proceedings of the International Conference on Machine Learning (ICML) (2017)
  8. French, R.M.: Catastrophic forgetting in connectionist networks. Trends in cognitive sciences 3(4), 128–135 (1999) find
  9. Goodfellow, I.J., Mirza, M., Xiao, D., Courville, A., Bengio, Y.: An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211 (2013) find
  10. Hebb, D.: The organization of behavior. 1949. New York Wiely (2002) find
  11. Huszár, F.: Note on the quadratic penalties in elastic weight consolidation. Proceedings of the National Academy of Sciences (2018). https://doi.org/10.1073/pnas.1717042115, http://www.pnas.org/content/early/2018/02/16/1717042115 — [doi:10.1073/pnas.1717042115](https://doi.org/10.1073/pnas.1717042115) ↗
  12. Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., Desjardins, G., Rusu, A.A., Milan, K., Quan, J., Ramalho, T., Grabska-Barwinska, A., et al.: Overcoming catastrophic forgetting in neural networks. arXiv preprint arXiv:1612.00796 (2016) find
  13. Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for fine-grained categorization. In: Proceedings of the IEEE International Conference on Computer Vision Workshops. pp. 554–561 (2013) find
  14. Krizhevsky, A.: One weird trick for parallelizing convolutional neural networks. arXiv preprint arXiv:1404.5997 (2014)
  15. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: Pereira, F., Burges, C.J.C., Bottou, L., Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 25, pp. 1097–1105. Curran Associates, Inc. (2012)
  16. Lee, S.W., Kim, J.H., Ha, J.W., Zhang, B.T.: Overcoming catastrophic forgetting by incremental moment matching. arXiv preprint arXiv:1703.08475 (2017)
  17. Li, Z., Hoiem, D.: Learning without forgetting. In: European Conference on Computer Vision. pp. 614–629. Springer (2016) find
  18. Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. In: Advances in Neural Information Processing Systems. pp. 6470–6479 (2017) find
  19. Maji, S., Kannala, J., Rahtu, E., Blaschko, M., Vedaldi, A.: Fine-grained visual classification of aircraft. Tech. rep. (2013) find
  20. McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of learning and motivation 24, 109–165 (1989) find
  21. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 (2013)
  22. Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A.Y.: Reading digits in natural images with unsupervised feature learning (2011) find
  23. Nilsback, M.E., Zisserman, A.: Automated flower classification over a large number of classes. In: Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing (Dec 2008) find
  24. Pentina, A., Lampert, C.H.: Lifelong learning with non-i.i.d. tasks. In: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (eds.) Advances in Neural Information Processing Systems 28, pp. 1540–1548 (2015) find
  25. Pentina, A., Lampert, C.H.: Lifelong learning with non-iid tasks. In: Advances in Neural Information Processing Systems. pp. 1540–1548 (2015) find
  26. Quadrianto, N., Petterson, J., Smola, A.J.: Distribution matching for transduction. In: Advances in Neural Information Processing Systems. pp. 1500–1508 (2009) find
  27. Quattoni, A., Torralba, A.: Recognizing indoor scenes. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 413–420. IEEE (2009) find
  28. Rannen, A., Aljundi, R., Blaschko, M.B., Tuytelaars, T.: Encoder based lifelong learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1320–1328 (2017) find
  29. Rebuffi, S.A., Kolesnikov, A., Lampert, C.H.: icarl: Incremental classifier and representation learning. arXiv preprint arXiv:1611.07725 (2016) find
  30. Ring, M.B.: Child: A first step towards continual learning. Machine Learning 28(1), 77–104 (1997) find
  31. Royer, A., Lampert, C.H.: Classifier adaptation at prediction time. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 1401–1409 (2015) find
  32. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV) 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-… ↗
  33. Rusu, A.A., Rabinowitz, N.C., Desjardins, G., Soyer, H., Kirkpatrick, J., Kavukcuoglu, K., Pascanu, R., Hadsell, R.: Progressive neural networks. arXiv preprint arXiv:1606.04671 (2016) find
  34. Shmelkov, K., Schmid, C., Alahari, K.: Incremental learning of object detectors without catastrophic forgetting. In: The IEEE International Conference on Computer Vision (ICCV) (2017) find
  35. Silver, D.L., Yang, Q., Li, L.: Lifelong machine learning systems: Beyond learning algorithms. In: AAAI Spring Symposium: Lifelong Machine Learning. pp. 49–55. Citeseer (2013) find
  36. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  37. Thrun, S., Mitchell, T.M.: Lifelong robot learning. Robotics and autonomous systems 15(1-2), 25–46 (1995) find
  38. Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: Caltech-UCSD Birds 200. Tech. Rep. CNS-TR-2010-001, California Institute of Technology (2010) find
  39. Zenke, F., Poole, B., Ganguli, S.: Improved multitask learning through synaptic intelligence. In: Proceedings of the International Conference on Machine Learning (ICML) (2017) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/aljundi-2017-memory-aware-synapses-learning-forget/