Elamur Research⌕
☰
home / papers / 2020 / Dataset Condensation with Gradient Matching

Dataset Condensation with Gradient Matching

Paper · 2020 · Bo Zhao, Konda Reddy Mopuri and Hakan Bilen

Summary

Modern deep learning depends on ever-larger training sets, which makes storing data and training models costly. Zhao, Mopuri and Bilen propose Dataset Condensation, which compresses a large labelled image collection into a few learned synthetic images that can train networks from scratch. Coreset methods pick existing samples using heuristics. Dataset Distillation (DD) unrolls an expensive nested optimization. This method instead updates the synthetic images so that the weight gradients a network computes on them match the gradients it computes on real data. Matching happens at many points along training, as a curriculum, and over many random initializations. This keeps the synthetic-set network on a similar trajectory in parameter space without backpropagating through the whole training process. On several vision benchmarks, the condensed sets beat coreset selection and DD by large margins. They also transfer better across architectures and need less memory and time to learn. The authors further show gains in continual learning and neural architecture search, where memory and compute budgets are limited.

Key points

  • Core idea: a gradient-matching objective between real and synthetic batches replaces DD's bi-level optimization. Model weights are decoupled from earlier training states, which avoids keeping a long recursive computation graph.

  • CIFAR100 accuracy: 12.8% with 1 image per class and 25.2% with 10 images per class. Both beat coreset baselines and Label Distillation (LD).

  • Comparison with DD at 10 images/class, one GTX1080-Ti GPU:

    SettingMethodMemoryTimeAccuracy
    MNIST/LeNetDD785 MB160 min79.5%
    MNIST/LeNetOurs653 MB46 min93.9%
    CIFAR10/AlexCifarNetDD3211 MB214 min36.8%
    CIFAR10/AlexCifarNetOurs1445 MB105 min39.1%

  • Cross-architecture transfer: MNIST images condensed with LeNet still train MLP, ConvNet, AlexNet, VGG11 and ResNet18 well. DD's images, which are tied to learned per-step learning rates, perform poorly on AlexNet, VGG and ResNet.

  • Resource: code is released at github.com/VICO-UoE/DatasetCondensation. The synthetic images look more interpretable than DD's, although realism was not the goal.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of EdinburghEdinburgh · United Kingdom381 works · 80,747 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Bo Zhao, Konda Reddy Mopuri and Hakan Bilen. 2020. Dataset Condensation with Gradient Matching. arXiv:2006.05929 · International Conference on Learning Representations 2021.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2006.05929 · arXiv 2006.05929

@misc{zhao2020dataset,
  title = {{Dataset Condensation with Gradient Matching}},
  author = {Bo Zhao, Konda Reddy Mopuri and Hakan Bilen},
  year = {2020},
  doi = {10.48550/arxiv.2006.05929},
  eprint = {2006.05929},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2006.05929}
}

References

60 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pankaj K Agarwal, Sariel Har-Peled, and Kasturi R Varadarajan. Approximating extent measures of points. Journal of the ACM (JACM), 51(4):606–635, 2004. find
  2. Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems, pp. 11816– 11825, 2019.
  3. Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In Advances in neural information processing systems, pp. 2654–2662, 2014. find
  4. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
  5. Eden Belouadah and Adrian Popescu. Scail: Classifier weights scaling for class incremental learning. In The IEEE Winter Conference on Applications of Computer Vision, 2020. find
  6. Ondrej Bohdal, Yongxin Yang, and Timothy Hospedales. Flexible dataset distillation: Learn labels instead of images. Neural Information Processing Systems Workshop, 2020. find
  7. Cristian Buciluǎ, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 535–541, 2006. find
  8. Francisco M Castro, Manuel J Marı́n-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 233–248, 2018. find
  9. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. find
  10. CVPR 2009. IEEE Conference on, pp. 248–255. find
  11. Justin Domke. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics, pp. 318–326, 2012. find
  12. Dan Feldman, Melanie Schmidt, and Christian Sohler. Turning big data into tiny data: Constant-size coresets for k-means, pca and projective clustering. In Proceedings of the twenty-fourth annual ACM-SIAM symposium on Discrete algorithms, pp. 1434–1453. SIAM, 2013. find
  13. Spyros Gidaris and Nikos Komodakis. Dynamic few-shot visual learning without forgetting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4367– 4375, 2018. find
  14. Jack Goetz and Ambuj Tewari. arXiv:2008.04489, 2020. — [arXiv:2008.04489](https://arxiv.org/abs/2008.04489) find
  15. Federated learning via synthetic data. Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pp. 2672–2680, 2014a.
  16. Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014b.
  17. Sariel Har-Peled and Soham Mazumdar. On coresets for k-means and k-median clustering. In Proceedings of the thirty-sixth annual ACM symposium on Theory of computing, 2004. find
  18. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026–1034, 2015. find
  19. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  20. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015.
  21. Jonathan J. Hull. A database for handwritten text recognition research. IEEE Transactions on pattern analysis and machine intelligence, 16(5):550–554, 1994. find
  22. Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ArXiv, abs/1502.03167, 2015. find
  23. Diederik P Kingma and Max Welling. arXiv:1312.6114, 2013.
  24. Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1885– 1894. JMLR. org, 2017.
  25. Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. find
  26. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pp. 1097–1105, 2012.
  27. Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. find
  28. Guang Li, Ren Togo, Takahiro Ogawa, and Miki Haseyama. Soft-label anonymous gastric x-ray image distillation. In 2020 IEEE International Conference on Image Processing (ICIP), pp. 305– 309. find
  29. Liam Li and Ameet Talwalkar. Random search and reproducibility for neural architecture search. In Uncertainty in Artificial Intelligence, pp. 367–377. PMLR, 2020. find
  30. Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. In LLD Workshop at Neural Information Processing Systems (NIPS ), 2017. find
  31. David Lopez-Paz et al. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467–6476, 2017. arXiv preprint Auto-encoding variational bayes. arXiv preprint Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improve neural net… find
  32. Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5188–5196, 2015. find
  33. Mehdi Mirza and Simon Osindero. arXiv:1411.1784, 2014. find
  34. Conditional generative adversarial nets. Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807–814, 2010. find
  35. Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. Zero-shot knowledge distillation in deep networks. In Proceedings of the 36th International Conference on Machine Learning, 2019. find
  36. Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011. find
  37. Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434, 2015.
  38. Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2001–2010, 2017. find
  39. Salah Rifai, Yoshua Bengio, Yann Dauphin, and Pascal Vincent. A generative process for sampling contractive auto-encoders. arXiv preprint arXiv:1206.6434, 2012. — [arXiv:1206.6434](https://arxiv.org/abs/1206.6434) find
  40. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550, 2014. find
  41. Kegan GG Samuel and Marshall F Tappen. Learning optimized map estimates in continuously-valued mrf models. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 477–484. find
  42. Mert Bulent Sariyildiz and Ramazan Gokberk Cinbis. Gradient matching generative networks for zero-shot learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2168–2178, 2019. find
  43. Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. ICLR, 2018. find
  44. Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  45. Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077–4087, 2017. find
  46. Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth O Stanley, and Jeff Clune. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. International Conference on Machine Learning, 2020. find
  47. Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. arXiv preprint arXiv:1910.02551, 2019. — [arXiv:1910.02551](https://arxiv.org/abs/1910.02551) find
  48. Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. ICLR, 2019. find
  49. Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016. arXiv preprint Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advan…
  50. Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811.10959, 2018.
  51. Max Welling. Herding dynamical weights to learn. In Proceedings of the 26th Annual International Conference on Machine Learning, pp. 1121–1128. ACM, 2009. find
  52. G W Wolf. Facility location: concepts, models, algorithms and case studies. 2011. find
  53. Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 374–382, 2019. find
  54. Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3–19, 2018. find
  55. Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747, 2017. — [arXiv:1708.07747](https://arxiv.org/abs/1708.07747) find
  56. Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In European conference on computer vision, pp. 818–833. Springer, 2014.
  57. Matthew D. Zeiler, Marc’Aurelio Ranzato, Rajat Monga, Mark Z. Mao, Kyeongcheol Yang, Quoc V. Le, Patrick Nguyen, Andrew W. Senior, Vincent Vanhoucke, Jeffrey Dean, and Geoffrey E. Hinton. On rectified linear units for speech processing. 2013 IEEE International Conference on Acoustics, Speech and Sig… find
  58. Bo Zhao, Konda Reddy Mopuri, and Hakan Bilen. idlg: Improved deep leakage from gradients. arXiv preprint arXiv:2001.02610, 2020. — [arXiv:2001.02610](https://arxiv.org/abs/2001.02610) find
  59. Yanlin Zhou, George Pu, Xiyao Ma, Xiaolin Li, and Dapeng Wu. Distilled one-shot federated learning. arXiv preprint arXiv:2009.07999, 2020. — [arXiv:2009.07999](https://arxiv.org/abs/2009.07999) find
  60. Ligeng Zhu, Zhijian Liu, and Song Han. Deep leakage from gradients. In Advances in Neural Information Processing Systems, pp. 14747–14756, 2019. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2020-dataset-condensation-gradient-matching/