Elamur Research⌕
☰
home / papers / 2018 / Dataset Distillation

Dataset Distillation

Paper · 2018 · Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba et al.

Summary

Large training sets require many thousands of update steps. This paper asks whether their knowledge can instead be packed into a handful of synthetic examples. Unlike model distillation, which shrinks the network, dataset distillation keeps the architecture fixed and learns a tiny artificial training set that need not resemble real data. The authors express the network weights after one or more gradient descent steps as a differentiable function of the synthetic inputs and a learned step size. They then optimize the pixels directly, so that the updated model performs well on the full original data.

They first handle a known starting point. They then average the objective over a distribution of initializations, extend the approach to multiple steps and epochs, and derive a lower bound on distilled set size for a linear model. Tests cover classification, fine-tuning of pre-trained networks and data poisoning, and the method generally beats real-image baselines. The work matters for studying how compressible datasets are and for rapidly "loading" knowledge into a network.

Key points

  • On MNIST, 10 distilled images (one per class) train LeNet from a fixed initialization to 94% test accuracy, versus 99% with all 60,000 images. For random initializations, 100 images reach about 80%.

  • On CIFAR10, 100 distilled images give 54% accuracy from a fixed initialization, compared with 80% with full training.

  • 100 images capturing the SVHN-to-MNIST domain gap raise a pre-trained SVHN network from 52% to 85% MNIST accuracy.

  • In the poisoning setting, 300 synthetic images applied in a single step drop accuracy on CIFAR10 "plane" from 82% to 7%.

  • The evaluation uses MNIST, CIFAR10, PASCAL-VOC and CUB-200, with four baselines:

    • random real images
    • selected real images
    • k-means centroids
    • per-class mean images

    Training uses Adam, and a typical run takes 1–4 hours on Titan Xp/V100 GPUs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States209 works · 244,782 citations🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations🔬MIT Computer Science and Artificial Intelligence LaboratoryCambridge · United States76 works · 32,207 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 297 works

Cite

Cite as Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba and Alexei A. Efros. 2018. Dataset Distillation. arXiv:1811.10959.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1811.10959 · arXiv 1811.10959

@misc{wang2018dataset,
  title = {{Dataset Distillation}},
  author = {Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba and Alexei A. Efros},
  year = {2018},
  doi = {10.48550/arxiv.1811.10959},
  eprint = {1811.10959},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1811.10959}
}

References

54 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anelia Angelova, Yaser Abu-Mostafam, and Pietro Perona. Pruning training sets for learning of object categories. In CVPR, 2005. 3 find
  2. Jimmy Ba and Rich Caruana. Do deep nets really need to be deep? In NIPS, 2014. 2 find
  3. Olivier Bachem, Mario Lucic, and Andreas Krause. Practical coreset constructions for machine learning. arXiv preprint arXiv:1703.06476, 2017. 3 — [arXiv:1703.06476](https://arxiv.org/abs/1703.06476) find
  4. David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In CVPR, 2017. 3 find
  5. Yoshua Bengio. Gradient-based optimization of hyperparameters. Neural computation, 12(8):1889–1900, 2000. 3 find
  6. Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In ICML, 2012. 6 find
  7. David A Cohn, Zoubin Ghahramani, and Michael I Jordan. Active learning with statistical models. Journal of artificial intelligence research, 4:129–145, 1996. 3 find
  8. Hal Daume III. Frustratingly easy domain adaptation. In ACL, 2007. 6
  9. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 6, 9 find
  10. Justin Domke. Generic methods for optimization-based modeling. In Artificial Intelligence and Statistics, pp. 318–326, 2012. 3, 5 find
  11. Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 88(2):303–338, 2010. 9 find
  12. Pedro F Felzenszwalb, Ross B Girshick, David McAllester, and Deva Ramanan. Object detection with discriminatively trained part-based models. PAMI, 32(9):1627–1645, 2010. 3 find
  13. Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International conference on artificial intelligence and statistics, 2010. 6, 7 find
  14. Sally A Goldman and Michael J Kearns. On the complexity of teaching. Journal of Computer and System Sciences, 50(1):20–31, 1995. 3 find
  15. Sariel Har-Peled and Akash Kushal. Smaller coresets for k-median and k-means clustering. Discrete & Computational Geometry, 37(1):3–19, 2007. 3 find
  16. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015. 6 find
  17. Marti A. Hearst, Susan T Dumais, Edgar Osuna, John Platt, and Bernhard Scholkopf. Support vector machines. IEEE Intelligent Systems and their applications, 13(4):18–28, 1998. 3 find
  18. Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in a neural network. In NIPS Deep Learning and Representation Learning Workshop, 2015. 1, 2
  19. Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. In CVPR, 2017. 2
  20. Jonathan J. Hull. A database for handwritten text recognition research. PAMI, 16(5):550–554, 1994. 9 find
  21. Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 14
  22. Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICML, 2017. 3, 6
  23. Alex Krizhevsky. cuda-convnet: High-performance c++/cuda implementation of convolutional neural networks. https://github.com/akrizhevsky/cuda-convnet2, 2012. 7 ↗
  24. Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009. 7 find
  25. Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In NIPS, 2012. 6, 9
  26. Agata Lapedriza, Hamed Pirsiavash, Zoya Bylinskii, and Antonio Torralba. Are all training examples equally valuable? arXiv preprint arXiv:1311.6510, 2013. 3 — [arXiv:1311.6510](https://arxiv.org/abs/1311.6510) find
  27. Yann LeCun. The mnist database of handwritten digits. http://yann.lecun. com/exdb/mnist/, 1998. 7 ↗
  28. Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. 1, 7 find
  29. Bo Li, Yining Wang, Aarti Singh, and Yevgeniy Vorobeychik. Data poisoning attacks on factorization-based collaborative filtering. In NIPS, 2016. 6 find
  30. Raphael Gontijo Lopes, Stefano Fenu, and Thad Starner. Data-free knowledge distillation for deep neural networks. arXiv preprint arXiv:1710.07535, 2017. 3 — [arXiv:1710.07535](https://arxiv.org/abs/1710.07535) find
  31. Dougal Maclaurin, David Duvenaud, and Ryan Adams. Gradient-based hyperparameter optimization through reversible learning. In ICML, 2015. 3, 5 find
  32. Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In CVPR, 2015. 3 find
  33. Saeid Motiian, Quinn Jones, Seyed Iranmanesh, and Gianfranco Doretto. Few-shot adversarial domain adaptation. In NIPS, 2017. 9 find
  34. Luis Muñoz-González, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongrassamee, Emil C Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 27–38, 2017. 6 find
  35. Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS workshop, 2011. 9 find
  36. J Arturo Olvera-López, J Ariel Carrasco-Ochoa, J Francisco Martínez-Trinidad, and Josef Kittler. A review of instance selection methods. Artificial Intelligence Review, 34(2):133–143, 2010. 3 find
  37. Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In ICLR Workshop, 2017. 5 find
  38. Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural computation, 6(1):147–160, 1994. 5 find
  39. Fabian Pedregosa. Hyperparameter optimization with approximate gradient. In ICML, 2016. 3 find
  40. Jean Ponce, Tamara L Berg, Mark Everingham, David A Forsyth, Martial Hebert, Svetlana Lazebnik, Marcin Marszalek, Cordelia Schmid, Bryan C Russell, Antonio Torralba, et al. Dataset issues in object recognition. In Toward category-level object recognition, pp. 29–48. 2006. 3 find
  41. Ilija Radosavovic, Piotr Dollár, Ross Girshick, Georgia Gkioxari, and Kaiming He. Data distillation: Towards omni-supervised learning. In CVPR, 2018. 2 find
  42. Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015. 2 find
  43. Frank Rosenblatt. The perceptron, a perceiving and recognizing automaton Project Para. Cornell Aeronautical Laboratory, 1957. 3 find
  44. Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV, 2010. 6 find
  45. Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR, 2018. 3 find
  46. Ayumi Shinohara and Satoru Miyano. Teachability in computational learning. New Generation Computing, 8(4): 337–347, 1991. 3 find
  47. Simon Tong and Daphne Koller. Support vector machine active learning with applications to text classification. JMLR, 2(Nov):45–66, 2001. 3 find
  48. Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR, pp. 1521–1528. IEEE, 2011. 3, 6 find
  49. Ivor W Tsang, James T Kwok, and Pak-Ming Cheung. Core vector machines: Fast svm training on very large data sets. JMLR, 6(Apr):363–392, 2005. 3 find
  50. C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute of Technology, 2011. 9 find
  51. Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014. 3
  52. Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Object detectors emerge in deep scene cnns. In ICLR, 2015. 3 find
  53. Xiaojin Zhu. Machine teaching for bayesian learners in the exponential family. In NIPS, 2013. 3 find
  54. Xiaojin Zhu. Machine teaching: An inverse problem to machine learning and an approach toward optimal education. In AAAI, 2015. 3 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2018-dataset-distillation/