Elamur Research⌕
☰
home / papers / 2023 / A Survey on Dataset Distillation: Approaches, Applications and Future Directions

A Survey on Dataset Distillation: Approaches, Applications and Future Directions

Paper · 2023 · Jiahui Geng, Zongxiong Chen, Yuandou Wang et al.

Summary

Modern deep learning relies on ever-larger training sets, which makes processing, storage and repeated training costly. This is especially true for hyperparameter tuning and neural architecture search. Dataset distillation addresses the problem by learning a small synthetic dataset on which models reach accuracy close to models trained on the full data. Unlike coreset selection, which picks real samples and is NP-hard, distillation optimizes the samples themselves. The authors present what they describe as the first systematic categorization of this young field. Their taxonomy covers three aspects: approaches (learning frameworks plus pluggable enhancements), data modalities, and downstream applications. A comparison table maps representative methods onto these categories. The survey finds that most work targets image data, while discrete text and graph data remain difficult and robustness is rarely studied. It also identifies open problems in efficiency, scaling and task coverage. For practitioners, including NLP teams, it offers a clear map of available techniques and shows where distillation for language data is still immature.

Key points

  • Learning frameworks. These split into two families:
    • Meta-learning: bi-level optimization solved with back-propagation through time (DD, with truncated BPTT) or with kernel ridge regression (KIP using NTK, FRePo, RFAD).
    • Surrogate objectives: parameter/trajectory matching (DC, DSA, MTT, IDC, HaBa, GCond) or distribution matching (DM, CAFE, IT-GAN, KFS).
  • Enhancement methods. Three add-on techniques can be combined with any framework: parameterization (e.g., factorization), augmentation (e.g., DSA) and label distillation.
  • Modalities and applications. Methods are grouped by data type: image, audio, text and graph. Applications fall into three groups:
    • Compute-heavy tasks: continual learning and NAS.
    • Privacy: dataset publishing (KIP ρ, RFAD ρ, differential privacy) and federated learning.
    • Robustness: backdoor/poisoning attacks and adversarially robust distilled sets.
  • Efficiency issues. BPTT approaches suffer from heavy memory use, biased short unrolls and exploding or vanishing gradients. MTT, KIP and FRePo hit GPU memory limits as images per class (IPC) grow. RFAD reduces Gram-matrix cost to linear in |S|.
  • Open challenges. At larger IPC, most methods drop to roughly random-sampling quality (citing Cui et al., 2022). Research also remains almost entirely focused on classification, leaving tasks like NER, summarization and machine translation unexplored.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands153 works · 174,584 citations🎓Technical University of MunichMunich · Germany66 works · 49,493 citations🔬Fraunhofer Institute for Open Communication SystemsBerlin · Germany3 works · 6,830 citations🎓University of StavangerStavanger · Norway3 works · 137 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 328 works

Cite

Cite as Jiahui Geng, Zongxiong Chen, Yuandou Wang, Herbert Woisetschlaeger, Sonja Schimmler, Ruben Mayer, Zhiming Zhao and Chunming Rong. 2023. A Survey on Dataset Distillation: Approaches, Applications and Future Directions. arXiv:2305.01975.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.01975 · arXiv 2305.01975

@misc{geng2023survey,
  title = {{A Survey on Dataset Distillation: Approaches, Applications and Future Directions}},
  author = {Jiahui Geng, Zongxiong Chen, Yuandou Wang, Herbert Woisetschlaeger, Sonja Schimmler, Ruben Mayer, Zhiming Zhao and Chunming Rong},
  year = {2023},
  doi = {10.48550/arxiv.2305.01975},
  eprint = {2305.01975},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.01975}
}

References

11 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [Abdal et al., 2019] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4432– 4441, 2019. [Andrychowicz et al., 2016] Marcin Andrychowicz, Misha Denil, Sergio… find
  2. Ieee, 2009. [Deng, 2022] Olga Deng, Zhiwei; Russakovsky. Remember the past: Distilling datasets into addressable memories for neural networks. arXiv preprint arXiv:2206.02916, 2022. — [arXiv:2206.02916](https://arxiv.org/abs/2206.02916) find
  3. [Dong et al., 2022] Tian Dong, Bo Zhao, and Lingjuan Lyu. Privacy for free: How does dataset condensation help privacy? arXiv preprint arXiv:2206.00240, 2022. [Dwork, 2006] Cynthia Dwork. Differential privacy. In Automata, Languages and Programming: 33rd International Colloquium, ICALP 2006, Venice,… — [arXiv:2206.00240](https://arxiv.org/abs/2206.00240) find
  4. Springer, 2006. [Fredrikson et al., 2015] Matt Fredrikson, Somesh Jha, and Thomas Ristenpart. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM SIGSAC conference on computer and communications security, pages 1322–1333, 2015. [Geng… — [arXiv:2110.09074](https://arxiv.org/abs/2110.09074) find
  5. PMLR, 2017. [Hu et al., 2022] Shengyuan Hu, Jack Goetz, Kshitiz Malik, Hongyuan Zhan, Zhe Liu, and Yue Liu. Fedsynth: Gradient compression via synthetic data in federated learning. arXiv preprint arXiv:2204.01273, 2022. [Jacot et al., 2018] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural… find
  6. PMLR, 2015. [Nguyen et al., 2020] Timothy Nguyen, Zhourong Chen, and Jaehoon Lee. Dataset meta-learning from kernel ridge-regression. arXiv preprint arXiv:2011.00050, 2020. [Nguyen et al., 2021] Timothy Nguyen, Roman Novak, Lechao Xiao, and Jaehoon Lee. Dataset distillation with infinitely wide conv… — [arXiv:2011.00050](https://arxiv.org/abs/2011.00050) find
  7. Springer, 2020. [Schmidhuber, 2015] Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural networks, 61:85– 117, 2015. [Sener and Savarese, 2017] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A coreset approach. arXiv preprint arXiv:1708.00489… find
  8. IEEE symposium on security and privacy (SP), pages 3– 18. IEEE, 2017. [Such et al., 2020] Felipe Petroski Such, Aditya Rawal, Joel Lehman, Kenneth Stanley, and Jeffrey Clune. Generative teaching networks: Accelerating neural architecture search by learning to generate synthetic training data. In Int… find
  9. PMLR, 2020. [Sucholutsky and Schonlau, 2021] Ilia Sucholutsky and Matthias Schonlau. Soft-label dataset distillation and text dataset distillation. In 2021 International Joint Conference on Neural Networks (IJCNN), pages 1–8. find
  10. IEEE, 2021. [Tsilivis et al., 2022] Nikolaos Tsilivis, Jingtong Su, and Julia Kempe. Can we achieve robustness from data alone? arXiv preprint arXiv:2207.11727, 2022. [Wang et al., 2018] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros. Dataset distillation. arXiv preprint arXiv:1811…
  11. PMLR, 2021. [Zhou et al., 2022] Yongchao Zhou, Ehsan Nezhadarya, and Jimmy Ba. Dataset distillation using neural feature regression. arXiv preprint arXiv:2206.00719, 2022. — [arXiv:2206.00719](https://arxiv.org/abs/2206.00719) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/geng-2023-survey-dataset-distillation-approaches-applications/