Elamur Research⌕
☰
home / papers / 2024 / A Label is Worth a Thousand Images in Dataset Distillation

A Label is Worth a Thousand Images in Dataset Distillation

Paper · 2024 · Tian Qin, Zhiwei Deng and David Alvarez-Melis

Summary

Dataset distillation compresses a large training set into a small synthetic one that trains models almost as well, but leading methods generate synthetic images in very different ways, so it is unclear what drives their success. The authors observe that nearly all methods that scale to ImageNet-1K share one feature: soft (probabilistic) labels, often produced by pretrained expert models. They run controlled ablations on ImageNet-1K and TinyImageNet, including a simple baseline that labels randomly sampled real training images with expert-generated soft labels. This baseline performs on par with SOTA methods such as SRe2L, which shows that label quality, not image synthesis, explains most of the gains. Soft labels help only when they carry semantic structure, such as similarity between related classes. The best labels come from experts stopped early, and the ideal stopping point depends on the images-per-class budget. The paper also fits a data-knowledge scaling law and empirically maps the Pareto frontier for learning efficiently from limited data. Together, these results suggest that distillation research should focus more on labels.

Key points

  • On ImageNet-1K with random real images, hard labels give only 37.8% accuracy at IPC=200. Soft-label baselines using ResNet18 or ResNet50 experts reach about 57–63%. For comparison, full-data training yields 69.8% for ResNet18 and 76.1% for ResNet50.
  • Soft labels must contain structured inter-class information to help. Different distillation budgets favor different label "profiles," which can be tuned by early-stopping the expert labeler.
  • Ensembles of experts further improve the learning value of soft labels.
  • Learning labels alone with truncated BPTT on TinyImageNet, without training any expert, recovers labels close to those of early-stopped experts. At IPC=1 the closest match is at epoch 13, near the optimal stop at epoch 11. In this label-only setting, dataset distillation converges to knowledge distillation.
  • Code is available at github.com/sunnytqin/no-distillation. The experiments cover only image classification.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 1Research subjects and languages the work is about.Efficient models & compression · 328 works

Cite

Cite as Tian Qin, Zhiwei Deng and David Alvarez-Melis. 2024. A Label is Worth a Thousand Images in Dataset Distillation. arXiv:2406.10485.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.10485 · arXiv 2406.10485

@misc{qin2024label,
  title = {{A Label is Worth a Thousand Images in Dataset Distillation}},
  author = {Tian Qin, Zhiwei Deng and David Alvarez-Melis},
  year = {2024},
  doi = {10.48550/arxiv.2406.10485},
  eprint = {2406.10485},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.10485}
}

References

18 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. Abdin et al. “Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone”. arXiv [cs.CL] (2024). arXiv: 2404.14219 [cs.CL]. J. Ba and R. Caruana. “Do deep nets really need to be deep?” Adv. Neural Inf. Process. Syst. (2013), pp. 2654–2662. eprint: 1312.6184. O. Bohdal, Y. Yang,…
  2. N. Sachdeva and J. McAuley. “Data Distillation: A Survey”. arXiv [cs.LG] (2023). arXiv: 2301.04272 [cs.LG].
  3. S. Shao et al. “Generalized large-scale data condensation via various backbone and Statistical Matching”. arXiv [cs.CV] (2023). arXiv: 2311.17950 [cs.CV]. — [arXiv:2311.17950](https://arxiv.org/abs/2311.17950) find
  4. Z. Shen and E. Xing. “A Fast Knowledge Distillation Framework for Visual Recognition”. arXiv [cs.CV] (2021). arXiv: 2112.01528 [cs.CV]. — [arXiv:2112.01528](https://arxiv.org/abs/2112.01528) find
  5. I. Sucholutsky and M. Schonlau. “Soft-Label Dataset Distillation and Text Dataset Distillation”. 2021 International Joint Conference on Neural Networks (IJCNN). IEEE, 2021, pp. 1–8. find
  6. Y. Tian, D. Krishnan, and P. Isola. “Contrastive Representation Distillation”. arXiv [cs.LG] (2019). arXiv: 1910.10699 [cs.LG]. — [arXiv:1910.10699](https://arxiv.org/abs/1910.10699) find
  7. K. Wang et al. “CAFE: Learning to Condense dataset by Aligning FEatures”. arXiv [cs.CV] (2022), pp. 12196–12205. arXiv: 2203.01531 [cs.CV]. — [arXiv:2203.01531](https://arxiv.org/abs/2203.01531) find
  8. T. Wang, J.-Y. Zhu, A. Torralba, and A. A. Efros. “Dataset Distillation”. arXiv [cs.LG] (2018). arXiv: 1811.10959 [cs.LG].
  9. W. Yang, Y. Zhu, Z. Deng, and O. Russakovsky. “What is Dataset Distillation Learning?” arXiv [cs.LG] (2024). arXiv: 2406.04284 [cs.LG]. find
  10. H. Yin et al. “Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion”. arXiv [cs.LG] (2019). arXiv: 1912.08795 [cs.LG]. — [arXiv:1912.08795](https://arxiv.org/abs/1912.08795) find
  11. Z. Yin and Z. Shen. “Dataset Distillation in Large Data Era”. arXiv [cs.CV] (2023). arXiv: 2311.18838 [cs.CV]. — [arXiv:2311.18838](https://arxiv.org/abs/2311.18838) find
  12. Z. Yin, E. Xing, and Z. Shen. “Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspective”. arXiv [cs.CV] (2023). arXiv: 2306.13092 [cs.CV]. — [arXiv:2306.13092](https://arxiv.org/abs/2306.13092) find
  13. L. Yuan et al. “Revisiting Knowledge Distillation via label smoothing regularization”. Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. (2020), pp. 3902–3910. find
  14. B. Zhao and H. Bilen. “Dataset Condensation with Differentiable Siamese Augmentation”. Proceedings of the 38th International Conference on Machine Learning. Ed. by M. Meila and T. Zhang. Vol. 139. Proceedings of Machine Learning Research. PMLR, 2021, pp. 12674–12685. find
  15. B. Zhao and H. Bilen. “Dataset Condensation with Distribution Matching”. 2023 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). IEEE, 2023, pp. 6514–6523. find
  16. B. Zhao, K. R. Mopuri, and H. Bilen. “Dataset Condensation with Gradient Matching”. arXiv [cs.CV] (2020). arXiv: 2006.05929 [cs.CV].
  17. H. Zhou et al. “Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective”. arXiv [cs.LG] (2021). arXiv: 2102.00650 [cs.LG]. — [arXiv:2102.00650](https://arxiv.org/abs/2102.00650) find
  18. Y. Zhou, E. Nezhadarya, and J. Ba. “Dataset Distillation using Neural Feature Regression”. arXiv [cs.LG] (2022). arXiv: 2206.00719 [cs.LG]. — [arXiv:2206.00719](https://arxiv.org/abs/2206.00719) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qin-2024-label-worth-thousand-images-dataset/