Elamur Research⌕
☰
home / papers / 2023 / SemDeDup: Data-efficient learning at web-scale through semantic deduplication

SemDeDup: Data-efficient learning at web-scale through semantic deduplication

Summary

Large web datasets such as LAION are usually cleaned only of exact copies. As a result, they retain many examples that are nearly identical in content even though they differ in pixels or tokens. The authors, from Meta AI (FAIR) and Stanford, propose SemDeDup, a method that searches for such "semantic duplicates" in the embedding space of pretrained models: CLIP for images and OPT for text. The embeddings are first grouped with k-means. Pairwise cosine similarity is then computed only within each cluster, so the method can be applied to datasets with billions of examples. Pairs whose similarity exceeds a set threshold are treated as duplicates, and one example is kept from each such group. On the LAION-440M subset, removing half of the data barely lowers CLIP accuracy, halves training time and even improves out-of-distribution performance. On the C4 text corpus, the method outperforms the previous best deduplication approach and saves about 15% of compute. The authors conclude that cheap semantic cleaning should be the first step before more sophisticated data selection, especially for web corpora that have seen almost no curation.

Key points

  • Roughly half of the examples in LAION have at least one semantic duplicate. The method clusters CLIP embeddings into 50,000 clusters and OPT embeddings for C4 into 11,000. For LAION, duplicates are identified from the images alone, without the captions.
  • CLIP ViT-B-16 was trained on 50% of LAION-440M after SemDeDup. With 75% of the baseline model's iterations, it reached 69.21% zero-shot Top1 on ImageNet, while the model trained on the full dataset reached 68.74%. With 50% of the iterations, the result was 68.27%.
  • Deduplication costs about 0.29 GPU-hours of one training epoch, i.e. less than 1% (≈0.0091) of the full 32-epoch training run.
  • For C4 the effect is more modest because this corpus is already partially cleaned (three-sentence repeats and MinHash approaches in earlier work). Even so, SemDeDup still yields about a 15% efficiency gain, and sometimes better language model quality as well.
  • Limitations: a pretrained embedding model for the relevant domain is required. The method removes neither generally redundant data nor harmful or incorrect data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

80 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 74 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.09540 · arXiv 2303.09540

@misc{abbas2023semdedup,
  title = {{SemDeDup: Data-efficient learning at web-scale through semantic deduplication}},
  author = {Abbas, Amro and Tirumala, Kushal and Simig, Dániel and Ganguli, Surya and Morcos, Ari S.},
  year = {2023},
  doi = {10.48550/arxiv.2303.09540},
  eprint = {2303.09540},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.09540}
}

References

44 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. Patwary, M. Ali, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. find
  2. J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. find
  3. T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020. find
  4. J. S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit. A constructive prediction of the generalization error across scales. International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=ryenvpEKDr. ↗
  5. M. A. Gordon, K. Duh, and J. Kaplan. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5915–5922, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguis… find
  6. D. Hernandez, J. Kaplan, T. Henighan, and S. McCandlish. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021. find
  7. X. Zhai, A. Kolesnikov, N. Houlsby, and L. Beyer. Scaling vision transformers. arXiv preprint arXiv:2106.04560, 2021. — [arXiv:2106.04560](https://arxiv.org/abs/2106.04560) find
  8. J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. v. d. Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, and L. Sifre. Training compute-optima… find
  9. C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022. find our card
  10. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021. find our card
  11. G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021. URL https://doi.org/10.5281/zenodo.5143773. If you use this software, please cite it as below. — [doi:10.5281/zenodo.5143773](https://doi.org/10.5281/zenodo.5143773) ↗
  12. A. Aghajanyan, L. Yu, A. Conneau, W.-N. Hsu, K. Hambardzumyan, S. Zhang, S. Roller, N. Goyal, O. Levy, and L. Zettlemoyer. Scaling laws for generative mixed-modal language models, 2023. URL https://arxiv.org/abs/2301.03728. ↗
  13. B. Sorscher, R. Geirhos, S. Shekhar, S. Ganguli, and A. S. Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. In Neural Information Processing Systems (NeurIPS), June 2022. find our card
  14. F. Radenovic, A. Dubey, A. Kadian, T. Mihaylov, S. Vandenhende, Y. Patel, Y. Wen, V. Ramanathan, and D. Mahajan. Filtering, distillation, and hard negatives for vision-language pre-training. arXiv preprint arXiv:2301.02280, 2023. — [arXiv:2301.02280](https://arxiv.org/abs/2301.02280) find
  15. V. Feldman and C. Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Adv. Neural Inf. Process. Syst., 33:2881–2891, 2020. find
  16. Y. Liao. Dataset Deduplication with Datamodels. PhD thesis, Massachusetts Institute of Technology, May 2022. find
  17. B. Thomee, D. A. Shamma, G. Friedland, B. Elizalde, K. Ni, D. Poland, D. Borth, and L.-J. Li. YFCC100M: the new data in multimedia research. Commun. ACM, 59(2):64–73, Jan. 2016. find
  18. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2019. URL https://arxiv.org/abs/1910.10683. ↗
  19. K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini. Deduplicating training data makes language models better, 2021. URL https://arxiv.org/abs/2107.06499. ↗
  20. A. Broder. On the resemblance and containment of documents. 06 1997. doi:10.1109/SEQUEN. 1997.666900. find
  21. J. W. Rae, S. Borgeaud, T. Cai, K. Millican, J. Hoffmann, F. Song, J. Aslanides, S. Henderson, R. Ring, S. Young, E. Rutherford, T. Hennigan, J. Menick, A. Cassirer, R. Powell, G. v. d. Driessche, L. A. Hendricks, M. Rauh, P.-S. Huang, A. Glaese, J. Welbl, S. Dathathri, S. Huang, J. Uesato, J. Mello… find
  22. N. Kandpal, E. Wallace, and C. Raffel. Deduplicating training data mitigates privacy risks in language models, 2022. URL https://arxiv.org/abs/2202.06539. ↗ our card
  23. E. Silcock, L. D’Amico-Wong, J. Yang, and M. Dell. Noise-Robust De-Duplication at scale. Dec. 2022. find
  24. W.-S. Choi, D.-S. Han, H. Lee, J. Park, and B.-T. Zhang. DUEL: Adaptive duplicate elimination on working memory for Self-Supervised learning. Oct. 2022. find
  25. C. Guo, B. Zhao, and Y. Bai. DeepCore: A comprehensive library for coreset selection in deep learning. Apr. 2022. find
  26. J. M. Phillips. Coresets and sketches. Jan. 2016. find
  27. M. Toneva, A. Sordoni, R. T. des Combes, A. Trischler, Y. Bengio, and G. J. Gordon. An empirical study of example forgetting during deep neural network learning. In ICLR, 2019. find
  28. M. Paul, S. Ganguli, and G. K. Dziugaite. Deep learning on a data diet: Finding important examples early in training. Adv. Neural Inf. Process. Syst., 34, Dec. 2021. find
  29. K. Chitta, J. M. Álvarez, E. Haussmann, and C. Farabet. Training data subset search with ensemble active learning. IEEE Trans. Intell. Transp. Syst., pages 1–12, 2021. find
  30. K. Meding, L. M. S. Buschoff, R. Geirhos, and F. A. Wichmann. Trivial or impossible— dichotomous data difficulty masks model differences (on ImageNet and beyond). In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=C_vsGwEIjAr. ↗
  31. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021. find our card
  32. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P. S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022. URL https://arxiv.org/abs/2205.0106… ↗
  33. A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020. find
  34. C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022. find our card
  35. C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114, 2021. find our card
  36. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017. find
  37. I. Loshchilov and F. Hutter. arXiv:1711.05101, 2017. Decoupled weight decay regularization. find
  38. D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. CVPR, 2021. find
  39. D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang, E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and J. Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. ICCV, 2021. find
  40. H. Wang, S. Ge, Z. Lipton, and E. P. Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506– 10518, 2019. find
  41. B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do ImageNet classifiers generalize to ImageNet? In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5389–5400. PMLR, 09–15 J… find
  42. A. Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Ne… find
  43. K. Lee, D. Ippolito, A. Nystrom, C. Zhang, D. Eck, C. Callison-Burch, and N. Carlini. Deduplicating training data makes language models better. arXiv preprint arXiv:2107.06499, 2021. find
  44. S. Iyer, X. V. Lin, R. Pasunuru, T. Mihaylov, D. Simig, P. Yu, K. Shuster, T. Wang, Q. Liu, P. S. Koura, X. Li, B. O’Horo, G. Pereyra, J. Wang, C. Dewan, A. Celikyilmaz, L. Zettlemoyer, and V. Stoyanov. Opt-iml: Scaling language model instruction meta learning through the lens of generalization, 202… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abbas-2023-semdedup-data-efficient-learning-web/