Elamur Research⌕
☰
home / papers / 2023 / EVA-CLIP: Improved Training Techniques for CLIP at Scale

EVA-CLIP: Improved Training Techniques for CLIP at Scale

Paper · 2023 · Quan Sun, Yuxin Fang, Ledell Wu et al.

Summary

Training CLIP-style image-text models at large scale is expensive and often unstable. EVA-CLIP, from the Beijing Academy of Artificial Intelligence, is a model family built around a set of training techniques that cut cost while raising zero-shot quality. The image encoder starts from EVA weights (EVA-01 or EVA-02, pre-trained with masked image modeling), and the text encoder starts from OpenAI CLIP or OpenCLIP. Optimization uses LAMB, which suits very large batches. Following FLIP, half of the image tokens are randomly dropped, and flash attention further speeds up training. The models are trained on LAION-400M, LAION-2B, or a Merged-2B dataset. They match or beat CLIP and OpenCLIP models of the same size after seeing far fewer image-text pairs. The flagship EVA-02-CLIP-E/14+ reaches 82.0% zero-shot top-1 on ImageNet-1K. The results hold across 27 image classification sets, several video benchmarks, and Flickr30K/COCO retrieval. Because all models are publicly released, strong CLIP backbones become accessible to groups with modest compute.

Key points

  • Top results: EVA-02-CLIP-E/14+ (5.0B parameters, 9B samples seen) scores 82.0% zero-shot on ImageNet-1K and 80.9% averaged over ImageNet variants and ObjectNet. Its gap between ImageNet and the robustness average is only 1.1 points.
  • Efficiency vs. OpenCLIP: EVA-02-CLIP-L/14+ (~430M parameters, 6B samples) reaches 80.4%. That beats OpenCLIP-G/14 (80.1%) with roughly one sixth of the parameters and training samples.
  • Ablations (ViT-B/16, shortened schedule):
    • EVA initialization adds 1.8 points while using about half the samples.
    • Replacing AdamW with LAMB adds another 0.7.
    • 50% token masking costs 0.7 but doubles speed.
    • Merged-2B reaches the same accuracy with half the samples.
  • Training cost: On 16 A100-40GB GPUs, masking plus flash attention lowers time per 40M samples from 132 to 55 minutes and memory per GPU from 33GB to 16GB. EVA-CLIP-B/16 converges in about 300 hours.
  • Retrieval: EVA-02-CLIP-E/14 trails OpenCLIP-G/14 slightly. The E/14+ variant, with a larger (695M) text encoder, surpasses OpenCLIP-G/14 on zero-shot text retrieval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Huazhong University of Science and TechnologyWuhan · China45 works · 6,345 citations🔬Beijing Academy of Artificial IntelligenceBeijing · China32 works · 7,511 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Artificial intelligence · 49 works

Cite

Cite as Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang and Yue Cao. 2023. EVA-CLIP: Improved Training Techniques for CLIP at Scale. arXiv:2303.15389.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.15389 · arXiv 2303.15389

@misc{sun2023eva,
  title = {{EVA-CLIP: Improved Training Techniques for CLIP at Scale}},
  author = {Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang and Yue Cao},
  year = {2023},
  doi = {10.48550/arxiv.2303.15389},
  eprint = {2303.15389},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.15389}
}

References

4 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Reaching 80 zero-shot accuracy with openclip: Vit-g/14 trained on laion-2b. https://laion.ai/blog/giant-openclip/.4 ↗
  2. Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint arXiv:2106.08254, 2021. 5, 6 — [arXiv:2106.08254](https://arxiv.org/abs/2106.08254) find
  3. Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurIPS, 2019. 3 find
  4. Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle L Alexander, David W Jacobs, and Peter N Belhumeur. Birdsnap: Large-EVA-01-CLIP-g / EVA-02-CLIP-g+ config find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2023-eva-clip-improved-training-techniques/