Elamur Research
☰
home / papers / 2023 / Sigmoid Loss for Language Image Pre-Training

Sigmoid Loss for Language Image Pre-Training

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2303.15343

Summary

Contrastive image-text pre-training methods such as CLIP and ALIGN usually use a softmax loss, which normalizes similarity scores over the whole batch. On multiple devices this requires all-gathers, holding a full |B|×|B| similarity matrix in memory, and an extra pass over the batch for numerical stability. The authors replace it with a pairwise sigmoid loss. Each image-text combination in a batch becomes a separate binary decision: matching pairs are positives and all other pairs are negatives. A learnable temperature and bias are added. With no global normalization, the loss can be computed in "chunks": devices pass text embeddings around so that only a small block is in memory at any time. The loss is tested in two setups: training from scratch (SigLIP) and Locked-image Tuning with a frozen image tower (SigLiT). Sigmoid clearly beats softmax at batch sizes under 16k and matches it at larger ones. It also makes batches of up to one million possible, though gains level off near 32k. As a result, strong vision-language models can be trained on only a few accelerators.

Key points

  • SigLiT with a frozen g/14 image backbone reaches 84.5% zero-shot ImageNet accuracy after two days on four TPUv4 chips. A B/8 backbone gets about 79.7% after one day on the same hardware.
  • SigLIP B/16 trained on WebLI reaches 73.4% zero-shot ImageNet accuracy in 5 days on 32 TPUv4 chips. FLIP and CLIP needed roughly 5 and 10 days on 256 TPUv3 cores.
  • Initialization matters. With bias b = −10 and t′ = log 10, the Base model scores 63.0% on ImageNet. Without a bias it scores 62.0%, and with zero bias and t′ = log 1 it drops to 53.7%.
  • Both sigmoid and softmax stop improving as batch size grows, and about 32k is close to optimal. The same holds for multilingual SigLIP trained on over 100 languages.
  • Sigmoid-trained models stay more robust than softmax ones when images, texts or pair alignments are corrupted. The models are released in the big_vision repository.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

367 papers of our library cite it (of the 140673 whose bibliographies we read); 9 of them have a page here, 358 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.15343 · arXiv 2303.15343

@misc{zhai2023sigmoid,
  title = {{Sigmoid Loss for Language Image Pre-Training}},
  author = {Zhai, Xiaohua and Mustafa, Basil and Kolesnikov, Alexander and Beyer, Lucas},
  year = {2023},
  doi = {10.48550/arxiv.2303.15343},
  eprint = {2303.15343},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.15343}
}

References

13 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ibrahim Alabdulmohsin, Xiaohua Zhai, Alexander Kolesnikov, and Lucas Beyer. Getting vit in shape: Scaling laws for compute-optimal model design. In NeurIPS, 2023. 7, 8, 17 find
  2. Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. ObjectNet: A large-scale bias-controlled dataset for pushing the limits of object recognition models. In NeurIPS, 2019. 7, 17 find
  3. Lucas Beyer, Olivier J. Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord. Are we done with imagenet? CoRR, abs/2006.07159, 2020. 2, 7, 9, 17 — [arXiv:2006.07159](https://arxiv.org/abs/2006.07159) find
  4. Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for imagenet-1k, 2022. 10, 17 find
  5. Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Big vision. https://github.com/google-research/ big\_vision, 2022. 10, 17 ↗
  6. Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon Kim. Coyo-700m: Image-text pair dataset. https://github.com/ kakaobrain/coyo-dataset, 2022. 1 ↗
  7. Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021. 1 find
  8. Feilong Chen, Duzhen Zhang, Minglun Han, Xiu-Yi Chen, Jing Shi, Shuang Xu, and Bo Xu. VLP: A survey on vision-language pre-training. Int. J. Autom. Comput., 20(1):38–56, 2023. 2 find
  9. Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pre-training from pixels. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Researc… find
  10. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 2, 4 find our card
  11. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325, 2015. 7, 17 find our card
  12. Xiangning Chen, Chen Liang, Da Huang, Esteban Real, Kaiyuan Wang, Yao Liu, Hieu Pham, Xuanyi Dong, Thang Luong, Cho-Jui Hsieh, Yifeng Lu, and Quoc V. Le. Symbolic discovery of optimization algorithms, 2023. 2, 6 find
  13. Xi Chen, Xiao Wang, Soravit Changpinyo, A. J. Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Ku… find our card