Elamur Research
☰
home / papers / 2022 / Text and Code Embeddings by Contrastive Pre-Training

Text and Code Embeddings by Contrastive Pre-Training

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2201.10005

Summary

Text embeddings power semantic search, classification and similarity scoring. In earlier work, each use case usually got its own dataset, training objective and architecture, and sentence-embedding research developed separately from neural retrieval. This OpenAI paper shows that one simple recipe can produce general-purpose embeddings for both text and code. A Transformer encoder starts from pre-trained generative weights: GPT for cpt-text and Codex for cpt-code. It is then trained contrastively with in-batch negatives on pairs that occur naturally and need no labels. For text, the pairs are adjacent text fragments from the Internet. For code, they are top-level docstrings matched with their function bodies from open-source repositories. The embedding is the final-layer state at a closing delimiter token, and similarity is measured by cosine. The authors found very large batches essential. On text tasks, quality grows steadily as models scale from 300M to 175B parameters. A single unsupervised model sets new records in linear-probe classification and in code search, and it clearly beats earlier unsupervised retrievers. Its weak spot is sentence similarity, which gets worse with longer training.

Key points

  • The cpt-text family has four sizes: S (300M), M (1.2B), L (6B) and XL (175B). Embedding dimensions run from 1024 to 12288, and batch sizes drop from 12288 for S to 4976 for XL.
  • On SentEval, averaged over 7 linear-probe tasks, the best unsupervised model improves on the previous best unsupervised embeddings by 4% (relative) and on the best supervised ones (SimCSE) by 1.8%. Fine-tuning on NLI data adds another 2.2% over the previous best transfer method.
  • Unsupervised search beats BM25 by 23.4% on MSMARCO (4M passages). Over 21M Wikipedia documents it beats Contriever by 14.7% on Natural Questions and 10.6% on TriviaQA. After fine-tuning on MS-MARCO, it gains 5.2% on zero-shot BEIR tasks.
  • cpt-code is trained on (text, code) pairs and sets a new state of the art on CodeSearchNet, 20.8% above GraphCodeBERT. Growing it from 300M to 1.2B parameters brings no improvement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

170 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 155 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2201.10005 · arXiv 2201.10005

@misc{neelakantan2022text,
  title = {{Text and Code Embeddings by Contrastive Pre-Training}},
  author = {Neelakantan, Arvind and Xu, Tao and Puri, Raul and Radford, Alec and Han, Jesse Michael and Tworek, Jerry and Yuan, Qiming and Tezak, Nikolas and Kim, Jong Wook and Hallacy, Chris and Heidecke, Johannes and Shyam, Pranav and Power, Boris and Nekoul, Tyna Eloundou and Sastry, Girish and Krueger, Gretchen and Schnurr, David and Such, Felipe Petroski and Hsu, Kenny and Thompson, Madeleine and Khan, Tabarak and Sherbakov, Toki and Jang, Joanne and Welinder, Peter and Weng, Lilian},
  year = {2022},
  doi = {10.48550/arxiv.2201.10005},
  eprint = {2201.10005},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2201.10005}
}

References

82 references, 43 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bengio, Y., Courville, A. C., and Vincent, P. Representation learning: A review and new perspectives. Transactions on pattern analysis and machine intelligence, 35 (8), 2012. find our card
  2. Bolukbasi, T., Chang, K., Zou, J. Y., Saligrama, V., and Kalai, A. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. 29, 2016. find our card
  3. D. Language models are few-shot learners. In Advances in Neural Information Processing Systems, 2020. find
  4. Caliskan, A., Bryson, J. J., and Narayanan, A. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186, 2017. find
  5. Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, H. P., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., Ray, A., Puri, R., Krueger, G., Petrov, M., Khlaaf, H., Sastry, G., Mishkin, P., Chan, B., Gray, S., Ryder, N., Pavlov, M., Power, A., Kaiser, L., Bavarian, M., Winter, C.… find our card
  6. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. E. A simple framework for contrastive learning of visual representations. In International conference on machine learning (ICML), 2020. find our card
  7. Chopra, S., Hadsell, R., and LeCun, Y. Learning a similarity metric discriminatively, with application to face verification. In Computer Vision and Pattern Recognition (CVPR). IEEE, 2005. find
  8. Conneau, A. and Kiela, D. Senteval: An evaluation toolkit for universal sentence representations. arXiv preprint arXiv:1803.05449, 2018. find our card
  9. Crawford, K. The trouble with bias. Keynote at NeurIPS, 2017. find
  10. Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference. In Conference on Empirical Methods in Natural Language Processing (EMNLP). ACL, 2015. find
  11. Deerwester, S., Dumais, S., Furnas, G., Landauer, T., and Harshman, R. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391–407, 1990. find
  12. Brown, P. F., Della Pietra, V. J., deSouza, P. V., Lai, J. C., and Mercer, R. L. Class-based n-gram models of natural language. Computational Linguistics, 18(4):467– 480, 1992. find
  13. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). ACL, 2019. find
  14. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess… — [arXiv:2005.12766](https://arxiv.org/abs/2005.12766) find
  15. Feng, Z., Guo, D., Tang, D., Duan, N., Feng, X., Gong, M., Shou, L., Qin, B., Liu, T., Jiang, D., and Zhou, M. Codebert: A pre-trained model for programming and natural languages. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. find our card
  16. Formal, T., Lassance, C., Piwowarski, B., and Clinchant, S. SPLADE v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10086, 2021. find
  17. Gao, T., Yao, X., and Chen, D. SimCSE: Simple contrastive learning of sentence embeddings. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. find our card
  18. Giorgi, J. M., Nitski, O., Bader, G. D., and Wang, B. Declutr: Deep contrastive learning for unsupervised textual representations. In Proceedings of ACL/IJCNLP, 2020. find our card
  19. Goodman, N. Seven strictures on similarity. Bobbs Merrill, 1972. find
  20. Guo, D., Ren, S., Lu, S., Feng, Z., Tang, D., Liu, S., Zhou, L., Duan, N., Svyatkovskiy, A., Fu, S., Tufano, M., Deng, S. K., Clement, C. B., Drain, D., Sundaresan, N., Yin, J., Jiang, D., and Zhou, M. Graphcodebert: Pre-training code representations with data flow. In International Conference on Le… find our card
  21. Gutmann, M. and Hyvärinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Conference on Artificial Intelligence and Statistics. PMLR, 2010. find
  22. Guu, K., Lee, K., Tung, Z., Pasupat, P., and Chang, M. REALM: retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. find
  23. Hadsell, R., Chopra, S., and LeCun, Y. Dimensionality reduction by learning an invariant mapping. In Computer Vision and Pattern Recognition (CVPR), volume 2, pp. 1735–1742. IEEE, 2006. find
  24. He, K., Fan, H., Wu, Y., Xie, S., and Girshick, R. B. Momentum contrast for unsupervised visual representation learning. In Computer Vision and Pattern Recognition (CVPR), 2020. find our card
  25. Izacard, G., Caron, M., Hosseini, L., Riedel, S., Bojanowski, P., Joulin, A., and Grave, E. Towards unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021. find our card
  26. Jain, P., Jain, A., Zhang, T., Abbeel, P., Gonzalez, J. E., and Stoica, I. Contrastive code representation learning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. find
  27. Jia, C., Yang, Y., Xia, Y., Chen, Y.-T., Parekh, Z., Pham, H., Le, Q. V., Sung, Y., Li, Z., and Duerig, T. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), 2021. find our card
  28. Johnson, J., Douze, M., and Jégou, H. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 2019. find our card
  29. Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Conference of the Association for Computational Linguistics (ACL). ACL, 2017. find our card
  30. Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020a. find our card
  31. Karpukhin, V., Oguz, B., Min, S., Wu, L., Edunov, S., Chen, D., and Yih, W. Dense passage retrieval for open-domain question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020b. find our card
  32. Khosla, P., Teterwak, P., Wang, C., Sarna, A., Tian, Y., Isola, P., Maschinot, A., Liu, C., and Krishnan, D. Supervised contrastive learning. arXiv preprint arXiv:2004.11362, 2020. — [arXiv:2004.11362](https://arxiv.org/abs/2004.11362) find
  33. Kim, W., Son, B., and Kim, I. Vilt: Vision-and-language transformer without convolution or region supervision. In International Conference on Machine Learning (ICML), 2021. find our card
  34. Hofstätter, S., Lin, S., Yang, J., Lin, J., and Hanbury, A. Efficiently teaching an effective dense retriever with balanced topic aware sampling. arXiv preprint arXiv:2104.06967, 2021. find our card
  35. Kingma, D. P. and Welling, M. Auto-Encoding Variational Bayes. In International Conference on Learning Representation (ICLR), 2014. find our card
  36. Husain, H., Wu, H.-H., Gazit, T., Allamanis, M., and Brockschmidt, M. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436, 2020. find our card
  37. Kiros, J., Zhu, Y., Salakhutdinov, R., Zemel, R. S., Torralba, A., Urtasun, R., and Fidler, S. Skip-thought vectors. In Advances in Neural Information Processing Systems (NeuriPS), 2015. find
  38. Kobayashi, S. Contextual augmentation: Data augmentation by words with paradigmatic relations. arXiv preprint arXiv:1805.06201, 2018. find our card
  39. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Kelcey, M., Devlin, J., Lee, K., Toutanova, K. N., Jones, L., Chang, M.-W., Dai, A., Uszkoreit, J., Le, Q., and Petrov, S. Natural questions: a benchmark for question answering research. T… find our card
  40. Lee, K., Chang, M., and Toutanova, K. Latent retrieval for weakly supervised open domain question answering. In Korhonen, A., Traum, D. R., and Màrquez, L. (eds.), Conference of the Association for Computational Linguistics (ACL), pp. 6086–6096. ACL. Lewis, P. S. H., Perez, E., Piktus, A., Petroni,… find our card
  41. Li, B., Zhou, H., He, J., Wang, M., Yang, Y., and Li, L. On the sentence embeddings from pre-trained language models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. find our card
  42. Liang, P. P., Li, I. M., Zheng, E., Lim, Y. C., Salakhutdinov, R., and Morency, L. Towards debiasing sentence representations. In Conference of the Association for Computational Linguistics (ACL), 2020. find our card
  43. Lin, J., Nogueira, R., and Yates, A. Pretrained transformers for text ranking: BERT and beyond. Synthesis Lectures on Human Language Technologies, 14(4):1–325, 2021. find
  44. Logeswaran, L. and Lee, H. An efficient framework for learning sentence representations. In International Conference on Learning Representation (ICLR), 2018. find our card
  45. Lu, J., Batra, D., Parikh, D., and Lee, S. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. arXiv preprint arXiv:1908.02265, 2019. find our card
  46. Nguyen, T., Rosenberg, M., Song, X., Gao, J., Tiwary, S., Majumder, R., and Deng, L. MS MARCO: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268, 2016. find our card
  47. Nogueira, R., Lin, J., and Epistemic, A. From doc2query to doctttttquery. Online preprint, 2019a. find
  48. Nogueira, R., Yang, W., Cho, K., and Lin, J. Multi-stage document ranking with BERT. arXiv preprint arXiv:1910.14424, 2019b. — [arXiv:1910.14424](https://arxiv.org/abs/1910.14424) find
  49. Park, J. H., Shin, J., and Fung, P. Reducing gender bias in abusive language detection. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018. find our card
  50. Pennington, J., Socher, R., and Manning, C. GloVe: Global vectors for word representation. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014. find
  51. Peters, M. E., Neumann, M., Iyyer, M., Gardner, M., Clark, C., Lee, K., and Zettlemoyer, L. Deep contextualized word representations. In Proceedings of NCAAL/IJCNLP, 2018. find
  52. Qu, Y., Ding, Y., Liu, J., Liu, K., Ren, R., Zhao, X., Dong, D., Wu, H., and Wang, H. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. In Conference of the Association for Computational Linguistics (ACL), 2021. find our card
  53. Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.00020, 2021. find our card
  54. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683, 2019. find
  55. Ramesh, A., Pavlov, M., Goh, G., Gray, S., Voss, C., Radford, A., Chen, M., and Sutskever, I. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092, 2021. find our card
  56. May, C., Wang, A., Bordia, S., Bowman, S. R., and Rudinger, R. On measuring social biases in sentence encoders. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019. find our card
  57. Reimers, N. and Gurevych, I. Sentence-bert: Sentence embeddings using siamese bert-networks. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019. find
  58. Mikolov, T., Chen, K., Corrado, G. S., and Dean, J. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013. find
  59. Robertson, S. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends® in Information Retrieval, 2009. find our card
  60. Rudinger, R., Naradowsky, J., Leonard, B., and Durme, B. V. Gender bias in coreference resolution. arXiv preprint arXiv:1804.09301, 2018. find our card
  61. Sachan, D. S., Patwary, M., Shoeybi, M., Kant, N., Ping, W., Hamilton, W. L., and Catanzaro, B. End-to-end training of neural retrievers for open-domain question answering. In Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of ACL/IJCNLP, pp. 6648–6662. ACL, 2021. find
  62. Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., and Zaharia, M. Colbertv2: Effective and efficient retrieval via lightweight late interaction. arXiv preprint arXiv:2112.01488, 2021. find our card
  63. Schroff, F., Kalenichenko, D., and Philbin, J. Facenet: A unified embedding for face recognition and clustering. In Computer Vision and Pattern Recognition (CVPR), 2015. find our card
  64. Shen, D., Zheng, M., Shen, Y., Qu, Y., and Chen, W. A simple but tough-to-beat data augmentation approach for natural language understanding and generation. arXiv preprint arXiv:2009.13818, 2020. — [arXiv:2009.13818](https://arxiv.org/abs/2009.13818) find
  65. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013. find
  66. Sohn, K. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems (NeuriPS), 2016. find
  67. Solaiman, I. and Dennison, C. Process for adapting language models to society (PALMS) with values-targeted datasets. arXiv preprint arXiv:2106.10328, 2021. find
  68. Su, J., Cao, J., Liu, W., and Ou, Y. Whitening sentence representations for better semantics and faster retrieval. arXiv preprint arXiv:2103.15316, 2021. — [arXiv:2103.15316](https://arxiv.org/abs/2103.15316) find
  69. Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C. Videobert: A joint model for video and language representation learning. In International Conference on Computer Vision (ICCV), 2019. find our card
  70. Thakur, N., Reimers, N., Rücklé, A., Srivastava, A., and Gurevych, I. BEIR: A heterogenous benchmark for zero-shot evaluation of information retrieval models. In Advances in Neural Information Processing Systems (NeuriPS), 2021. find our card
  71. Tian, Y., Krishnan, D., and Isola, P. Contrastive multi-view coding. European Conference on Computer Vision (ECCV), 2019. van den Oord, A., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K. Wavenet: A generative model for raw audio. arXiv… find our card
  72. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems (NeuriPS), 2017. find
  73. Vervaeke, J., Lillicrap, T. P., and Richards, B. A. Relevance realization and the emerging framework in cognitive science. Journal of logic and computation, 22(1):79–99, 2012. find
  74. Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., and Zhou, M. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. arXiv preprint arXiv:2002.10957, 2020. find
  75. Wei, J. W. and Zou, K. EDA: easy data augmentation techniques for boosting performance on text classification tasks. arXiv preprint arXiv:1901.11196, 2019. find our card
  76. Williams, A., Nangia, N., and Bowman, S. A broad-coverage challenge corpus for sentence understanding through inference. In Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). ACL, 2018. find
  77. Wu, Z., Xiong, Y., Yu, S. X., and Lin, D. Unsupervised feature learning via non-parametric instance-level discrimination. In Computer Vision and Pattern Recognition (CVPR), 2018. find
  78. Xiong, L., Xiong, C., Li, Y., Tang, K., Liu, J., Bennett, P. N., Ahmed, J., and Overwijk, A. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808, 2020. find our card
  79. Yih, W.-t., Toutanova, K., Platt, J. C., and Meek, C. Learning discriminative projections for text similarity measures. In Conference on Computational Natural Language Learning (CoNLL). ACL, 2011. find
  80. Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning (ICML), 2021. find
  81. Zhang, Y., He, R., Liu, Z., Lim, K. H., and Bing, L. An unsupervised sentence embedding method by mutual information maximization. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020. find
  82. Zhao, J., Wang, T., Yatskar, M., Ordonez, V., and Chang, K. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876, 2018. find our card