Elamur Research⌕
☰
home / papers / 2021 / COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining

Summary

Masked-language-model pretraining becomes wasteful once a network has learned basic patterns. Encoders also tend to produce sentence vectors crowded into a narrow cone. ELECTRA made pretraining more efficient by training a main encoder to spot tokens swapped in by a small auxiliary generator. However, its binary objective leaves the encoder without language modeling ability and may further distort the representation space.

COCO-LM keeps the ELECTRA-style generator setup and replaces the training signal with two tasks:

  • Corrective Language Modeling (CLM): a token-level, multi-task objective in which the encoder both flags corrupted positions and recovers the original tokens.
  • Sequence Contrastive Learning (SCL): a sequence-level objective that pulls together two views derived from the same input (a corrupted version and a cropped version) while keeping representations spread out.

On GLUE and SQuAD, COCO-LM beats strong base- and large-sized baselines by more than one GLUE average point at equal training steps, and it reaches their MNLI accuracy with substantially less compute. Unlike ELECTRA, it can also be used for prompt-based fine-tuning.

Key points

  • COCO-LM matches the MNLI accuracy of RoBERTa using about 60% of its pretraining GPU hours, and that of ELECTRA using about 50%.

  • COCO-LM Large++ has 367M parameters and reaches the MNLI accuracy of Megatron 3.9B.

  • In few-shot prompt-based fine-tuning on MNLI-m (LM-BFF setup, 16 examples per class), COCO-LM outperforms RoBERTa at both sizes:

    SizeCOCO-LMRoBERTa
    Base++66.560.1
    Large++72.070.7

  • SCL helps most when labels are scarce:

    • With 1% of MNLI labels, it adds 0.8/0.5 points on MNLI-m/mm.
    • With only 10%/20% of labels, COCO-LM roughly equals RoBERTa (Ours)/ELECTRA (Ours) trained on the full label set.

  • Only about 7–8% of tokens are replaced by the generator. Code and models are released at github.com/microsoft/COCO-LM.

Interesting ideas

  • I1. Split a lopsided objective into explicit subtasks. Recovering tokens from a mostly unchanged sequence is mostly trivial copying. Adding a separate "was this token changed?" head stabilizes learning, whereas plain all-token MLM learned copying poorly. This trick could carry over to any denoising or error-correction model, for example spelling or OCR correction.
  • I2. Contrast cheap views from the pretraining pipeline itself. The positive pairs come from corruption and cropping that the pipeline already performs, so sequence-level regularization costs little extra data engineering. The paper shows that without such a term, even near-duplicate sequences can end up less similar than random pairs.
  • I3. Diagnose embedding geometry with similarity histograms. Comparing cosine-similarity distributions for random pairs versus related pairs is a quick, model-agnostic way to detect anisotropy before using embeddings for retrieval or clustering.
  • I4. Report label-fraction curves alongside full-data scores. Evaluating with 1–20% of training labels reveals benefits of representation regularization that full-data scores hide. This is especially relevant for low-resource languages and domains.
  • I5. Keep a generative head on discriminative pretraining. Retaining language modeling ability preserves compatibility with prompt-based and few-shot methods, at no loss in standard fine-tuning accuracy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · Computing & GPUs

Cited by

74 papers of our library cite it (of the 140884 whose bibliographies we read); 6 of them have a page here, 68 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2102.08473 · arXiv 2102.08473

@misc{meng2021coco,
  title = {{COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining}},
  author = {Meng, Yu and Xiong, Chenyan and Bajaj, Payal and Tiwary, Saurabh and Bennett, Paul and Han, Jiawei and Song, Xia},
  year = {2021},
  doi = {10.48550/arxiv.2102.08473},
  eprint = {2102.08473},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2102.08473}
}

References

63 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hangbo Bao, Li Dong, Furu Wei, Wenhui Wang, Nan Yang, Xiaodong Liu, Yu Wang, Songhao Piao, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. UniLMv2: Pseudo-masked language models for unified language model pre-training. In ICML, 2020. find our card
  2. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009. find
  3. Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In International Workshop on Semantic Evaluation (SemEval), 2017. find
  4. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. find our card
  5. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. What does BERT look at? an analysis of BERT’s attention. In ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2019. find
  6. Kevin Clark, Minh-Thang Luong, Quoc Le, and Christopher D Manning. Pre-training transformers as energy-based cloze models. In EMNLP, 2020. find
  7. Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. ELECTRA: Pre-training text encoders as discriminators rather than generators. In ICLR, 2020. find
  8. Peter Clark, Oyvind Tafjord, and Kyle Richardson. Transformers as soft reasoners over language. In IJCAI, 2020. find our card
  9. Andy Coenen, Emily Reif, Ann Yuan, Been Kim, Adam Pearce, Fernanda Viégas, and Martin Wattenberg. Visualizing and measuring the geometry of BERT. In NeurIPS, 2019. find our card
  10. Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, 2005. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019. find
  12. William B Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In International Workshop on Paraphrasing (IWP), 2005. find our card
  13. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In NeurIPS, 2019. find our card
  14. Jun Gao, Di He, Xu Tan, Tao Qin, Liwei Wang, and Tie-Yan Liu. Representation degeneration problem in training natural language generation models. In ICLR, 2019. find our card
  15. Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In ACL, 2021. find our card
  16. Tianyu Gao, Xingcheng Yao, and Danqi Chen. SimCSE: Simple contrastive learning of sentence embeddings. In EMNLP, 2021. find our card
  17. Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third pascal recognizing textual entailment challenge. In ACL-PASCAL workshop on textual entailment and paraphrasing, 2007. find
  18. Aaron Gokaslan and Vanya Cohen. Openwebtext corpus. http://Skylion007.github.io/OpenWebTextCorpus, 2019. ↗
  19. Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning. In ICLR, 2021. find our card
  20. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. REALM: Retrieval-augmented language model pre-training. In ICML, 2020. find
  21. R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. In PASCAL Challenges Workshop on Recognising Textual Entailment, 2006. find
  22. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. find our card
  23. Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-enhanced bert with disentangled attention. In ICLR, 2021. find our card
  24. Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. In TACL, 2019. find our card
  25. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. find
  26. Guolin Ke, Di He, and Tie-Yan Liu. Rethinking the positional encoding in language pre-training. In ICLR, 2021. find
  27. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. ALBERT: A lite BERT for self-supervised learning of language representations. In ICLR, 2020. find
  28. Teven Le Scao and Alexander M Rush. How many data points is a prompt worth? In NAACL-HLT, 2021. find our card
  29. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL, 2019. find our card
  30. Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. On the sentence embeddings from pre-trained language models. In EMNLP, 2020. find our card
  31. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019. find
  32. Yi Luan, Jacob Eisenstein, Kristina Toutanove, and Michael Collins. Sparse, dense, and attentional representations for text retrieval. In TACL, 2021. find our card
  33. Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. Text classification using label names only: A language model self-training approach. In EMNLP, 2020. find our card
  34. Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748, 2018. find our card
  35. Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. FAIRSEQ: A fast, extensible toolkit for sequence modeling. In NAACL-HLT Demonstrations, 2019. find our card
  36. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL-HLT, 2018. find
  37. Senthil Purushwalkam and Abhinav Gupta. Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases. In NeurIPS, 2020. find
  38. Yanru Qu, Dinghan Shen, Yelong Shen, Sandra Sajeev, Jiawei Han, and Weizhu Chen. CoDA: Contrast-enhanced and diversity-promoting data augmentation for natural language understanding. In ICLR, 2021. find
  39. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  40. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2019. find
  41. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP, 2016. find our card
  42. Anirudh Ravula, Chris Alberti, Joshua Ainslie, Li Yang, Philip Minh Pham, Qifan Wang, Santiago Ontanon, Sumit Kumar Sanghai, Vaclav Cvicek, and Zach Fisher. ETC: Encoding long and structured inputs in transformers. In EMNLP, 2020. find our card
  43. Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In EMNLP, 2019. find
  44. Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In EMNLP, 2020. find
  45. Timo Schick and Hinrich Schütze. Exploiting cloze questions for few-shot text classification and natural language inference. In EACL, 2021. find our card
  46. Timo Schick and Hinrich Schütze. It’s not just size that matters: Small language models are also few-shot learners. In NAACL-HLT, 2021. find our card
  47. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In ACL, 2015. find
  48. Iyer Shankar, Dandekar Nikhil, and Csernai Kornél. First Quora dataset release: Question pairs, 2017. find
  49. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using gpu model parallelism. arXiv preprint arXiv:1909.08053, 2019. find
  50. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013. find
  51. Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. In NAACL-HLT, 2021. find our card
  52. Trieu H Trinh and Quoc V Le. arXiv:1806.02847, 2018. A simple method for commonsense reasoning. find
  53. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, 2017. find
  54. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In EMNLP Workshop BlackboxNLP, 2018. find our card
  55. Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML, 2020. find
  56. Wei Wang, Bin Bi, Ming Yan, Chen Wu, Zuyi Bao, Jiangnan Xia, Liwei Peng, and Luo Si. StructBERT: Incorporating language structures into pre-training for deep language understanding. In ICLR, 2020. find
  57. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. In TACL, 2019. find
  58. Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL-HLT, 2018. find
  59. Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. CLEAR: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466, 2020. — [arXiv:2012.15466](https://arxiv.org/abs/2012.15466) find
  60. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In ICLR, 2021. find our card
  61. Zhenhui Xu, Linyuan Gong, Guolin Ke, Di He, Shuxin Zheng, Liwei Wang, Jiang Bian, and Tie-Yan Liu. MC-BERT: Efficient language pre-training via a meta controller. arXiv preprint arXiv:2006.05744, 2020. — [arXiv:2006.05744](https://arxiv.org/abs/2006.05744) find
  62. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLNet: Generalized autoregressive pretraining for language understanding. In NeurIPS, 2019. find our card
  63. Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In ICCV, 2015. arXiv preprint find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/meng-2021-coco-lm-correcting-contrasting-text/