Elamur Research⌕
☰
home / papers / 2021 / CPM-2: Large-scale Cost-effective Pre-trained Language Models

CPM-2: Large-scale Cost-effective Pre-trained Language Models

Paper · 2021 · Zhengyan Zhang, Yuxian Gu, Xu Han et al.

Summary

Pre-trained language models now reach hundreds of billions of parameters, and three costs make them hard to use. Pre-training needs enormous compute, every downstream task needs its own stored fine-tuned copy, and inference usually requires several GPUs. The authors, from Tsinghua University and BAAI, address all three with one pipeline. To speed up pre-training, they use knowledge inheritance: each training stage starts from the previous model instead of from scratch. Training moves from Chinese-only data, to Chinese-English data, to an expansion into a Mixture-of-Experts model. For adaptation, they study prompt tuning, which saves only prompt embeddings. They find that where the prompt is placed matters a lot, and that combining prompt tuning with fine-tuning gives better results. For deployment, they build InfMoE, a toolkit that schedules parameter offloading so very large MoE models can run on a single GPU. The outcome is CPM-2, a bilingual encoder-decoder trained on WuDaoCorpus, and its MoE variant. CPM-2 slightly outperforms mT5-XXL on the CUGE benchmark. Code and weights are public, which lowers the barrier to large-model research.

Key points

  • Models: CPM-2 is an 11B-parameter Chinese-English encoder-decoder (24 layers, d_model 4,096). CPM-2-MoE has 198B parameters. It uses top-1 expert routing, with expert load balanced by the BASE Layers approach.
  • Pre-training data: WuDaoCorpus, with about 2.3TB of cleaned Chinese text and 300GB of cleaned English text.
  • Pre-training objective: span masking that corrupts 15% of tokens, with an average span length of 10.
  • Tokenizer: sentencepiece was replaced by a simpler prefix-matching tokenizer.
  • Bilingual stage:
    • English embeddings were initialized from the embeddings of their prefixes.
    • English and Chinese data were mixed at a 1:2 ratio to limit catastrophic forgetting.
  • MoE stage: the bilingual model was copied to create the experts. Routing used a fixed random-projection hashing gate that was not updated during this stage.
  • Evaluation: experiments used CUGE, a benchmark built from 40 Chinese datasets, of which the authors picked one representative dataset per language capability.
    • On the lite version of CUGE (the first 6 datasets), the overall score was 198 for CPM-2 versus 190 for mT5-XXL.
  • Efficiency:
    • Prompt tuning stores less than 0.01% of model parameters per task.
    • InfMoE overlaps parameter transfers with computation, enabling single-GPU inference of CPM-2-MoE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Zhengyan Zhang, Yuxian Gu, Xu Han, Shengqi Chen, Chaojun Xiao, Zhenbo Sun, Yuan Yao, Fanchao Qi, Jian Guan, Pei Ke, Yanzheng Cai, Guoyang Zeng, Zhixing Tan, Zhiyuan Liu, Minlie Huang, Wentao Han, Yang Liu, Xiaoyan Zhu and Maosong Sun. 2021. CPM-2: Large-scale Cost-effective Pre-trained Language Models. arXiv:2106.10715.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2106.10715 · arXiv 2106.10715

@misc{zhang2021cpm,
  title = {{CPM-2: Large-scale Cost-effective Pre-trained Language Models}},
  author = {Zhengyan Zhang, Yuxian Gu, Xu Han, Shengqi Chen, Chaojun Xiao, Zhenbo Sun, Yuan Yao, Fanchao Qi, Jian Guan, Pei Ke, Yanzheng Cai, Guoyang Zeng, Zhixing Tan, Zhiyuan Liu, Minlie Huang, Wentao Han, Yang Liu, Xiaoyan Zhu and Maosong Sun},
  year = {2021},
  doi = {10.48550/arxiv.2106.10715},
  eprint = {2106.10715},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2106.10715}
}

References

44 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Loïc Barrault, Magdalena Biesialska, Ondřej Bojar, Marta R. Costa-jussà, Christian Federmann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Sant… — Findings of the 2020 Conference on Machine Translation (WMT20)
  2. Yoshua Bengio. 2013. Deep learning of representations: Looking forward. In Proceedings of SLSP, volume 7978, pages 1–37. Springer. find
  3. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  4. Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. 2016. Training deep nets with sublinear memory cost. arXiv preprint, arXiv:1604.06174. find
  5. Weize Chen, Xu Han, Yankai Lin, Hexu Zhao, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2021. Fully hyperbolic neural networks. Technical report. find
  6. J. Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT.
  7. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv pre…
  8. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint, arXiv:2101.03961. find
  9. Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: word-level adversarial reprogramming. arXiv preprint arXiv:2101.00121. find
  10. Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021. PTR: Prompt tuning with rules for text classification. arXiv preprint, arXiv:2105.11259. find
  11. Sariel Har-Peled, Piotr Indyk, and Rajeev Motwani. 2012. Approximate nearest neighbor: Towards removing the curse of dimensionality. Theory Comput., 8(1):321–350. find
  12. Jiaao He, Jiezhong Qiu, Aohan Zeng, Zhilin Yang, Jidong Zhai, and Jie Tang. 2021. FastMoE: A Fast Mixture-of-Expert Training System. arXiv preprint, arXiv:2103.13262. find
  13. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of CVPR.
  14. Baotian Hu, Qingcai Chen, and Fangze Zhu. 2015. Lcsts: A large scale chinese short text summarization dataset. In Proceedings of EMNLP, pages 1967– 1972. find
  15. Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. 1991. Adaptive mixtures of local experts. Neural Comput., 3(1):79–87. find
  16. Taku Kudo and John Richardson. 2018. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of EMNLP.
  17. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. 2020. Gshard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint, arXiv:2006.16668. find
  18. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.
  19. Mike Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, and Luke Zettlemoyer. 2021. BASE layers: Simplifying training of large, sparse models. arXiv preprint, arXiv:2103.16716. find
  20. Wenhao Li, Fanchao Qi, Maosong Sun, Xiaoyuan Yi, and Jiarui Zhang. 2021. Ccpm: A chinese classical poetry matching dataset. arXiv preprint arXiv:2106.01979. — [arXiv:2106.01979](https://arxiv.org/abs/2106.01979) find
  21. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. find
  22. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. GPT understands, too. arXiv preprint arXiv:2103.10385.
  23. Xin Liu, Qingcai Chen, Chong Deng, Huajun Zeng, Jing Chen, Dongfang Li, and Buzhou Tang. 2018a. LCQMC:a large-scale Chinese question matching corpus. In Proceedings of COLING, pages 1952– 1962.
  24. Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2018b. Entity-duet neural ranking: Understanding the role of knowledge graph semantics in neural information retrieval. In Proceedings of ACL, pages 2395–2405. find
  25. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2018. Mixed precision training. In Proceedings of ICLR. find
  26. Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying lms with mixtures of soft prompts. In Proceedings of NAACL-HLT, pages 5203–5212.
  27. Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2021. Knowledge inheritance for pre-trained language models. arXiv preprint, arXiv:2105.13880. find
  28. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
  29. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. Zero: Memory optimizations toward training trillion parameter models. In Proceedings of SC, pages 1–16. find
  30. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of KDD, pages 3505–3506. find
  31. Timo Schick and Hinrich Schütze. 2020a. Exploiting cloze questions for few-shot text classification and natural language inference. arXiv preprint, arXiv:2001.07676.
  32. Timo Schick and Hinrich Schütze. 2020b. It’s not just size that matters: Small language models are also few-shot learners. arXiv preprint, arXiv:2009.07118.
  33. Zhihong Shao, Minlie Huang, Jiangtao Wen, Wenfei Xu, et al. 2019. Long and diverse text generation with planning-based hierarchical variational model. In Proceedings of EMNLP-IJCNLP, pages 3248– 3259. find
  34. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053. find
  35. Chenglei Si, Zhengyan Zhang, Yingfa Chen, Fanchao Qi, Xiaozhi Wang, Zhiyuan Liu, and Maosong Sun. 2021. ShuoWenJieZi: Linguistically informed tokenizers for chinese language model pretraining. Technical report. find
  36. Jianlin Su. 2020. Wobert: Word-based chinese bert model - zhuiyiai. Technical report. find
  37. Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. 2020. Investigating prior knowledge for challenging chinese machine reading comprehension. TACL, 8:141–155.
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NeurIPS, pages 5998– 6008.
  39. Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017. Deep neural solver for math word problems. In Proceedings of EMNLP, pages 845–854. find
  40. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint, arXiv:2010.11934.
  41. Yuan Yao, Qingxiu Dong, Jian Guan, Boxi Cao, Fanchao Qi, Jinliang Lu, Jinran Nie, Junwei Bao, Kun Zhou, Shuhuai Ren, Xiaozhi Wang, Xuancheng Huang, Zheni Zeng, Zile Zhou, Zhiyuan Liu, Erhong Yang, Zhifang Sui, Maosong Sun, Jiajun Zhang, Juanzi Li, Minlie Huang, Rui Yan, Xianpei Han, Xiaodong He, Xia… find
  42. Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, and Zhilin Yang. 2021. Wudaocorpora: A super large-scale chinese corpora for pre-training language models. Preprint. find
  43. Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, et al. 2021. Pangu-α: Large-scale autoregressive pretrained chinese language models with auto-parallel computation. arXiv preprint arXiv:2104.12369.
  44. Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin, Yusheng Su, Haozhe Ji, Jian Guan, et al. 2020. CPM: A large-scale generative chinese pre-trained language model. arXiv preprint arXiv:2012.00413.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2021-cpm-2-large-scale/