Elamur Research⌕
☰
home / papers / 2021 / PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

Summary

GPT-3 showed that huge autoregressive models can solve many tasks from only a few examples, or none. However, it was trained mainly on English and is reachable only through a restricted API. This technical report presents PanGu-α, a family of Chinese autoregressive language models scaling up to 200 billion parameters. The authors describe it as the largest Chinese PLM at the time. The architecture is a unidirectional Transformer decoder topped with an extra query layer, which uses an embedding of the next position as the attention query when predicting the following token. Pretraining used a 1.1TB Chinese corpus, filtered and cleaned from about 80TB of raw text. Training ran on 2048 Ascend 910 processors under MindSpore Auto-parallel, which combines five kinds of parallelism. Perplexity drops as model capacity grows, and larger variants generally perform better in few-shot and zero-shot settings across many Chinese NLP tasks. The models also generate fluent dialogue, fiction, summaries and answers. The work offers a major non-English PLM and an open-sourced recipe for large-scale distributed training.

Key points

  • Three model sizes were trained on the same corpus: PanGu-α 2.6B, PanGu-α 13B and PanGu-α 200B. Few-shot benchmarks were run for the 2.6B and 13B versions.

  • The corpus (1.1TB of high-quality Chinese text) combines Common Crawl, e-Books, encyclopedias, news and other sources, followed by several filtering and cleaning stages.

  • MindSpore Auto-parallel combines five techniques:

    • splitting training data across devices,
    • partitioning individual operators across devices,
    • dividing model layers into sequential pipeline stages,
    • optimizer model parallelism,
    • rematerialization.

    This parallel functionality is open-sourced in MindSpore and needs little or no code change compared with single-device training.

  • Evaluation covered cloze tasks, reading comprehension, closed-book QA, Winograd-style tasks, commonsense reasoning, NLI and text classification. Results generally improved with model size.

  • Release of the model weights, code or pretraining data was still under consideration when the report was published (possibly via APIs).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

71 papers of our library cite it (of the 140884 whose bibliographies we read); 17 of them have a page here, 54 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2104.12369 · arXiv 2104.12369

@misc{zeng2021pangu,
  title = {{PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation}},
  author = {Zeng, Wei and Ren, Xiaozhe and Su, Teng and Wang, Hui and Liao, Yi and Wang, Zhiwei and Jiang, Xin and Yang, ZhenZhang and Wang, Kaisheng and Zhang, Xiaoda and Li, Chen and Gong, Ziyan and Yao, Yifan and Huang, Xinjing and Wang, Jun and Yu, Jianfeng and Guo, Qi and Yu, Yue and Zhang, Yan and Wang, Jin and Tao, Hengtao and Yan, Dasen and Yi, Zexuan and Peng, Fang and Jiang, Fangqing and Zhang, Han and Deng, Lingfeng and Zhang, Yehong and Lin, Zhe and Zhang, Chao and Zhang, Shaojie and Guo, Mingyue and Gu, Shanzhi and Fan, Gaojun and Wang, Yaowei and Jin, Xuefeng and Liu, Qun and Tian, Yonghong},
  year = {2021},
  doi = {10.48550/arxiv.2104.12369},
  eprint = {2104.12369},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2104.12369}
}

References

42 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, C… find
  2. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL-HLT, pages 4171–4186, 2019. find
  3. Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin, Yusheng Su, Haozhe Ji, Jian Guan, Fanchao Qi, Xiaozhi Wang, Yanan Zheng, Guoyang Zeng, Huanqi Cao, Shengqi Chen, Daixuan Li, Zhenbo Sun, Zhiyuan Liu, Minlie Huang, Wentao Han, Jie Tang, Juanzi Li, Xiaoyan Zhu, and Maosong Sun… find
  4. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V. Le. XLNet: Generalized autoregressive pretraining for language understanding. In Proc. NeurIPS, pages 5753–5763, 2019. find our card
  5. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692, 2019. find
  6. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21:1–67, 2020. find
  7. Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. ERNIE: Enhanced representation through knowledge integration. arXiv preprint arXiv:1904.09223, 2019. find our card
  8. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. ERNIE: Enhanced language representation with informative entities. In Proc. ACL, pages 1441–1451, 2019. find
  9. Junqiu Wei, Xiaozhe Ren, Xiaoguang Li, Wenyong Huang, Yi Liao, Yasheng Wang, Jiashu Lin, Xin Jiang, Xiao Chen, and Qun Liu. Nezha: Neural contextualized representation for chinese language understanding, 2019. find
  10. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. 2018. find
  11. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  12. Heng Liao, Jiajin Tu, Jing Xia, and Xiping Zhou. DaVinci: A scalable architecture for neural network computing. In IEEE Hot Chips 31 Symposium (HCS), pages 1–44, 2019. find
  13. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. NeurIPS, pages 5998–6008, 2017. find
  14. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization in the transformer architecture. In Proc. ICML, pages 10524–10533, 2020. find
  15. Liang Xu, Xuanwei Zhang, and Qianqian Dong. CLUECorpus2020: A large-scale Chinese corpus for pre-training language model. arXiv preprint arXiv:2003.01355, 2020. find
  16. Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, Jie Zhang, Jianwei Zhang, Xu Zou, Zhikang Li, Xiaodong Deng, Jie Liu, Jinbao Xue, Huiling Zhou, Jianxin Ma, Jin Yu, Yong Li, Wei Lin, Jingren Zhou, Jie Tang, and Hongxia Yang. M6: A chine… find
  17. Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanantakool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake Hechtman. Mesh-TensorFlow: Deep learning for supercomputers. In Proc. NeurIPS, 2018. find our card
  18. Zhihao Jia, Sina Lin, Charles R. Qi, and Alex Aiken. Exploring hidden dimensions in accelerating convolutional neural networks. In Proc. ICML, 2018. find
  19. Minjie Wang, Chien-chin Huang, and Jinyang Li. Supporting very large models using automatic dataflow graph partitioning. In Proc. EuroSys, 2019. find
  20. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. GShard: Scaling giant models with conditional computation and automatic sharding. arXiv preprint arXiv:2006.16668, 2020. find
  21. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. find
  22. Linghao Song, Fan Chen, Youwei Zhuo, Xuehai Qian, Hai Li, and Yiran Chen. Accpar: Tensor partitioning for heterogeneous deep learning accelerators. In Proc. HPCA, 2020. find
  23. Zhihao Jia, Matei Zaharia, and Alex Aiken. Beyond data and model parallelism for deep neural networks. In A. Talwalkar, V. Smith, and M. Zaharia, editors, Proc. MLSys, 2019. find
  24. Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Yonghui Wu, and zhifeng Chen. GPipe: Efficient training of giant neural networks using pipeline parallelism. In Proc. NeurIPS, 2019. find
  25. Deepak Narayanan, Aaron Harlap, Amar Phanishayee, Vivek Seshadri, Nikhil R. Devanur, Gregory R. Ganger, Phillip B. Gibbons, and Matei Zaharia. PipeDream: Generalized pipeline parallelism for DNN training. In Proc. SOSP, 2019. find
  26. Shiqing Fan, Yi Rong, Chen Meng, Zongyan Cao, Siyu Wang, Zhen Zheng, Chuan Wu, Guoping Long, Jun Yang, Lixue Xia, Lansong Diao, Xiaoyong Liu, and Wei Lin. DAPPLE: A pipelined data parallel approach for training large models. In Proc. PPoPP, 2021. find
  27. Jakub M Tarnawski, Amar Phanishayee, Nikhil Devanur, Divya Mahajan, and Fanny Nina Paravecino. Efficient algorithms for device placement of dnn graph operators. In Proc. NeurIPS, 2020. find
  28. Jay H. Park, Gyeongchan Yun, Chang M. Yi, Nguyen T. Nguyen, Seungmin Lee, Jaesik Choi, Sam H. Noh, and Young ri Choi. Hetpipe: Enabling large DNN training on (whimpy) heterogeneous GPU clusters through integration of pipelined model parallelism and data parallelism. In Proc. USENIX ATC, 2020. find
  29. Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In Proc. SC, 2020. find
  30. Chujie Zheng, Minlie Huang, and Aixin Sun. ChID: A large-scale Chinese IDiom dataset for cloze test. In Proc. ACL, pages 778–787, 2019. find
  31. Yiming Cui, Ting Liu, Zhipeng Chen, Shijin Wang, and Guoping Hu. Consensus attention-based neural networks for Chinese reading comprehension. In Proc. COLING, pages 1777–1786, 2016. find
  32. Yiming Cui, Ting Liu, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. Dataset for the first evaluation on Chinese machine reading comprehension. In Proc. International Conference on Language Resources and Evaluation (LREC), 2018. find
  33. Yiming Cui, Ting Liu, Ziqing Yang, Zhipeng Chen, Wentao Ma, Wanxiang Che, Shijin Wang, and Guoping Hu. A sentence cloze dataset for Chinese machine reading comprehension. In Proc. COLING, pages 6717–6723, 2020. find
  34. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proc. ACL, pages 1525–1534, 2016. find our card
  35. Yiming Cui, Ting Liu, Wanxiang Che, Li Xiao, Zhipeng Chen, Wentao Ma, Shijin Wang, and Guoping Hu. A span-extraction dataset for Chinese machine reading comprehension. In Proc. EMNLP-IJCNLP, pages 5886–5891, 2019. find our card
  36. Chih Chieh Shao, Trois Liu, Yuting Lai, Yiying Tseng, and Sam Tsai. DRCD: A Chinese machine reading comprehension dataset. arXiv preprint arXiv:1806.00920, 2018. find
  37. Wei He, Kai Liu, Jing Liu, Yajuan Lyu, Shiqi Zhao, Xinyan Xiao, Yuan Liu, Yizhong Wang, Hua Wu, Qiaoqiao She, Xuan Liu, Tian Wu, and Haifeng Wang. DuReader: A Chinese machine reading comprehension dataset from real-world applications. arXiv preprint arXiv:1711.05073, 2017. find our card
  38. Peng Li, Wei Li, Zhengyan He, Xuguang Wang, Ying Cao, Jie Zhou, and Wei Xu. Dataset and neural recurrent sequence labeling model for open-domain factoid question answering. arXiv preprint arXiv:1607.06275, 2016. — [arXiv:1607.06275](https://arxiv.org/abs/1607.06275) find
  39. Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipe… find our card
  40. Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer, 2017. find
  41. William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity, 2021. find
  42. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeng-2021-pangu-alpha-large-scale/