Elamur Research⌕
☰
home / papers / 2021 / CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation

CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation

Paper · 2021 · Yunfan Shao, Zhichao Geng, Yitao Liu et al.

Summary

Chinese pre-trained models are usually built either for understanding (BERT-style encoders) or for generation (GPT- or BART-style models). This forces practitioners to choose a backbone per task and prevents knowledge from being shared between NLU and NLG. The authors propose CPT, which splits a Transformer into three parts. A deep shared encoder (S-Enc) feeds two shallow heads. The first is an understanding decoder (U-Dec) with full self-attention, trained with masked language modeling using Whole Word Masking. The second is a generation decoder (G-Dec) with causal attention and cross-attention, trained with BART-style denoising auto-encoding. Both objectives are optimized jointly on unlabeled text. The two branches can be combined in several ways at fine-tuning time. The deep-shallow layout keeps the model's size close to BERT and BART, and the light generation decoder makes decoding much faster. Experiments covered classification, sequence labeling, reading comprehension, summarization and data-to-text tasks. CPT stayed competitive with strong Chinese baselines while generating faster than a comparable Chinese BART. This makes it a practical single backbone for mixed Chinese NLP workloads.

Key points

  • The architecture is unbalanced: a deep S-Enc plus two shallow decoders. U-Dec is pre-trained with MLM using WWM, and G-Dec with DAE using token infilling (one [MASK] per selected word) and sentence permutation.
  • Masking selects 15% of words after Chinese Word Segmentation. Of the masked characters, BERT's scheme replaces 80% with [MASK] and 10% with random tokens.
  • Classification tasks can be fine-tuned in at least five modes, using different combinations of the encoder and decoders.
  • Generation runs about 1.4–1.5× faster than BART at base size and up to 1.7× faster at large size, with comparable output quality. Evaluation sets include ADGEN data-to-text.
  • A Chinese BART is released as a by-product, and the code is available at github.com/fastnlp/CPT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China202 works · 22,509 citations🔬Zhejiang LabHangzhou · China9 works · 1,776 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

Cite

Cite as Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Hang Yan, Fei Yang, Li Zhe, Hujun Bao and Xipeng Qiu. 2021. CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation. arXiv:2109.05729.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1007/s11432-021-3536-5 · arXiv 2109.05729

@misc{shao2021cpt,
  title = {{CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation}},
  author = {Yunfan Shao, Zhichao Geng, Yitao Liu, Junqi Dai, Hang Yan, Fei Yang, Li Zhe, Hujun Bao and Xipeng Qiu},
  year = {2021},
  doi = {10.1007/s11432-021-3536-5},
  eprint = {2109.05729},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1007/s11432-021-3536-5}
}

References

7 references, 5 of them with a card of ours. ↗ leads to the work.

  1. X. Qiu, T. Sun, Y. Xu, Y. Shao, N. Dai, and X. Huang, “Pre-trained models for natural language processing: A survey,” SCIENCE CHINA Technological Sciences, vol. 63, no. 10, p. 1872–1897, 2020. J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for l…
  2. PMLR, 2020, pp. 642–652. [Online]. Available: http://proceedings.mlr.press/v119/bao20a.html Z. Du, Y. Qian, X. Liu, M. Ding, J. Qiu, Z. Yang, and J. Tang, “All NLP tasks are generation tasks: A general pretraining framework,” CoRR, vol. abs/2103.10360, 2021. [Online]. Available: https://arxiv.org/a… ↗
  3. AAAI Press, 2020, pp. 8968–8975. [Online]. Available: — GSum: A General Framework for Guided Neural Abstractive Summarization https://aaai.org/ojs/index.php/AAAI/article/view/6428 Z. Sun, X. Li, X. Sun, Y. Meng, X. Ao, Q. He, F. Wu, and J. Li, “Chinesebert: Chinese pretraining enhanced by glyph and pinyin information,” CoRR, vol. abs/2106.16038, 2021. [Online]. Availabl… ↗
  4. Z. Lin, A. Madotto, G. I. Winata, and P. Fung, “Mintl: Minimalist transfer learning for task-oriented dialogue systems,” in EMNLP 2020, Online, November 16-20, 2020, B. Webber, T. Cohn, Y. He, and Y. Liu, Eds., 2020, pp. 3391–3405. [Online]. Available: https://doi.org/10.18653/v1/2020.emnlp-main.273… ↗
  5. OpenReview.net, 2021. [Online]. Available: https://openreview.net/forum?id=KpfasTaLUpq X. Sun, T. Ge, F. Wei, and H. Wang, “Instantaneous grammatical error correction with shallow aggressive decoding,” in ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, C. Zong, F. Xia, W.… ↗
  6. ACL, 2005. [Online]. Available: https://www.aclweb.org/anthology/I05-3017/G. Levow, “The third international chinese language processing bakeoff: Word segmentation and named entity recognition,” in SIGHAN@COLING/ACL 2006, Sydney, Australia, July 22-23, 2006, H. T. Ng and O. O. Y. Kwong, Eds., 2006,… ↗
  7. EMNLP 2020, 2020, pp. 2887–2897. Y. Cui, T. Liu, W. Che, L. Xiao, Z. Chen, W. Ma, S. Wang, and G. Hu, “A span-extraction dataset for chinese machine reading comprehension,” in EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, K. Inui, J. Jiang, V. Ng, and X. Wan, Eds., 2019, pp. 5882–5888. [O…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shao-2021-cpt-pre-trained-unbalanced-transformer/