玲珑：一个小规模的高质量中文预训练语言模型

Summary

Large autoregressive Chinese PLMs are expensive to train and deploy. They rely on word-level vocabularies, which bring segmentation errors and out-of-vocabulary problems, and they only see left-to-right context. LingLong is a compact GPT-style decoder (about 317M parameters) designed to address all three issues. The authors collected roughly 0.72 TB of public Chinese text (news, encyclopedia, Common Crawl, patents, education, fiction). After extraction, rule-based cleaning, deduplication and filtering, about 50 GB remained for pre-training. Instead of word segmentation, they use a character-level vocabulary anchored in the official standard character list. They also train a second, identical model on reversed text, LingLong_B, so that downstream predictions can draw on both directions. Every downstream task is reformulated as text generation using forward and reversed prompt templates. Across tasks from the CUGE benchmark, LingLong beats similarly sized Chinese PLMs on six datasets and outperforms several much larger models on others. This shows that a small, cheaply deployable model can remain competitive.

Key points

• Model size and architecture: 316,989,440 trainable parameters, with sparse multi-head self-attention decoder blocks. Embedding and hidden size are both 1,024, and the context window is 1,024 tokens. Pre-training ran on 20 NVIDIA Tesla V100S GPUs with a global batch size of 80.
• Vocabulary: 13,312 tokens, about a quarter to half the size of word-based Chinese vocabularies. It is seeded with all 6,500 level-1 and level-2 characters of the 通用规范汉字表 (List of Commonly Used Standard Chinese Characters). Frequent other characters and non-Chinese tokens from the corpus are added on top.
• Training data: about 50 GB of cleaned text, with news the largest share (36.26 GB). The cleaning pipeline builds on the authors' earlier NKCorpus work.
• Evaluation datasets: CEPSUM 2.0, LCSTS, AdGen, E-Reviews, KBQA, Weibo/MSR word segmentation, LCQMC, Math23K, CMRC and SIGHAN13. For Math23K, the model generates a formula, which is then computed with Python's eval.
• Reported results and release: the English abstract claims LingLong beats large models on four tasks, while the Chinese abstract says five datasets. Code and pre-trained weights for LingLong and LingLong_B are released under GNU GPLv3.