CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model

Paper · 2020 · Liang Xu, Xuanwei Zhang and Qianqian Dong

Summary

Large English pre-training corpora such as C4 and WebText are openly available. Chinese has had only small public collections, such as Chinese Wikipedia (about 1 GB) and THUCTC (about 2 GB). The CLUE organization addresses this gap with CLUECorpus2020, also called C5. It contains roughly 100 GB of Chinese text taken from Common Crawl snapshots from July to December 2019. The text was cleaned with C4-inspired heuristics adapted for Chinese and stored in a format ready for pre-training.

The authors also built vocab_clue, a reduced character vocabulary about one-third the size of Google's Chinese BERT vocabulary. To test the corpus, they ran experiments on the CLUE benchmark:

BERT-base trained on a 1 GB slice of C5 outperformed the same model trained on Wiki.
More data and more training steps improved results further.
A large model, RoBERTa-large-clue, matched the strongest Chinese baseline, RoBERTa-wwm-large.

The corpus, vocabulary, code, and models are all released on GitHub. This gives Chinese NLP a reusable large-scale resource for self-supervised training.

Key points

Corpus scale and format: C5 holds about 35 billion Chinese characters in roughly 29k files. It is split randomly 99 : 0.5 : 0.5 into train, dev, and test. Each line holds one sentence, and documents are separated by blank lines.
Cleaning pipeline:
- Lines are trimmed back to Chinese terminal punctuation.
- Sentences with entries from a profanity list, curly brackets, or "Javascript" are dropped.
- Repeated four-sentence spans are deduplicated.
- Text is split into sentences with pyltp, and only sentences longer than 5 characters are kept.
vocab_clue: About 8K tokens, removing traditional Chinese, Japanese, Korean, emoji, and redundant tokens. It gives about 9.84% fewer parameters and about 15% faster pre-training than BERT-base, with comparable or slightly better CLUE scores.
Corpus quality check: On 1 GB with 125K steps, BERT-base reached masked-LM accuracy of 77.94% on C5 versus 72.24% on Wiki.
Released models:
- RoBERTa-large-clue averages 69.68 on CLUE, against 69.64 for RoBERTa-wwm-large.
- RoBERTa-tiny-clue (4 layers, hidden size 312) averages 63.60 and runs roughly 8–10× faster than BERT-base.
- RoBERTa-pair is pre-trained on CMNLI and improves AFQMC by about 0.8–4 points.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields, topics and concepts

📚 Shelves 4Computer Science · 16,667 works Artificial Intelligence · 16,194 works Arts and Humanities · 16,039 works Language and Linguistics · 15,891 works

🧭 Topics 1Language models & pretraining · 3,320 works

Cite

Cite as Liang Xu, Xuanwei Zhang and Qianqian Dong. 2020. CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model. arXiv:2003.01355.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2003.01355 · arXiv 2003.01355

@misc{xu2020cluecorpus2020,
  title = {{CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model}},
  author = {Liang Xu, Xuanwei Zhang and Qianqian Dong},
  year = {2020},
  doi = {10.48550/arxiv.2003.01355},
  eprint = {2003.01355},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.01355}
}

References

1 references, 0 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Wanxiang Che, Zhenghua Li, and Ting Liu. 2010. Ltp: A chinese language technology platform. In Proceedings of the 23rd International Conference on Computational Linguistics: Demonstrations, pages 13–16. Association for Computational Linguistics. \#\# Conclusion find

Updated 30 October 2026

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2020-cluecorpus2020-large-scale-chinese/