探索中文预训练模型的混合粒度编码和IDF遮蔽

Summary

Most Chinese pre-trained language models encode text character by character. This produces long input sequences and high Transformer cost. Word-level encoding shortens the input, but it depends on an external segmenter and suffers from out-of-vocabulary words and sparse data. Shao, Sun and Qiu (Fudan University) propose a BERT-base-style model with a mixed-grained vocabulary, in which characters, words and phrases coexist. The vocabulary is learned directly from the pre-training corpus with a SentencePiece unigram language model, and sentences are tokenized by Viterbi decoding. Any unseen word falls back to single characters, so OOV is avoided.

The larger vocabulary has a long-tailed token distribution, which could slow training. To compensate, the authors introduce IDF-masking: the model preferentially masks rare, informative words chosen by corpus-level inverse document frequency rather than masking at random.

The model was trained from scratch on 13GB of open data. It matches or beats BERT, RoBERTa and multi-granularity models such as AMBERT, LICHEE-BERT and Lattice-BERT on CLUE and other Chinese benchmarks. It also shortens inputs and speeds up inference, especially for long texts.

Key points

• Vocabulary comparison:
  • The mixed-grained vocabulary has 60k entries, about 58k of them Chinese tokens. Chinese BERT has 20k entries, about 7k Chinese.
  • Encoded-to-raw length ratio falls from 0.969 with BERT to 0.565, measured on 5,000 sampled sentences.
• Pre-training data and setup:
  • Corpus: 13GB, about 9M documents. SentencePiece was trained on 20M sampled sentences.
  • IDF lexicon: about 1M words, with words of frequency below 10 removed.
  • Model and schedule: BERT-base architecture, 1M steps, batch size 384, maximum length 512. Training took roughly 7 days on 8 RTX3090 GPUs.
• How IDF-masking works:
  • A conventional segmenter splits each sentence into words.
  • The top 30% of words by IDF form a candidate pool.
  • 15% of the sentence's words are sampled from this pool for masking.
  • Masking is applied to the contiguous mixed-grained tokens that cover each chosen word.
• Results:
  • Mixed-grained encoding with IDF-masking reaches 76.13% average accuracy on the CLUE dev set.
  • In the ablation, IDF-masking beats whole-word masking for both character-level and mixed-grained models, with the larger gain for the mixed-grained one.
  • Best (mean) test accuracies: ChnSentiCorp 96.2 (96.0), THUCNews 98.0 (97.9), LCQMC 87.7 (87.5).
• Limitation: The authors leave tasks that need fine-grained, segmentation-aligned outputs to future work. These include POS tagging, NER and span-based reading comprehension.