📄 CPQD: 用于提升大语言模型文学表现力的中文诗词引用数据集

Summary

This paper asks how well large language models can weave classical Chinese poetry into modern prose. The authors frame poetry quotation as a sequence-to-sequence task: given a passage with a masked slot, the model must generate a fitting line of verse instead of picking one from a fixed candidate list. They mined quoted paragraphs from the MNBVC corpus and used a BERT+CNN poetry/non-poetry classifier to keep only real verse. They then filtered by length, context size and quote frequency, producing CPQD, along with an out-of-domain test set drawn from high-scoring Gaokao essays. They also propose automatic and human evaluation criteria for poeticness, contextual coherence and fluency. Eight Chinese and foreign LLMs, both open and commercial, were tested in zero-shot and few-shot modes. Only one domestic commercial model outperformed a fine-tuned LSTM baseline, and adding examples helped little. Supervised fine-tuning of open models on CPQD raised BLEU roughly tenfold, beating the commercial systems. This shows the dataset is useful for improving the literary quality of generated text.

Key points

• CPQD-MNBVC: 429,044 context–poem pairs covering about 35.5K distinct verses (35,561 or 35,562 depending on the section). It was built from roughly 25M quoted paragraphs found in 430GB of cleaned MNBVC text, and is released on ModelScope.
• Filtering rules: Verses are 5–25 characters long and at most 30% of the passage. Contexts are capped at 3 sentences and 256 characters, and each verse must occur at least 3 times.
• Classifier and splits:
  • The poetry classifier was trained on 13,672 poetry and 12,469 non-poetry items, reaching F1 of 93.38 (poetry) and 92.23 (non-poetry).
  • The verse inventory is long-tailed: 28,679 rare, 5,638 common and 1,245 frequent verses.
  • Validation and test sets contain 6,000 and 3,000 pairs, balanced across the three frequency groups.
• CPQD-Gaokao: 1,300 manually checked out-of-domain samples taken from 2,300 exam essays.
• Results:
  • Models tested include GPT-4, ERNIE-4.0, GLM-3-Turbo, GLM-4 and ChatGLM-6B/2-6B/3-6B.
  • In zero-shot mode, most models reached only 15–93% of the fine-tuned LSTM's BLEU.
  • Fine-tuning open models on CPQD raised their BLEU nearly 10×.