Elamur Research⌕
☰
home / papers / 2026 / ChangJuan: A Comprehensive Benchmark for Book-Length Chinese Story Evaluation

ChangJuan: A Comprehensive Benchmark for Book-Length Chinese Story Evaluation

Paper · 2026 · Dingyi Yang, Mingshuo Wang and Qin Jin

Summary

Automatically judging the quality of full-length novels is still poorly studied, especially for languages other than English. The authors present ChangJuan, which they describe as the first benchmark for evaluating book-length Chinese fiction. It covers 300 novels published in the last five years, with the newest titles held out for testing to reduce contamination risk. For each novel the benchmark provides metadata, average ratings and user reviews collected from Douban. Individual reviews are subjective, so the authors built an LLM-based pipeline that turns them into consensus pros and cons, grouped under a unified taxonomy of evaluation aspects. They also propose an improved summary-based evaluator. It draws on summaries that balance length against detail, plus selected excerpts, and it weights aspects by genre before producing per-aspect reviews and an overall score. This approach matches or beats methods that read the entire book. Training an 8B model, CLEM, within this framework clearly improves agreement with human ratings. The work therefore provides a reproducible and culturally specific resource for long-form story evaluation.

Key points

  • Dataset scale: ChangJuan's novels average about 292K tokens. The benchmark contains 97.5K aspect-guided reviews and 22.1K distilled general viewpoints. For copyright reasons, only summaries are released, not the full texts.
  • Viewpoint distillation pipeline:
    • Deepseek-Chat extracts viewpoints from the reviews.
    • High-vote and long-tail reviews are processed separately, which keeps minority opinions in the final set.
    • All viewpoints are mapped onto eight major aspects.
  • Viewpoint quality: Automatic checks measure precision at 0.946 and recall at 0.932. Human verification gives 0.91.
  • CLEM results: CLEM outperforms open-source baselines. It lifts Qwen3's Kendall's tau with human judgments from 24.8 to 34.1.
  • Genre differences: Aspect importance varies by genre. Historical and military fiction depend most on world-building, while mystery and suspense depend most on creativity and plot.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanyang Technological UniversitySingapore · Singapore184 works · 27,484 citations🎓Renmin University of ChinaBeijing · China139 works · 17,752 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Dingyi Yang, Mingshuo Wang and Qin Jin. 2026. ChangJuan: A Comprehensive Benchmark for Book-Length Chinese Story Evaluation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2044

@inproceedings{yang2026changjuan,
  title = {{ChangJuan: A Comprehensive Benchmark for Book-Length Chinese Story Evaluation}},
  author = {Dingyi Yang, Mingshuo Wang and Qin Jin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2044},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2044}
}

References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jian Guan, Zhexin Zhang, Zhuoer Feng, Zitao Liu, Wenbiao Ding, Xiaoxi Mao, Changjie Fan, and Minlie Huang. 2021. Openmeva: A benchmark for evaluating open-ended story generation metrics. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internati… find
  2. Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit Iyyer. 2024. Booookscore: A systematic exploration of book-length summarization in the era of llms. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net.
  3. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. \#\#…
  4. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. find
  5. Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, Yuxuan Lai, Chongyang Tao, and Shuai Ma. 2024. Leveraging large language models for nlg evaluation: Advances and challenges. Preprint, arXiv:2401.07103.
  6. Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. 2024. Deepseek-v2: A strong, economical, and efficient mixture-of-experts language model. arXiv preprint arXiv:2405.04434, arXiv:2405.04434. find
  7. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022.… — Training language models to follow instructions with human feedback
  8. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311–318. ACL.
  9. Karl Pearson. 1895. Vii. note on regression and inheritance in the case of two parents. proceedings of the royal society of London, 58(347-352):240–242. find
  10. Charles Spearman. 1961. The proof and measurement of association between two things. The American Journal of Psychology, 15:72–101. find
  11. Danqing Wang, Kevin Yang, Hanlin Zhu, Xiaomeng Yang, Andrew Cohen, Lei Li, and Yuandong Tian. 2024. Learning personalized story evaluation. Preprint, arXiv:2310.03304. — [arXiv:2310.03304](https://arxiv.org/abs/2310.03304) find
  12. Zhuohan Xie, Trevor Cohn, and Jey Han Lau. 2023. The next chapter: A study of large language models in storytelling. Preprint, arXiv:2301.09790. find
  13. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… — Qwen3 Technical Report
  14. Dingyi Yang and Qin Jin. 2025. What matters in evaluating book-length stories? a systematic study of long story evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16375– 16398, Vienna, Austria. Association for Computa… find
  15. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2026-changjuan-comprehensive-benchmark-book-length/