Elamur Research⌕
☰
home / papers / 2026 / CLASE: A Hybrid Method for Chinese Legalese Stylistic Evaluation

CLASE: A Hybrid Method for Chinese Legalese Stylistic Evaluation

Summary

LLMs can write Chinese legal text that is factually sound but stylistically off. It may be too colloquial or use non-standard terms, or it may overcorrect with verbose, archaic phrasing and even invented legal concepts. Expert rubrics for these tacit norms are hard to write. Reference-based metrics blur meaning with style, and plain LLM judges are opaque and inconsistent. The authors propose CLASE, a reference-free metric for the stylistic quality of Chinese legal writing.

They start from the reasoning sections of real judgment documents. An LLM first strips away the professional style, then reconstructs it, producing contrastive pairs of authentic versus LLM-restored text with no human scoring. These pairs are used to learn weights for linguistic features and to build a pool of scoring "experiences" for an LLM judge. The final score combines both signals.

On 200 Chinese legal documents, CLASE matches human judgments considerably better than traditional metrics and pure LLM-as-a-judge baselines. It also returns interpretable score breakdowns and concrete revision suggestions, which makes it practical for legal text generation pipelines.

Key points

  • Three-stage pipeline:
    • Contrastive pair synthesis: t_reverse = π1(t_gold), then t_restored = π2(t_reverse).
    • Training-free contrastive learning that fills experience pools.
    • Hybrid scoring.
  • Two components:
    • CLASE-Obj uses linguistic features weighted by logistic regression. It works best with k = 25 features, and adding more features introduces noise.
    • CLASE-Subj is an experience-guided LLM judge that gives natural-language feedback.
  • Why hybrid: feature-only scoring used as a reinforcement learning reward risks Goodhart's Law, since models could game surface markers. The LLM component adds semantic grounding and actionable advice.
  • Limitations:
    • The method is tested only on Chinese civil judgments, and the objective features are language-specific.
    • The synthetic pairs were not manually audited at scale.
    • It is costlier to run than BLEU.
  • Code and data: github.com/rexera/CLASE (LREC 2026, pp. 642–653).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Law

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2xrbbj7oaghv

@inproceedings{ma2026clase,
  title = {{CLASE: A Hybrid Method for Chinese Legalese Stylistic Evaluation}},
  author = {Ma, Yiran Rex and Ye, Yuxiao and Xie, Huiyuan},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2xrbbj7oaghv},
  url = {https://doi.org/10.63317/2xrbbj7oaghv}
}

References

57 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shlomo Argamon, Casey Whitelaw, Paul Chase, Sobhan Raj Hota, Navendu Garg, and Shlomo Levitan. 2007. Stylistic text classification using functional lexical features. Journal of the American Society for Information Science and Technology, 58(6):802–822. ↗
  2. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72. find our card
  3. Yehonatan Bitton, Elad Bitton, and Shai Nisan. 2025. Detecting stylistic fingerprints of large language models. arXiv preprint arXiv:2503.01659. ↗
  4. Ilias Chalkidis, Manos Fergadiotis, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. Legal-bert: The muppets straight out of law school. arXiv preprint arXiv:2010.02559. ↗ our card
  5. Ilias Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. Lexglue: A benchmark dataset for legal language understanding in english. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4310–4330. ↗ our card
  6. Chi-Min Chan, Weize Chen, Yujia Su, Jiahui Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Zhiyuan Liu. 2023. Chateval: Towards better llm-based evaluators through multi-agent debate. arXiv preprint arXiv:2308.07201. ↗ our card
  7. Court Writing Committee. 2010. Court writing guide. Legal writing guidelines for professional practice. find
  8. Jiaxi Cui, Zongjian Li, Yang Yan, Bohua Chen, and Li Yuan. 2023. Chatlaw: Open-source legal large language model with integrated external knowledge bases. CoRR. find
  9. Yongfu Dai, Duanyu Feng, Jimin Huang, Haochen Jia, Qianqian Xie, Yifang Zhang, Weiguang Han, Wei Tian, and Hao Wang. 2025. LAiW: A Chinese legal large language models benchmark. In Proceedings of the 31st International conference on computational linguistics, pages 10738–10766. find
  10. Aniket Deroy, Kripabandhu Ghosh, and Saptarshi Ghosh. 2023. How ready are pre-trained abstractive models and llms for legal case judgement summarization? arXiv preprint arXiv:2306.01248. ↗
  11. Zhiwei Fei, Xiaoyu Shen, Dawei Zhu, Fengzhe Zhou, Zhuo Han, Alan Huang, Songyang Zhang, Kai Chen, Zhixin Yin, Zongwen Shen, et al. 2024. Lawbench: Benchmarking legal knowledge of large language models. In Proceedings of the 2024 conference on empirical methods in natural language processing, pages 7933– 7962. ↗ our card
  12. Richard Foley. 2002. Language, law and legal writing: An introduction to legal discourse. Legal Writing: The Journal of the Legal Writing Institute, 8:1–35. find
  13. Ruoli Gan, Duanyu Feng, Chen Zhang, Zhihang Lin, Haochen Jia, Hao Wang, Zhenyang Cai, Lei Cui, Qianqian Xie, Jimin Huang, et al. 2025. Ucl-bench: A chinese user-centric legal benchmark for large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 7945–7988. find
  14. Neel Guha, Julian Nyarko, Daniel Ho, Christopher Ré, Adam Chilton, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel Rockmore, Diego Zambrano, et al. 2023. Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in neural information processing systems, 36:44123–44279. ↗ our card
  15. Abe Bohan Hou, William Jurayj, Nils Holzenberger, Andrew Blair-Stanek, and Benjamin Van Durme. 2024. Gaps or hallucinations? gazing into machine-generated legal analysis for fine-grained text evaluations. ↗
  16. Quzhe Huang, Mingxu Tao, Chen Zhang, Zhenwei An, Cong Jiang, Zhibin Chen, Zirui Wu, and Yansong Feng. 2023. Lawyer llama technical report. ↗ our card
  17. Takumi Ito, Kees van Deemter, and Jun Suzuki. 2025. Reference-free evaluation metrics for text generation: A survey. ↗
  18. Hang Jiang, Xiajie Zhang, Robert Mahari, Daniel Kessler, Eric Ma, Tal August, Irene Li, Alex Pentland, Yoon Kim, Deb Roy, et al. 2024. Leveraging large language models for learning complex legal concepts through storytelling. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7194–7219. ↗
  19. Di Jin, Zhijing Jin, Zhiting Hu, Olga Vechtomova, and Rada Mihalcea. 2022. Deep learning for text style transfer: A survey. Computational Linguistics, 48(1):155–205. ↗
  20. Patrick Juola et al. 2008. Authorship attribution. Foundations and Trends® in Information Retrieval, 1(3):233–334. find
  21. Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo. 2024. GPT-4 passes the bar exam. Philosophical Transactions of the Royal Society A, 382(2270):20230254. ↗
  22. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1/2):81–93. ↗
  23. Klaus Krippendorff. 2011. Computing Krippendorff’s Alpha Reliability. University of Pennsylvania. find
  24. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles. find
  25. Haitao Li, Jiaying Ye, Yiran Hu, Jia Chen, Qingyao Ai, Yueyue Wu, Junjie Chen, Yifan Chen, Cheng Luo, Quan Zhou, and Yiqun Liu. 2025. Casegen: A benchmark for multi-stage legal case documents generation. arXiv preprint arXiv:2502.17943. ↗
  26. Huixian Li. 2022. Lexical, syntactic and textual features of shipping legal documents in chinese and english. SCIREA Journal of Sociology, 6(2):83–103. ↗
  27. Shifang Li and Yifan Wang. 2021. A study of cohesion in the chinese legal text: Based on criminal procedure law of the people’s republic of china. Theory and Practice in Language Studies, 11(12):1709–1716. ↗
  28. Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, and Chongyang Tao. 2024. Leveraging large language models for nlg evaluation: A survey. CoRR. find
  29. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81. find
  30. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. Geval: Nlg evaluation using gpt-4 with better human alignment. find
  31. Nan Lu and Chuanyou Yuan. 2021. Legal reasoning: A textual perspective on common law judicial opinions and chinese judgments. Text & Talk, 41(1):71–93. ↗
  32. Yiran Rex Ma. 2025. Do androids question electric sheep? a multi-agent cognitive simulation of philosophical reflection on hybrid table reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 143–164, Vienna, Austria. Association for Computational Linguistics. ↗
  33. Chris Mellish and Robert Dale. 1998. Evaluation in the context of natural language generation. Computer Speech & Language, 12(4):349–373. ↗
  34. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. arXiv preprint arXiv:1707.06875. ↗
  35. OpenAI. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276. ↗ our card
  36. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. ↗ our card
  37. Ajay Patel, Jiacheng Zhu, Justin Qiu, Zachary Horvitz, Marianna Apidianaki, Kathleen McKeown, and Chris Callison-Burch. 2025. StyleDistance: Stronger content-independent style embeddings with synthetic parallel examples. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 8662–8685, Albuquerque, New Mexico. Association for Computational Linguistics. ↗
  38. Karl Pearson. 1895. Note on regression and inheritance in the case of two parents. Proceedings of the royal society of London, 58(347-352):240– 242. find
  39. Xinying Qiu, Kebin Deng, Likun Qiu, and Xin Wang. 2018. Exploring the Impact of Linguistic Features for Chinese Readability Assessment. In Xuanjing Huang, Jing Jiang, Dongyan Zhao, Yansong Feng, and Yu Hong, editors, Natural Language Processing and Chinese Computing, volume 10619, pages 771–783. Springer International Publishing, Cham. find
  40. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. 2025. Qwen2.5 technical report. find
  41. Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan Van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André FT Martins, and Alon Lavie. 2021. Are references really needed? unbabel-ist 2021 submission for the metrics shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 1030– 1040. find our card
  42. Ehud Reiter and Anja Belz. 2009. An investigation into the validity of some metrics for automatically evaluating natural language generation systems. Computational Linguistics, 35(4):529–558. ↗
  43. Jaromir Savelka, Kevin D. Ashley, Morgan A. Gray, Hannes Westermann, and Huihui Xu. 2023. Explaining legal concepts with augmented large language models (gpt-4). find
  44. Peizhang Shao, Linrui Xu, Jinxi Wang, Wei Zhou, and Xingyu Wu. 2025. When large language models meet law: Dual-lens taxonomy, technical advances, and ethical governance. ↗
  45. Charles Spearman. 1904. The proof and measurement of association between two things. The American journal of psychology, 15(1):72–101. ↗
  46. Weihang Su, Baoqing Yue, Qingyao Ai, Yiran Hu, Jiaqi Li, Changyue Wang, Kaiyuan Zhang, Yueyue Wu, and Yiqun Liu. 2025. JuDGE: Benchmarking judgment document generation for chinese legal system. find
  47. Yuxiu Sun and Le Cheng. 2017. Linguistic variation and legal representation in legislative discourse: A corpus-based multi-dimensional study. International Journal of Legal Discourse, 2(2):315–339. ↗
  48. Peter Meijes Tiersma. 1999. Legal Language. University of Chicago Press. find
  49. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al. 2023. A survey on large language model based autonomous agents. arXiv preprint arXiv:2308.11432. ↗ our card
  50. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837. find our card
  51. Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, and Jianfeng Xu. 2018. Cail2018: A large-scale legal dataset for judgment prediction. ↗ our card
  52. Li yunhan and Wu gengshen. 2025. LegalEval-Q: A new benchmark for the quality evaluation of llm-generated legal text. find
  53. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. ↗
  54. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 563–578, Hong Kong, China. Association for Computational Linguistics. ↗ our card
  55. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  56. Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang, Zhiyuan Liu, and Maosong Sun. 2020. Jec-qa: a legal-domain question answering dataset. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 9701–9708. find
  57. Zhi Zhou, Jiang-Xin Shi, Peng-Xiao Song, Xiao-Wen Yang, Yi-Xuan Jin, Lan-Zhe Guo, and Yu-Feng Li. 2024. Lawgpt: A chinese legal knowledge-enhanced large language model. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/ma-2026-clase-hybrid-method-chinese-legalese/