Elamur Research⌕
☰
home / papers / 2026 / From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks

From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks

Paper · 2026 · Qingyu Ren, Tianjun Pan, Xingzhou Chen et al.

Summary

LLMs still have trouble with writing tasks whose prompts contain many explicit constraints. Existing reward-model benchmarks such as RewardBench and JudgeBench score models only broadly. Current training approaches either depend on a costly LLM judge or on reward models that rate general qualities like fluency. The authors address both gaps with one idea: requirement dropout. They take complex writing prompts, remove progressively more of their constraints, and have DeepSeek-R1 answer each reduced prompt. Answers to prompts with fewer remaining constraints should satisfy the original request less well, which yields a ranking without manual labeling.

WEval uses these rankings as a benchmark. It checks how closely a reward model's ordering of the candidate answers matches the expected one. WRL uses the same answer pairs to train reward models with a Bradley–Terry objective, then uses those models as rewards for GRPO policy training.

The trained reward models improve writing performance and transfer to other benchmarks. Rankings from WEval agree with downstream RL results, which suggests WEval can predict how useful a reward model will be.

Key points

  • Seed prompts come from LMSYS-1M, WildChat and PRISM. They are sorted into five task categories (e.g., Creative Writing & Narrative, Long-form Academic Writing) by cosine similarity to category centroids built from 10 human-picked examples each.
  • Constraints added to prompts fall into four types: Content, Style, Format and Length. WEval scores reward models with three ranking metrics: Correlation, IL and PL.
  • In the dataset statistics, each candidate-response position (a1–a5) contains 550 items.
  • On RewardBench, a reward model trained this way from a Qwen3-32B base averaged 85.3, compared with 83.1 for the untrained base. Its category scores were 97.8 Chat, 85.1 Safety and 93.4 Reasoning, but it was weaker on Chat Hard (64.9). It averaged below Writing-Zero's Pairwise GenRM (87.4) while using a simpler training pipeline.
  • The stated limitations are that requirement types are limited and that the method was not tested on larger policy models. Code and data are available at github.com/Rainier-rq1/From_Coarse_to_Fine.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China117 works · 11,281 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

Cite

Cite as Qingyu Ren, Tianjun Pan, Xingzhou Chen and Xuhong Wang. 2026. From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.134

@inproceedings{ren2026coarse,
  title = {{From Coarse to Fine: Benchmarking and Reward Modeling for Writing-Centric Generation Tasks}},
  author = {Qingyu Ren, Tianjun Pan, Xingzhou Chen and Xuhong Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.134},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.134}
}

References

40 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. Longwriter: Unleashing 10,000+ word generation from long context llms. arXiv preprint arXiv:2408.07055.
  2. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324– 345. find
  3. Jianghao Chen, Wei Sun, Qixiang Yin, Zhixing Tan, and Jiajun Zhang. 2025. Ace-rl: Adaptive constraint-Fengxiang Cheng. 2026. Empowering llms with symbolic representation and reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 41040–41041. find
  4. Yujuan Ding, Yunshan Ma, Wenqi Fan, Yige Yao, Tat-Seng Chua, and Qing Li. 2024. Fashionregen: Llm-empowered fashion report generation. In Companion Proceedings of the ACM Web Conference 2024, pages 991–994. find
  5. Mingxuan Du, Benfeng Xu, Chiwei Zhu, Xiaorui Wang, and Zhendong Mao. 2025. Deepresearch bench: A comprehensive benchmark for deep research agents. arXiv preprint arXiv:2506.11763.
  6. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  7. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  8. Qianyu He, Jie Zeng, Qianxi He, Jiaqing Liang, and Yanghua Xiao. 2024. From complex to simple: Enhancing multi-constraint complex instruction following ability of large language models. arXiv preprint arXiv:2404.15846. find
  9. Hui Huang, Yancheng He, Wei Liu, Muyun Yang, Jiaheng Liu, Kehai Chen, Bing Xu, Conghui Zhu, Hailong Cao, and Tiejun Zhao. 2026. Long-form rewardbench: Evaluating reward models for long-form generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 31149– 31157. find
  10. Ruipeng Jia, Yunyi Yang, Yongbo Gai, Kai Luo, Shihao Huang, Jianhe Lin, Xiaoxi Jiang, and Guanjun Jiang. 2025. Writing-zero: Bridge the gap between non-verifiable tasks and verifiable rewards. arXiv preprint arXiv:2506.00103. — [arXiv:2506.00103](https://arxiv.org/abs/2506.00103) find
  11. Hannah R Kirk, Alexander Whitefield, Paul Röttger, Andrew Bean, Katerina Margatina, Juan Ciro, Rafael Mosquera, Max Bartolo, Adina Williams, He He, et al. 2024. The prism alignment dataset: What participatory, representative and individualised human feedback reveals about the subjective and multicul… find
  12. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, Lester James Validad Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. 2025. Rewardbench: Evaluating reward models for language modeling. In Findings of the Association for Computational Linguistics: N… find
  13. Weiyue Li, Mingxiao Song, Zhenda Shen, Dachuan Zhao, Yunfan Long, Yi Li, Yongce Li, Ruyi Yang, and Mengyu Wang. 2026. Llm review: Enhancing creative writing via blind peer review feedback. arXiv preprint arXiv:2601.08003. — [arXiv:2601.08003](https://arxiv.org/abs/2601.08003) find
  14. Jianxing Liao, Tian Zhang, Xiao Feng, Yusong Zhang, Rui Yang, Haorui Wang, Bosi Wen, Ziying Wang, and Runzhi Shi. 2025. Rlmr: Reinforcement learning with mixed rewards for creative writing. arXiv preprint arXiv:2508.18642. — [arXiv:2508.18642](https://arxiv.org/abs/2508.18642) find
  15. David Y Liu, Xanthe Muston, Aditya Joshi, and Sebastian Sequoiah-Grayson. 2026. Retell, reward, repeat: Reinforcement learning for narrative theory-informed story generation. arXiv preprint arXiv:2601.17226. — [arXiv:2601.17226](https://arxiv.org/abs/2601.17226) find
  16. Tohru Nagano, Gakuto Kurata, Samuel Thomas, Hong-Kwang J Kuo, Daniel Bolanos, Hyun Jung, and George Saon. 2025. Llm based text generation for improved low-resource speech recognition models. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages… find
  17. Hao Peng, Yunjia Qi, Xiaozhi Wang, Bin Xu, Lei Hou, and Juanzi Li. 2025. Verif: Verification engineering for reinforcement learning in instruction following. arXiv preprint arXiv:2506.09942.
  18. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  19. Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Y Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. 2024. Judgebench: A benchmark for evaluating llm-based judges. arXiv preprint arXiv:2410.12784. find
  20. Saranya Venkatraman, Nafis Irtiza Tripto, and Dongwon Lee. 2025. Collabstory: Multi-llm collaborative story generation and authorship analysis. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3665–3679. find
  21. Phoebe J Wang and Max Kreminski. 2024. Guiding and diversifying llm-based story generation via answer set programming. arXiv preprint arXiv:2406.00554. — [arXiv:2406.00554](https://arxiv.org/abs/2406.00554) find
  22. Zhuhao Wang, Yihua Sun, Zihan Li, Xuan Yang, Fang Chen, and Hongen Liao. 2025. Llm-rg4: Flexible and factual radiology report generation across diverse input contexts. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 8250–8258. find
  23. Chau Minh Pham, Simeng Sun, and Mohit Iyyer. 2024. Suri: Multi-constraint instruction following for long-form text generation. arXiv preprint arXiv:2406.19371. — [arXiv:2406.19371](https://arxiv.org/abs/2406.19371) find
  24. Xiaolong Wei, Bo Lu, Xingyu Zhang, Zhejun Zhao, Dongdong Shen, Long Xia, and Dawei Yin. 2025. Igniting creative writing in small language models: Llm-as-a-judge versus multi-agent refined rewards. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17171–…
  25. Shanghaoran Quan, Tianyi Tang, Bowen Yu, An Yang, Dayiheng Liu, Bofei Gao, Jianhong Tu, Yichang Zhang, Jingren Zhou, and Junyang Lin. 2024. Language models can self-lengthen to generate long texts. arXiv preprint arXiv:2410.23933.
  26. Junchao Wu, Shu Yang, Runzhe Zhan, Yulin Yuan, Lidia Sam Chao, and Derek Fai Wong. 2025a. A survey on llm-generated text detection: Necessity, methods, and future directions. Computational Linguistics, 51(1):275–338.
  27. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741.
  28. Yuhao Wu, Yushi Bai, Zhiqiang Hu, Roy Ka-Wei Lee, and Juanzi Li. 2025b. Longwriter-zero: Mastering ultra-long text generation via reinforcement learning. arXiv preprint arXiv:2506.18841. — [arXiv:2506.18841](https://arxiv.org/abs/2506.18841) find
  29. Qingyu Ren, Qianyu He, Bowei Zhang, Jie Zeng, Jiaqing Liang, Yanghua Xiao, Weikang Zhou, Zeye Sun, and Fei Yu. 2025a. Instructions are all you need: Self-supervised reinforcement learning for instruction following. arXiv preprint arXiv:2510.14420.
  30. Qingyu Ren, Jie Zeng, Qianyu He, Jiaqing Liang, Yanghua Xiao, Weikang Zhou, Zeye Sun, and Fei Yu. 2025b. Step-by-step mastery: Enhancing soft constraint following ability of large language models. arXiv preprint arXiv:2501.04945. find
  31. Jaydip Sen, Rohit Pandey, and Hetvi Waghela. 2025. Context-enhanced contrastive search for improved llm text generation. arXiv preprint arXiv:2504.21020. — [arXiv:2504.21020](https://arxiv.org/abs/2504.21020) find
  32. Yuhao Wu, Yushi Bai, Zhiqiang Hu, Juanzi Li, and Roy Ka-Wei Lee. 2025c. Superwriter: Reflection-driven long-form generation with large language models. arXiv preprint arXiv:2506.04180.
  33. Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, et al. 2025d. Writingbench: A comprehensive benchmark for generative writing. arXiv preprint arXiv:2503.05244. — [arXiv:2503.05244](https://arxiv.org/abs/2503.05244) find
  34. Lei Xu, Pierre Beckmann, Marco Valentino, and André Freitas. 2026. Adaptive llm-symbolic reasoning via dynamic logical solver composition. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1187–1208.
  35. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  36. Dong Yuan, Eti Rastogi, Gautam Naik, Sree Prasanna Rajagopal, Sagar Goyal, Fen Zhao, Bharath Chintagunta, and Jeffrey Ward. 2024a. A continued pre-trained llm approach for automatic medical note generation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Co… find
  37. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason E Weston. 2024b. Self-rewarding language models. In Forty-first International Conference on Machine Learning. find
  38. Dingling Zhang, He Zhu, Jincheng Ren, Kangqi Song, Xinran Zhou, Boyu Feng, Shudong Liu, Jiabin Luo, Weihao Xie, Zhaohui Wang, et al. 2025. How far are we from genuinely useful deep research agents? arXiv preprint arXiv:2512.01948. — [arXiv:2512.01948](https://arxiv.org/abs/2512.01948) find
  39. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470.
  40. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P Xing, et al. 2023. Lmsys-chat-1m: A large-scale real-world llm conversation dataset. arXiv preprint arXiv:2309.11998.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ren-2026-coarse-fine-benchmarking-reward-modeling/