Elamur Research⌕
☰
home / papers / 2026 / Reward Modeling for Scientific Writing Evaluation

Reward Modeling for Scientific Writing Evaluation

Paper · 2026 · Furkan Şahinuç, Subhabrata Dutta and Iryna Gurevych

Summary

The paper tackles the automatic evaluation of scientific writing, where off-the-shelf LLM judges and general-purpose reward models tend to misapply task-specific criteria and rubrics and to make assumptions beyond them. The authors build open-source, low-cost reward models, SciRM and SciRM-Ref, that receive an explicit evaluation "constitution" (criteria, scoring rubric, worked examples) as input during both training and inference. They train with GRPO in two stages. The first stage instills scientific evaluation preferences, and the second strengthens reasoning so the model can reinterpret the given rules and correct itself. Training is joint across several tasks and scores each artifact on multiple aspects rather than with one aggregate number. On four tasks, the models outperform baselines, including two tasks never seen in training. SciRM-Ref leads on average, with its edge coming mainly from the reasoning-heavy novelty task. This matters because a single evaluator can be reused for new scientific writing tasks without retraining, which helps in data-scarce settings.

Key points

  • Models and training: SciRM is trained with stage 1 only, and SciRM-Ref with both GRPO stages. Both are 7B-scale models, a size the authors chose because of GRPO's compute cost.
  • Seen and unseen tasks: Training covers related work sections and peer-review utility. Testing adds two tasks absent from training: novelty assessment alignment and instruction-based paper revisions.
  • Reasoning ablation: Disabling reasoning at evaluation time lowers average performance by 0.07 for SciRM and 0.08 for SciRM-Ref. Novelty evaluation suffers most, dropping by 0.21 and 0.25 respectively.
  • Trace analysis: Manual inspection found the models' reasoning fully consistent with their final scores. Baselines, by contrast, often invented criteria or overlooked edited text.
  • Resources and limitations: Code is released at GitHub UKPLab/acl2026-expert-rm and data on TUdatalib. Because evaluation datasets are scarce, most scoring rubrics are binary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany109 works · 35,920 citations🔬Hessian Center for Artificial IntelligenceDarmstadt · Germany12 works · 529 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksHistory and Philosophy of Science · 53 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Furkan Şahinuç, Subhabrata Dutta and Iryna Gurevych. 2026. Reward Modeling for Scientific Writing Evaluation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.567

@inproceedings{sahinuc2026reward,
  title = {{Reward Modeling for Scientific Writing Evaluation}},
  author = {Furkan Şahinuç, Subhabrata Dutta and Iryna Gurevych},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.567},
  url = {https://doi.org/10.18653/v1/2026.acl-long.567}
}

References

47 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Osama Mohammed Afzal, Preslav Nakov, Tom Hope, and Iryna Gurevych. 2026. Beyond “not novel enough”: Enriching scholarly critique with LLM-assisted feedback. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2… find
  2. Andrei Alexandru, Antonia Calvi, Henry Broomfield, Jackson Golden, Kyle Dai, Mathias Leys, Maurice Burger, Max Bartolo, Roman Engeler, Sashank Pisu-pati, Toby Drane, and Young Sun Park. 2025. Atla selene mini: A general purpose evaluation model. Preprint, arXiv:2501.17195. — [arXiv:2501.17195](https://arxiv.org/abs/2501.17195) find
  3. Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D. Chang, and Prithviraj Ammanabrolu. 2024. Critique-out-loud reward models. Preprint, arXiv:2408.11791. find
  4. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, and 32 others. 20… — Constitutional AI: Harmlessness from AI Feedback
  5. Tuhin Chakrabarty, Philippe Laban, and Chien-Sheng Wu. 2025. AI-slop to AI-polish? aligning language models through edit-based writing rewards and test-time computation. In Second Conference on Language Modeling. find
  6. Nuo Chen, Zhiyuan Hu, Qingyun Zou, Jiaying Wu, Qian Wang, Bryan Hooi, and Bingsheng He. 2025. Judgelrm: Large reasoning models as a judge. Preprint, arXiv:2504.00050. — [arXiv:2504.00050](https://arxiv.org/abs/2504.00050) find
  7. Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian, Yu Wang, Hongru WANG, Yu Zhang, Denghui Zhang, Tong Zhang, Hanghang Tong, and Heng Ji. 2026. RM-r1: Reward modeling as reasoning. In Fourteenth International Conference on Learning Representations.
  8. Furkan Şahinuç, Subhabrata Dutta, and Iryna Gurevych. 2025. Expert preference-based evaluation of automated related work generation. Preprint, arXiv:2508.07955. — [arXiv:2508.07955](https://arxiv.org/abs/2508.07955) find
  9. Unsloth team Daniel Han, Michael Han. 2023. Unsloth. find
  10. Flow AI. 2024. Flow judge: An open small language model for llm system evaluations. https://flow-ai.com/blog/flow-judge. Technical report, accessed: 2025-12-27. ↗
  11. Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios Nikolas Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2025. How to evaluate reward models for RLHF. In Thirteenth International Conference on Learning Representations. find
  12. Mingqi Gao, Xinyu Hu, Xunjian Yin, Jie Ruan, Xiao Pu, and Xiaojun Wan. 2025. LLM-based NLG evaluation: Current status and challenges. Computational Linguistics, 51:661–687.
  13. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and 1 others. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  14. Daya Guo, Dejian Yang, Haowei Zhang, and 1 others. 2025a. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645:633– 638. find
  15. Jiaxin Guo, Zewen Chi, Li Dong, Qingxiu Dong, Xun Wu, Shaohan Huang, and Furu Wei. 2025b. Reward reasoning models. In Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  16. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  17. IBM. 2025. Granite-3.3-8b-instruct. https://huggingface.co/ibm-granite/granite-3. 3-8b-instruct. Accessed: 2025-12-11. ↗
  18. Ai Jian, Jingqing Ruan, Xing Ma, Dailin Li, Qian-Lin Zhou, Ke Zeng, and Xunliang Cai. 2025. Patarm: Bridging pairwise and pointwise signals via preference-aware task-adaptive reward modeling. Preprint, arXiv:2510.24235. find
  19. Leane Jourdan, Nicolas Hernandez, Florian Boudin, and Richard Dufour. 2025. Identifying reliable evaluation metrics for scientific text revision. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6731–6756, Vienna, Austria. Asso… find
  20. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Metho…
  21. Seungone Kim, Ian Wu, Jinu Lee, Xiang Yue, Seongyun Lee, Mingyeong Moon, Kiril Gashteovski, Carolin Lawrence, Julia Hockenmaier, Graham Neubig, and Sean Welleck. 2025. Scaling evaluation-time compute with reasoning models as process evaluators. Preprint, arXiv:2503.19877. — [arXiv:2503.19877](https://arxiv.org/abs/2503.19877) find
  22. Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, and Jiaya Jia. 2024. Step-dpo: Step-wise preference optimization for long-chain reasoning of llms. Preprint, arXiv:2406.18629.
  23. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2025. RewardBench: Evaluating reward models for language modeling. In Findings of the Association for Computational… find
  24. Xiangci Li and Jessica Ouyang. 2024. Related work and citation text generation: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 13846–13864, Miami, Florida, USA. Association for Computational Linguistics. find
  25. Zhen Li, Xiaohan Xu, Tao Shen, Can Xu, Jia-Chen Gu, Yuxuan Lai, Chongyang Tao, and Shuai Ma. 2024. Leveraging large language models for NLG evaluation: Advances and challenges. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 16028–16045, Miami, Florid…
  26. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, Daniel A. McFarland, and James Zou. 2024. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 1(8):A… find
  27. Xiaobo Liang, Haoke Zhang, Juntao Li, Kehai Chen, Qiaoming Zhu, and Min Zhang. 2025. Generative reward modeling via synthetic criteria preference learning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26755–26769, Vienna, A… find
  28. Tianci Liu, Ran Xu, Tony Yu, Ilgee Hong, Carl Yang, Tuo Zhao, and Haoyu Wang. 2025a. Openrubrics: Towards scalable synthetic rubric generation for reward modeling and llm alignment. Preprint, arXiv:2510.07743.
  29. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
  30. Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. 2025b. Inference-time scaling for generalist reward modeling. Preprint, arXiv:2504.02495.
  31. Qiyao Ma, Yunsheng Shi, Hongtao Tian, Chao Wang, Weiming Chang, and Ting Yao. 2025. From faithfulness to correctness: Generative reward models that think critically. Preprint, arXiv:2509.25409. — [arXiv:2509.25409](https://arxiv.org/abs/2509.25409) find
  32. Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, and Nathan Lambert. 2026. Rewardeval: Advancing reward model evaluation. In Fourteenth International Conference on Learning Representations. find
  33. OpenAI. 2025. Gpt-5.1: A smarter, more conversational chatgpt. https://openai.com/index/gpt-5-1/. Accessed: 2025-12-11. ↗
  34. Sukannya Purkayastha, Zhuang Li, Anne Lauscher, Lizhen Qu, and Iryna Gurevych. 2025. LazyReview: A dataset for uncovering lazy thinking in NLP peer reviews. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3280–3308, Vienna, Au… find
  35. Abdelrahman Sadallah, Tim Baumgärtner, Iryna Gurevych, and Ted Briscoe. 2025. The good, the bad and the constructive: Automatically measuring peer review’s utility for authors. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 28979–29009, Suzhou, China… find
  36. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  37. Tu Shiwen, Zhao Liang, Chris Yuhao Liu, Liang Zeng, and Yang Liu. 2024. Skywork critic model series. https://huggingface.co/Skywork. ↗
  38. Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. 2022. Defining and characterizing reward gaming. In Advances in Neural Information Processing Systems, volume 35, pages 9460– 9471. Curran Associates, Inc. find
  39. Annalisa Szymanski, Noah Ziems, Heather A. Eicher-Miller, Toby Jia-Jun Li, Meng Jiang, and Ronald A. Metoyer. 2025. Limitations of the llm-as-a-judge approach for evaluating llm outputs in expert knowledge tasks. In Proceedings of the 30th International Conference on Intelligent User Interfaces, IUI… find
  40. Zecheng Tang, Baibei Ji, Quantong Qiu, Haitian Wang, Xiaobo Liang, Juntao Li, and Min Zhang. 2025. Longrm: Revealing and unlocking the context boundary of reward modeling. Preprint, arXiv:2510.06915. — [arXiv:2510.06915](https://arxiv.org/abs/2510.06915) find
  41. Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi-Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. 2024. Self-taught evaluators. Preprint, arXiv:2408.02666.
  42. Xumeng Wen, Zihan Liu, Shun Zheng, Shengyu Ye, Zhirong Wu, Yang Wang, Zhijian Xu, Xiao Liang, Junjie Li, Ziming Miao, Jiang Bian, and Mao Yang. 2026. Reinforcement learning with verifiable rewards implicitly incentivizes correct reasoning in base LLMs. In Fourteenth International Conference on Learn… find
  43. Chenxi Whitehouse, Tianlu Wang, Ping Yu, Xian Li, Jason E Weston, Ilia Kulikov, and Swarnadeep Saha. 2026. J1: Incentivizing thinking in LLM-as-a-judge via reinforcement learning. In Fourteenth International Conference on Learning Representations. find
  44. RevUtil Actionability Grounding Verifiability Ext. Verifiability Helpfulness Total An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Ta… — Qwen3 Technical Report
  45. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025b. Qwen2.5 technical report. Preprint, arXiv:24… find
  46. Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. 2024. Regularizing hidden states enables learning generalizable reward model for llms. In Advances in Neural Information Processing Systems, volume 37, pages 62279–62309. Curran Associates, Inc. find
  47. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 3…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sahinuc-2026-reward-modeling-scientific-writing-evaluation/