Elamur Research⌕
☰
home / papers / 2026 / Preference Optimization for Review Question Generation Improves Writing Quality

Preference Optimization for Review Question Generation Improves Writing Quality

Paper · 2026 · Karun Sharma, Vidushee Vats, Shengzhi LI et al.

Summary

LLM-drafted peer-review questions are often shallow. They tend to echo keywords or the introduction instead of engaging critically with the whole paper. To address this, the authors built a corpus of reviewer questions from ICLR 2024 reviews on OpenReview, using Gemini 2.0 for extraction and olmOCR for paper text. They then ran an expert annotation study that scored question–paper pairs on three binary criteria: Effort, Evidence and Grounding. In this study, human-written questions clearly beat those from frontier models. The annotations were used to train IntelliReward, a reward model that adds trainable multi-head transformer layers on top of a frozen autoregressive LLM. IntelliReward agrees with experts better than SFT-tuned API judges. It then served as the reward signal for DAPO reinforcement learning, producing IntelliAsk. In human evaluation, IntelliAsk's questions are more grounded and effortful than those of SFT and frontier baselines, and they depend less on first-page content. The gains also carry over to general reasoning and writing benchmarks. This suggests that learning to ask good questions reflects broader model ability.

Key points

  • Dataset: 15.5k filtered reviewer questions from 5,841 papers, split into 13.2k for training and 2.3k for testing.
  • Preference study: 572 annotated question–paper pairs drawn from 300 ICLR 2025 submissions.
    • On the 0–3 rubric, human questions averaged 0.78 points above the best LLM and 1.53 above the weakest.
    • The weakest LLM was Qwen2.5-32B; 55% of the words in its questions came from the paper's first page.
  • Methods: IntelliReward, a frozen LLM with multi-head transformer heads, serves as the reward model. IntelliAsk is trained with DAPO at 7B and 32B scales.
  • Transfer results: IntelliAsk-32B outperforms Qwen3-32B on MuSR (68.3 vs 64.7 accuracy) and WritingBench (8.31 vs 8.07). It also improves on Arena Hard.
  • Release: The authors have publicly shared their code, the cleaned set of reviewer questions, the expert-labeled data, and both trained models (IntelliAsk and IntelliReward). The data is released under CC BY 4.0.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Question answering · 441 works

Cite

Cite as Karun Sharma, Vidushee Vats, Shengzhi LI, Yuxiang Wang, Zhongtian Sun and Prayag Tiwari. 2026. Preference Optimization for Review Question Generation Improves Writing Quality. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1256

@inproceedings{sharma2026preference,
  title = {{Preference Optimization for Review Question Generation Improves Writing Quality}},
  author = {Karun Sharma, Vidushee Vats, Shengzhi LI, Yuxiang Wang, Zhongtian Sun and Prayag Tiwari},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1256},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1256}
}

References

27 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tim Baumgärtner, Ted Briscoe, and Iryna Gurevych. 2025. PeerQA: A scientific question answering dataset from peer reviews. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers… find
  2. Eric Chamoun, Michael Schlichtkrull, and Andreas Vlachos. 2024. Automated focused feedback generation for scientific writing assistance. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9742–9763, Bangkok, Thailand. Association for Computational Linguistics. find
  3. Maitreya Prafulla Chitale, Ketaki Mangesh Shetye, Harshit Gupta, Manav Chaudhary, and Vasudeva Varma. 2025. Autorev: Automatic peer review system for academic research papers. arXiv preprint arXiv: 2505.14376. — [arXiv:2505.14376](https://arxiv.org/abs/2505.14376) find
  4. Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. 2025. SFT memorizes, RL generalizes: A comparative study of foundation model post-training. In Forty-second International Conference on Machine Learning.
  5. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL.
  6. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. Marg: Multi-agent review generation for scientific papers. Preprint, arXiv:2401.04259. find
  7. Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-seeking questions and answers anchored in research papers. arXiv preprint arXiv: 2105.03011.
  8. D. Davis. 2021. Cvpr 2021 training materials: Reference slides. find
  9. Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, Haoran Ranran Zhang, Vipul Gupta, Yinghui Li, Tao Li, Fei Wang, Qin Liu, Tianlin Liu, Pengzhi Gao, Congying Xia, and 21 others. 2024. LLMs assist NLP resea…
  10. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. Preprint, arXiv:1903.00161.
  11. ICLR. 2025. Leveraging llm feedback to enhance review quality. find
  12. Maximilian Idahl and Zahra Ahmadi. 2025. OpenReviewer: A specialized large language model for generating critical scientific paper reviews. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Syst… find
  13. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. Preprint, arXiv:2406.11939.
  14. Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, Daniel A. McFarland, and James Y. Zou. 2024. Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews. In ICML. find
  15. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser-assis…
  16. NeurIPS. 2023. Reviewer guidelines. find
  17. Kun-Peng Ning, Shuo Yang, Yuyang Liu, Jia-Yu Yao, Zhenhui Liu, Yonghong Tian, Yibing Song, and Li Yuan. 2025. PiCO: Peer review in LLMs based on consistency optimization. In The Thirteenth International Conference on Learning Representations. find
  18. Jake Poznanski, Jon Borchardt, Jason Dunkelberger, Regan Huff, Daniel Lin, Aman Rangapur, Christopher Wilhelm, Kyle Lo, and Luca Soldaini. 2025. olmOCR: Unlocking Trillions of Tokens in PDFs with Vision Language Models. Preprint, arXiv:2502.18443.
  19. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. Gpqa: A graduate-level google-proof qa benchmark. Preprint, arXiv:2311.12022.
  20. Shruti Singh, Nandan Sarkar, and Arman Cohan. 2024. SciDQA: A deep reading comprehension dataset over scientific papers. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20908–20923, Miami, Florida, USA. Association for Computational Linguistics. find
  21. Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. 2024. Musr: Testing the limits of chain-of-thought with multistep soft reasoning. Preprint, arXiv:2310.16049. find
  22. Anirudh Sundar, Jin Xu, William Gay, Christopher Gordon Richardson, and Larry Heck. 2024. cPAPERS: A dataset of situated and multimodal interactive conversations in scientific papers. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  23. Cheng Tan, Dongxin Lyu, Siyuan Li, Zhangyang Gao, Jingxuan Wei, Siqi Ma, Zicheng Liu, and Stan Z. Li. 2025. Peer review as a multi-turn and long-context dialogue with role-based interactions: Benchmarking large language models. find
  24. Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, and James Zou. 2025. Can llm feedback enhance review quality? a randomized study of 20k reviews at iclr 2025. arXiv preprint arXiv: 2504.09737. find
  25. Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and Fei Huang. 2025. Writingbench: A comprehensive benchmark for generative writing. Preprint, arXiv:2503.05244. — [arXiv:2503.05244](https://arxiv.org/abs/2503.05244) find
  26. Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, and 16 others. 2025. Dapo: An open-source llm reinforcement learning s…
  27. Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. 2025. DeepReview: Improving LLM-based paper review with human-like deep thinking process. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29330–29355, Vienna, Austria. Associ… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sharma-2026-preference-optimization-review-question-generation/