Elamur Research⌕
☰
home / papers / 2026 / Teaching LLM to be Persuasive: Reward-Enhanced Policy Optimization for Alignment from Heterogeneous Rewards

Teaching LLM to be Persuasive: Reward-Enhanced Policy Optimization for Alignment from Heterogeneous Rewards

Paper · 2026 · Xia Zeng, Yihan Chen, Luhui Liu et al.

Summary

The paper describes deploying an LLM as a business development agent for an online travel agency (Fliggy, Alibaba). The agent contacts hotel managers to talk them into lowering prices. It must follow a multi-stage SOP, keep several related prices straight, avoid over-promising and hallucinations, and still sound like a skilled salesperson across long dialogues. The authors argue that no single reward source covers all of this. They introduce REPO, an RL post-training method that blends three kinds of signal: a reward model (RM) learned from preference data, an LLM serving as a judge (RJ) that scores against a task rubric, and reward functions (RF) built mostly from regular expressions. Judge and rule scores are summed, clipped and used to multiplicatively scale the RM reward. The RM therefore stays dominant, while new failure modes can be patched by editing a rubric or adding a rule. They used Qwen3-32B-Instruct with LoRA and a 6,632-sample preference set. REPO outperformed SFT, DPO, PPO and GRPO in expert evaluation and improved real outcomes in a production A/B test, offering a template for alignment under industrial constraints.

Key points

  • Reward combination: the auxiliary score (RJ + RF) is clipped to [−n, n] with n = 100. The RM reward is then multiplied by (1 ± E_enh/n). Positive auxiliary scores boost rewards or soften penalties, and negative ones do the opposite. REPO uses no reference model or KL penalty.
  • Training data: 6,632 preference samples from several expert sources: 252 production cases, 3,178 from language experts, 1,211 from human BDs, and 1,991 SFT samples converted into preference pairs.
  • Expert evaluation (3 experts, 30 online and 45 bad-case conversations):
    • REPO reached a dialogue rating of 4.63, compared with 3.43 for the base model and 4.30 for GRPO.
    • 66.67% of REPO's conversations contained at least one excellent response, versus 43.33% for GRPO.
    • REPO fixed 93.33% of bad cases, with 75.56% of them fixed cleanly.
  • Production A/B test (9,653 conversations, against an intent-driven system):
    • Response rate rose from 46.72% to 58.86%.
    • Task success rose from 19.32% to 25.26% (p < 0.001), which is roughly the level of human BDs.
  • Baseline issues: vanilla PPO lowered quality, and GRPO left 28.88% of bad cases unresolved.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Xia Zeng, Yihan Chen, Luhui Liu, Chao Luo, Ye Chen and Zhuangzhuoran. 2026. Teaching LLM to be Persuasive: Reward-Enhanced Policy Optimization for Alignment from Heterogeneous Rewards. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-industry.34

@inproceedings{zeng2026teaching,
  title = {{Teaching LLM to be Persuasive: Reward-Enhanced Policy Optimization for Alignment from Heterogeneous Rewards}},
  author = {Xia Zeng, Yihan Chen, Luhui Liu, Chao Luo, Ye Chen and Zhuangzhuoran},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 6: Industry Track)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-industry.34},
  url = {https://doi.org/10.18653/v1/2026.acl-industry.34}
}

References

10 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yasaman Jafari, Dheeraj Mekala, Rose Yu, and Taylor Berg-Kirkpatrick. 2024. MORL-prompt: An empirical analysis of multi-objective reinforcement learning for discrete prompt optimization. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9878–9889, Miami, Florida, USA. A… find
  2. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  3. Hao Peng, Yunjia Qi, Xiaozhi Wang, Zijun Yao, Bin Xu, Lei Hou, and Juanzi Li. 2025. Agentic reward modeling: Integrating human preferences with verifiable correctness signals for reliable reward systems.
  4. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xu… find
  5. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model.
  6. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models.
  7. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jin… — Qwen3 Technical Report
  8. Zihao Yi, Jiarui Ouyang, Zhe Xu, Yuwen Liu, Tianhao Liao, Haohao Luo, and Ying Shen. 2025. A survey on recent advances in llm-based multi-turn dialogue systems.
  9. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2025. A survey of l…
  10. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zeng-2026-teaching-llm-be-persuasive-reward/