Elamur Research⌕
☰
home / papers / 2026 / Triviality Corrected Endogenous Reward

Triviality Corrected Endogenous Reward

Paper · 2026 · Xinda Wang, Zhengxu Hou, Yangshijie Zhang et al.

Summary

Reinforcement fine-tuning for open-ended writing lacks verifiable rewards. It usually depends on costly proprietary LLM judges or on judge models trained with annotated preference data. The authors ask whether confidence-based endogenous rewards, which work for unsupervised RL in math, can serve writing instead. They find that rewarding a fine-tuned model's own log-probabilities does beat SFT, but it also causes what they call Triviality Bias. The policy drifts toward safe, high-probability, low-entropy text and loses diversity and substance.

Their fix, TCER, keeps the confidence term and adds a correction. This term measures how much more likely a token is under the in-domain specialist (the SFT model) than under its generalist base model. A weight then shrinks this correction for tokens the specialist already finds highly probable. Training uses GRPO augmented with ground-truth reference samples. TCER improves results across several writing benchmarks and model architectures and also transfers to mathematical reasoning, all without external supervision. This makes judge-free RL a practical option for generation tasks.

Key points

  • The reward is log π_s + k·(1 − π_s)^λ·log(π_s/π_b). Here π_s is the SFT specialist, π_b is the same model before fine-tuning, and ε-smoothing is applied for numerical stability.
  • Optimization uses Reference Augmented GRPO, which adds high-quality reference outputs to each rollout group.
  • In a sentence-level analysis, Gemini 2.5 Pro, Claude-Opus4 and GPT-4o judged which sentences were of exceptional quality. TCER gave these sentences higher rewards than EndoR and scored ordinary sentences about the same.
  • Ablations used Qwen2.5-7B-Instruct with DeepWriting data. Dropping the confidence term caused the largest collapse (e.g., 86.3 → 76.2 on the first benchmark). Removing the gating weight, the full correction, or reference augmentation also lowered scores.
  • The authors list three limitations: an inherent ceiling for self-rewarding methods, reliance on LLM-judge benchmarks for evaluation, and limited interpretability at the token level.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Xinda Wang, Zhengxu Hou, Yangshijie Zhang, Yanbingren, Jialin Liu, ChenZhuo Zhao, Zhibo Yang, Bin-Bin Yang and Feng Xiao. 2026. Triviality Corrected Endogenous Reward. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.883

@inproceedings{wang2026triviality,
  title = {{Triviality Corrected Endogenous Reward}},
  author = {Xinda Wang, Zhengxu Hou, Yangshijie Zhang, Yanbingren, Jialin Liu, ChenZhuo Zhao, Zhibo Yang, Bin-Bin Yang and Feng Xiao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.883},
  url = {https://doi.org/10.18653/v1/2026.acl-long.883}
}

References

39 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, and 1 others. 2025. The entropy mechanism of reinforcement learning for reasoni… find
  2. Yuanning Feng, Sinan Wang, Zhengxiang Cheng, Yao Wan, and Dongping Chen. 2025. Are we on the right way to assessing llm-as-a-judge? arXiv preprint arXiv:2512.16041. — [arXiv:2512.16041](https://arxiv.org/abs/2512.16041) find
  3. Sian Gooding, Lucia Lopez-Rivilla, and Edward Grefenstette. 2025. Writing as a testbed for open ended agents. arXiv preprint arXiv:2503.19711. — [arXiv:2503.19711](https://arxiv.org/abs/2503.19711) find
  4. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, and 1 others. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting…
  5. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  6. Hui Huang, Xingyuan Bu, Hongli Zhou, Yingqi Qu, Jing Liu, Muyun Yang, Bing Xu, and Tiejun Zhao. 2025a. An empirical study of llm-as-a-judge for llm evaluation: Fine-tuned judge model is not a general substitute for gpt-4. In Findings of the Association for Computational Linguistics: ACL 2025, pages… find
  7. Zeyu Huang, Tianhao Cheng, Zihan Qiu, Zili Wang, Yinghui Xu, Edoardo M Ponti, and Ivan Titov. 2025b. Blending supervised and reinforcement fine-tuning with prefix sampling. arXiv preprint arXiv:2507.01679. — [arXiv:2507.01679](https://arxiv.org/abs/2507.01679) find
  8. Hangzhan Jin, Sicheng Lv, Sifan Wu, and Mohammad Hamdaqa. 2025. Rl is neither a panacea nor a mirage: Understanding supervised vs. reinforcement learning fine-tuning for llms. arXiv preprint arXiv:2508.16546. — [arXiv:2508.16546](https://arxiv.org/abs/2508.16546) find
  9. Zae Myung Kim, Anand Ramachandran, Farideh Tavazoee, Joo-Kyung Kim, Oleg Rokhlenko, and Dongyeop Kang. 2025. Align to structure: Aligning large language models with structural information. arXiv preprint arXiv:2504.03622. — [arXiv:2504.03622](https://arxiv.org/abs/2504.03622) find
  10. Komal Kumar, Tajamul Ashraf, Omkar Thawakar, Rao Muhammad Anwer, Hisham Cholakkal, Mubarak Shah, Ming-Hsuan Yang, Phillip HS Torr, Fahad Shahbaz Khan, and Salman Khan. 2025. Llm post-training: A deep dive into reasoning large language models. arXiv preprint arXiv:2502.21321. — [arXiv:2502.21321](https://arxiv.org/abs/2502.21321) find
  11. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, and 1 others. 2022. Solving quantitative reasoning problems with language models. Advances in neural information processing systems, 35:3843–3857…
  12. Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, and 1 others. 2024. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13(9):… find
  13. Pengyi Li, Matvey Skripkin, Alexander Zubrey, Andrey Kuznetsov, and Ivan Oseledets. 2025a. Confidence is all you need: Few-shot rl fine-tuning of language models. arXiv preprint arXiv:2506.06395. — [arXiv:2506.06395](https://arxiv.org/abs/2506.06395) find
  14. Tianjian Li, Yiming Zhang, Ping Yu, Swarnadeep Saha, Daniel Khashabi, Jason Weston, Jack Lanchantin, and Tianlu Wang. 2025b. Jointly reinforcing diversity and quality in language model generations. arXiv preprint arXiv:2509.02534. — [arXiv:2509.02534](https://arxiv.org/abs/2509.02534) find
  15. Yi-Chen Li, Tian Xu, Yang Yu, Xuqin Zhang, Xiong-Hui Chen, Zhongxiang Ling, Ningjing Chao, Lei Yuan, and Zhi-Hua Zhou. 2025c. Generalist reward models: Found inside large language models. arXiv preprint arXiv:2506.23235. — [arXiv:2506.23235](https://arxiv.org/abs/2506.23235) find
  16. Xun Lu. 2025. Writing-zero: Bridge the gap between non-verifiable problems and verifiable rewards. arXiv preprint arXiv:2506.00103. reasoning in open language models. arXiv preprint arXiv:2402.03300. — [arXiv:2506.00103](https://arxiv.org/abs/2506.00103) find
  17. Hongze Tan, Jianfei Pan, Jinghao Lin, Tao Chen, Zhihang Zheng, Zhihao Tang, and Haihua Yang. 2025. Gtpo and grpo-s: Token and sequence-level reward shaping with policy entropy. arXiv preprint arXiv:2508.04349. — [arXiv:2508.04349](https://arxiv.org/abs/2508.04349) find
  18. Guiyao Tie, Zeli Zhao, Dingjie Song, Fuyang Wei, Rong Zhou, Yurou Dai, Wen Yin, Zhejian Yang, Jiangyue Yan, Yao Su, and 1 others. 2025. A survey on post-training of large language models. arXiv e-prints, pages arXiv–2503. find
  19. Haozhe Wang, Haoran Que, Qixin Xu, Minghao Liu, Wangchunshu Zhou, Jiazhan Feng, Wanjun Zhong, Wei Ye, Tong Yang, Wenhao Huang, and 1 others. 2025a. Reverse-engineered reasoning for open-ended generation. arXiv preprint arXiv:2509.06160. — [arXiv:2509.06160](https://arxiv.org/abs/2509.06160) find
  20. Shenzhi Wang, Le Yu, Chang Gao, Chujie Zheng, Shixuan Liu, Rui Lu, Kai Dang, Xionghui Chen, Jianxin Yang, Zhenru Zhang, and 1 others. 2025b. Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning. arXiv preprint arXiv:2506.01939.
  21. Tiannan Wang, Jiamin Chen, Qingrui Jia, Shuai Wang, Ruoyu Fang, Huilin Wang, Zhaowei Gao, Chunzhao Xie, Chuou Xu, Jihong Dai, and 1 others. 2024. Weaver: Foundation models for creative writing. arXiv preprint arXiv:2401.17268.
  22. Xinda Wang, Zhengxu Hou, Yangshijie Zhang, Bingren Yan, Zhibo Yang, Xingsheng Zhang, Luxi Xing, Qiang Zhou, and Chen Zhang. 2025c. Evolvr: Self-evolving pairwise reasoning for story evaluation to enhance generation. arXiv preprint arXiv:2508.06046.
  23. Jingwei Ni, Yu Fan, Vilém Zouhar, Donya Rooein, Alexander Hoyle, Mrinmaya Sachan, Markus Leippold, Dirk Hovy, and Elliott Ash. 2025. Can reasoning help large language models capture human annotator disagreement. Preprint. find
  24. Xiaolong Wei, Bo Lu, Xingyu Zhang, Zhejun Zhao, Dongdong Shen, Long Xia, and Dawei Yin. 2025. Igniting creative writing in small language models: Llm-as-a-judge versus multi-agent refined rewards. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 17171–…
  25. Mihir Prabhudesai, Lili Chen, Alex Ippoliti, Katerina Fragkiadaki, Hao Liu, and Deepak Pathak. 2025. Maximizing confidence alone improves reasoning. arXiv preprint arXiv:2505.22660. — [arXiv:2505.22660](https://arxiv.org/abs/2505.22660) find
  26. Peter West and Christopher Potts. 2025. Base models beat aligned models at randomness and creativity. arXiv preprint arXiv:2505.00047. find
  27. Haoran Que, Feiyu Duan, Liqun He, Yutao Mou, Wangchunshu Zhou, Jiaheng Liu, Wenge Rong, Zekun Moore Wang, Jian Yang, Ge Zhang, and 1 others. 2024. Hellobench: Evaluating long text generation capabilities of large language models. arXiv preprint arXiv:2409.16191. — [arXiv:2409.16191](https://arxiv.org/abs/2409.16191) find
  28. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical Mian Wu, Gavin Zhang, Sewon Min, Sergey Levine, and Aviral Kumar. 2025a. Rlac: Reinforcement learning with advers… — [arXiv:2511.01758](https://arxiv.org/abs/2511.01758) find
  29. Yongliang Wu, Yizhou Zhou, Zhou Ziheng, Yingzhe Peng, Xinyu Ye, Xinting Hu, Wenbo Zhu, Lu Qi, Ming-Hsuan Yang, and Xu Yang. 2025b. On the generalization of sft: A reinforcement learning perspective with reward rectification. arXiv preprint arXiv:2508.05629. — [arXiv:2508.05629](https://arxiv.org/abs/2508.05629) find
  30. Yuhao Wu, Yushi Bai, Zhiqiang Hu, Roy Ka-Wei Lee, and Juanzi Li. 2025c. Longwriter-zero: Mastering ultra-long text generation via reinforcement learning. arXiv preprint arXiv:2506.18841. — [arXiv:2506.18841](https://arxiv.org/abs/2506.18841) find
  31. Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, Shaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, and 1 others. 2025d. Writingbench: A comprehensive benchmark for generative writing. arXiv preprint arXiv:2503.05244. — [arXiv:2503.05244](https://arxiv.org/abs/2503.05244) find
  32. Jianhao Yan, Yafu Li, Zican Hu, Zhi Wang, Ganqu Cui, Xiaoye Qu, Yu Cheng, and Yue Zhang. 2025. Learning to reason under off-policy guidance. arXiv preprint arXiv:2504.14945. find
  33. Shuangshuang Ying, Yunwen Li, Xingwei Qu, Xin Li, Sheng Jin, Minghao Liu, Zhoufutu Wen, Xeron Du, Tianyu Zheng, Yichi Zhang, and 1 others. 2025. Beyond correctness: Evaluating subjective writing preferences across cultures. arXiv preprint arXiv:2510.14616. — [arXiv:2510.14616](https://arxiv.org/abs/2510.14616) find
  34. Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. 2025. Does reinforcement learning really incentivize reasoning capacity in llms beyond the base model? arXiv preprint arXiv:2504.13837.
  35. Qingyang Zhang, Haitao Wu, Changqing Zhang, Peilin Zhao, and Yatao Bian. 2025a. Right question is already half the answer: Fully unsupervised llm reasoning incentivization. arXiv preprint arXiv:2504.05812. find
  36. Yiming Zhang, Harshita Diddee, Susan Holm, Hanchen Liu, Xinyue Liu, Vinay Samuel, Barry Wang, and Daphne Ippolito. 2025b. Noveltybench: Evaluating language models for humanlike diversity. arXiv preprint arXiv:2504.05228. — [arXiv:2504.05228](https://arxiv.org/abs/2504.05228) find
  37. Xuandong Zhao, Zhewei Kang, Aosong Feng, Sergey Levine, and Dawn Song. 2025. Learning to reason without external rewards. arXiv preprint arXiv:2505.19590. find
  38. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.
  39. Yuxin Zuo, Kaiyan Zhang, Li Sheng, Shang Qu, Ganqu Cui, Xuekai Zhu, Haozhan Li, Yuchen Zhang, Xinwei Long, Ermo Hua, and 1 others. 2025. Ttrl: Test-time reinforcement learning. arXiv preprint arXiv:2504.16084. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-triviality-corrected-endogenous-reward/