Elamur Research⌕
☰
home / papers / 2026 / AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling

AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling

Paper · 2026 · Yongliang Miao, Yangyang Liang and Mengnan Du

Summary

Most reward models for LLM alignment reduce a prompt–response sequence to one scalar using a fixed pooling rule, usually the last token. The authors identify two problems with this design. First, a single pooling rule cannot suit every task: some evidence sits at the end of a response (a wrong final answer), while other evidence is spread throughout (harmful content). Second, the backbone's hidden states come from generative training and are poorly suited to fine-grained pairwise discrimination.

AdaJudge addresses both problems in two stages. It first refines backbone states with a few lightweight gated transformer blocks, mixed by a depth router. It then scores the response with three pooling experts, whose outputs are combined by a prompt-conditioned gate. The model is trained with a Bradley-Terry objective and needs no token-level or process supervision.

On RM-Bench and JudgeBench, AdaJudge beats strong off-the-shelf reward models and same-backbone fixed-pooling baselines. This suggests that the reward head's design, and not only data or scale, limits reward-model quality.

Key points

  • Stage I (refinement): K sequential refinement blocks process the backbone states. A depth router then combines the intermediate states into a discrimination-oriented representation.
  • Stage II (Domain-Aware Gated Mixture-of-Pooling): last-token, mean and attention pooling each produce a scalar score through an MLP. A router uses these features plus a mean-pooled prompt vector to weight the scores into the final reward.
  • Routing analysis (Qwen3-4B-based AdaJudge): gating weights differ systematically across domains. The last-token expert receives little weight overall, and attention pooling appears to take over much of its role.
  • Scope and limitations:
    • Backbones were tested only up to 8B parameters and on text-only data.
    • The extra blocks and router add parameters and inference latency.
    • The set of three experts is fixed in advance.
  • Missing results: the supplied excerpt omits the results tables, so exact benchmark scores are not recorded here.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Emory UniversityAtlanta · United States36 works · 39,779 citations🎓Chinese University of Hong Kong, ShenzhenShenzhen · China47 works · 2,596 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Yongliang Miao, Yangyang Liang and Mengnan Du. 2026. AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.440

@inproceedings{miao2026adajudge,
  title = {{AdaJudge: Adaptive Multi-Perspective Judging for Reward Modeling}},
  author = {Yongliang Miao, Yangyang Liang and Mengnan Du},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.440},
  url = {https://doi.org/10.18653/v1/2026.acl-long.440}
}

References

32 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. 2024. Rlhf workflow: From reward modeling to online rlhf. Preprint, arXiv:2405.07863. find
  2. Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219.
  3. Maha Elbayad, Jiatao Gu, Edouard Grave, and Michael Auli. 2020. Depth-adaptive transformer. In International Conference on Learning Representations (ICLR). find
  4. Sofiane Ennadir and 1 others. 2025. Pool me wisely: On the effect of pooling in transformer-based models. arXiv preprint arXiv:2510.03339. — [arXiv:2510.03339](https://arxiv.org/abs/2510.03339) find
  5. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  6. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. Preprint, arXiv:2305.20050.
  7. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. arXiv preprint arXiv:1708.02002.
  8. Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. 2024a. Skywork-reward: Bag of tricks for reward modeling in llms. Preprint, arXiv:2410.18451.
  9. Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu, Yang Liu, and Yahui Zhou. 2025. Skywork-reward-v2: Scaling preference data curation via human-ai synergy. arXiv preprint arXiv:2507.01352.
  10. Yantao Liu, Zijun Yao, Rui Min, Yixin Cao, Lei Hou, and Juanzi Li. 2024b. Rm-bench: Benchmarking reward models of language models with subtlety and style. arXiv preprint arXiv:2410.16184.
  11. Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, and Junge Zhang. 2025. Uncertainty-aware reward model: Teaching reward models to know what is unknown. Preprint, arXiv:2410.00847. — [arXiv:2410.00847](https://arxiv.org/abs/2410.00847) find
  12. Yuan Lu and 1 others. 2024a. Routing to the expert: Efficient reward-guided ensemble of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL). find
  13. Yuan Lu and 1 others. 2024b. Routing to the expert: Efficient reward-guided ensemble of large language models. In Proceedings of NAACL. find
  14. Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jian-guang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. 2023. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.09583.
  15. OpenAI. 2024. Gpt-4o system card. https://openai.com/index/gpt-4o-system-card/. Accessed: 2025-01. ↗
  16. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, and 1 others. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744.
  17. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290.
  18. David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. 2024. Mixture-of-depths: Dynamically allocating compute in transformer-based language models. arXiv preprint arXiv:2404.02258. find
  19. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992.
  20. Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. Preprint, arXiv:2009.01325. find
  21. Sijun Tan, Siyuan Zhuang, Kyle Montgomery, Willian Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca Ada Popa, and Ion Stoica. 2024. Judgebench: A benchmark for evaluating llm-based judges. find
  22. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024a. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845.
  23. Haoxiang Wang, Wei Xiong, and 1 others. 2024b. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Findings of the Association for Computational Linguistics (ACL).
  24. J. Wang and 1 others. 2024c. Pooling and attention: What are effective designs for llm-based embedding models? OpenReview. find
  25. Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Hoo-Chang Shin, Felipe Soares, Alexander Bukharin, Ellie Evans, Yi Dong, and Oleksii Kuchaiev. 2025. Helpsteer3-preference: Open human-annotated preference data across diverse tasks and languages. Preprint, arXiv:2505.11475. — [arXiv:2505.11475](https://arxiv.org/abs/2505.11475) find
  26. Xinle Wu and Yao Lu. 2025. Reward model routing in alignment. arXiv preprint arXiv:2510.02850. — [arXiv:2510.02850](https://arxiv.org/abs/2510.02850) find
  27. Xi Xiao, Yunbei Zhang, Lin Zhao, Yiyang Liu, Xiaoying Liao, Zheda Mai, Xingjian Li, Xiao Wang, Hao Xu, Jihun Hamm, and 1 others. 2025. Prompt-based adaptation in large-scale vision models: A survey. arXiv preprint arXiv:2510.13219. — [arXiv:2510.13219](https://arxiv.org/abs/2510.13219) find
  28. Zhenghao Xu, Qin Lu, Qingru Zhang, Liang Qiu, Ilgee Hong, Changlong Yu, Wenlin Yao, Yao Liu, Haoming Jiang, Lihong Li, and 1 others. 2025. Ask a strong llm judge when your reward model is uncertain. In Advances in Neural Information Processing Systems (NeurIPS). find
  29. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  30. Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. 2024. Regularizing hidden states enables learning generalizable reward model for llms. Preprint, arXiv:2406.10216. — [arXiv:2406.10216](https://arxiv.org/abs/2406.10216) find
  31. Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, Karthik Ganesan, Wei-Lin Chiang, Jian Zhang, and Jiantao Jiao. 2024. Starling-7b: Improving helpfulness and harmlessness with rlaif. In Conference on Language Modeling (COLM). find
  32. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/miao-2026-adajudge-adaptive-multi-perspective-judging/