Elamur Research⌕
☰
home / papers / 2026 / Self-Guided Alignment: Adaptive Preference Sensing for Multi-Objective Generation

Self-Guided Alignment: Adaptive Preference Sensing for Multi-Objective Generation

Paper · 2026 · Ning Wang, Zhanyang Liu, Taotao Zhou et al.

Summary

Prompt-conditioned methods for multi-objective LLM alignment, such as RiC, CPO, and SteerLM, have a mismatch between training and inference. They are trained with reference preference scores from reward models or annotators, but at inference users must supply numeric targets themselves (e.g., how helpful versus how harmless). Users find these numbers hard to choose, and their guesses may not match the training distribution. Self-Guided Alignment (SGA) instead lets the model infer the user's implicit preference profile from the raw prompt. A shared backbone feeds two heads: a regression head that estimates a preference score vector, and an LM head that generates a response conditioned on that vector. The objective models the joint distribution of response and scores. This gives a cross-entropy term plus a λ-weighted squared-error term, which comes from a Gaussian assumption on the scores. Both an SFT and a DPO variant are provided. Experiments on Llama and Qwen models with HelpSteer2, UltraFeedback, and HH-RLHF show that SGA frequently beats state-of-the-art baselines on multi-objective trade-offs. It adds little inference cost and requires no manual preference input.

Key points

  • Two variants, SGA_SFT and SGA_DPO, are tested on models including Llama3.2-3B and Qwen3-4B.
  • On HelpSteer2, SGA's predicted scores are often within 0.2 of the coarse annotations, which take values {0, 0.25, 0.5, 0.75, 1} over helpfulness, correctness, coherence, complexity, and verbosity. When a prompt lacks context, the model defaults to uniformly high scores.
  • On Llama3.2-3B with HelpSteer2, adaptive sensing reaches a 58.3% win rate. This beats fixed steering at 1.0 (54.8%) and ground-truth steering (53.8%).
  • With fixed-value steering, win rate increases with the score, from 47.1% at 0.2 to 54.8% at 1.0. This indicates that the LM head responds to the conditioning signal.
  • Stated limitations: RL-based optimizers such as PPO and GRPO and larger models were not tested. Code is available at github.com/python-yyds/SGA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Jiangnan University · Silicon Austria Labs.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

Cite

Cite as Ning Wang, Zhanyang Liu, Taotao Zhou, Xinrui Zhang, Zongru Shao and Haojie Zhou. 2026. Self-Guided Alignment: Adaptive Preference Sensing for Multi-Objective Generation. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2184

@inproceedings{wang2026self,
  title = {{Self-Guided Alignment: Adaptive Preference Sensing for Multi-Objective Generation}},
  author = {Ning Wang, Zhanyang Liu, Taotao Zhou, Xinrui Zhang, Zongru Shao and Haojie Zhou},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2184},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2184}
}

References

40 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, and 12 others. 202… find
  2. Ruizhe Chen, Wenhao Chai, Zhifei Yang, Xiaotian Zhang, Ziyang Wang, Tony Quek, Joey Tianyi Zhou, Soujanya Poria, and Zuozhu Liu. 2025. DiffPO: Diffusion-styled preference optimization for inference time alignment of large language models. In Proceedings of the 63rd Annual Meeting of the Association… find
  3. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. Preprint, arXiv:2310.01377. find
  4. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. Deepseek-r1: Incentivizing reasoning capability in ll…
  5. Natural Language Processing, pages 10122–10140, Miami, Florida, USA. Association for Computational Linguistics. find
  6. Yi Dong, Zhilin Wang, Makesh Sreedhar, Xianchao Wu, and Oleksii Kuchaiev. 2023. SteerLM: Attribute conditioned SFT as an (user-steerable) alternative to RLHF. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11275– 11288, Singapore. Association for Computational Lingui… find
  7. Long-Fei Li, Yu-Yang Qian, Peng Zhao, and Zhi-Hua Zhou. 2025b. Provably efficient online rlhf with one-pass reward modeling. Preprint, arXiv:2502.07193. — [arXiv:2502.07193](https://arxiv.org/abs/2502.07193) find
  8. Tingchen Fu, Yupeng Hou, Julian McAuley, and Rui Yan. 2025. Unlocking decoding-time controllability: Gradient-free multi-objective alignment with contrastive prompts. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human… find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  10. Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. Controllable preference optimization: Toward controllable multi-objective alignment. In Proceedings of the 2024 Conference on Empirical Methods in Natu… find
  11. Minchan Kwon, Junwon Ko, Kangil Kim, and Junmo Kim. 2025. Preference distillation via value based reinforcement learning. Preprint, arXiv:2509.16965. — [arXiv:2509.16965](https://arxiv.org/abs/2509.16965) find
  12. Alex M Lamb, Anirudh Goyal ALIAS PARTH GOYAL, Ying Zhang, Saizheng Zhang, Aaron C Courville, and Yoshua Bengio. 2016. Professor forcing: A new algorithm for training recurrent networks. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc. find
  13. Chengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue, Xiang Ao, and Qing He. 2025a. Gradient-adaptive policy optimization: Towards multi-objective alignment of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages… find
  14. Jiahui Li, Hanlin Zhang, Fengda Zhang, Tai-Wei Chang, Kun Kuang, Long Chen, and Jun Zhou. 2024a. Optimizing language models with fair and stable reward composition in reinforcement learning. In Proceedings of the 2024 Conference on Empirical Methods in Moxin Li, Yuantao Zhang, Wenjie Wang, Wentao Sh… find
  15. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024b. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. Preprint, arXiv:2406.11939.
  16. Baijiong Lin, Weisen Jiang, Yuancheng Xu, Hao Chen, and Ying-Cong Chen. 2025. PARM: Multi-objective test-time alignment via preference-aware autoregressive reward model. In Forty-second International Conference on Machine Learning. find
  17. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
  18. Qi Liu, Jingqing Ruan, Hao Li, Haodong Zhao, Desheng Wang, Jiansong Chen, Wan Guanglu, Xunliang Cai, Zhi Zheng, and Tong Xu. 2025. AMoPO: Adaptive multi-objective preference optimization without reward models and reference models. In Findings of the Association for Computational Linguistics: ACL 202… find
  19. OpenAI, :, Ahmed El-Kishky, Alexander Wei, Andre Saraiva, Borys Minaiev, Daniel Selsam, David Dohan, Francis Song, Hunter Lightman, Ignasi Clavera, Jakub Pachocki, Jerry Tworek, Lorenz Kuhn, Lukasz Kaiser, Mark Chen, Max Schwarzer, Mostafa Rohaninejad, Nat McAleese, and 7 others. 2025. Competitive p… find
  20. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… — Training language models to follow instructions with human feedback
  21. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  22. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc.
  23. Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2023. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In Thirty-seventh Conference on Neural Information Processing Sy… find
  24. Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hannaneh Hajishirzi, Noah A. Smith, and Simon S. Du. 2024. Decoding-time language model alignment with multiple objectives. In Advances in Neural Information Processing Systems, volume 37, pages 48875–48920. Curran Associates, Inc. find
  25. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. ↗
  26. Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. 2024a. Arithmetic control of LLMs for diverse user preferences: Directional preference alignment with multi-objective rewards. In Proceedings of the 62nd Annual Meeting of the Association for Computation…
  27. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024b. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10582– 10592, Miami, Florida, USA. Association for Computatio…
  28. Zhilin Wang, Alexander Bukharin, Olivier Delalleau, Daniel Egert, Gerald Shen, Jiaqi Zeng, Oleksii Kuchaiev, and Yi Dong. 2025a. Helpsteer2-preference: Complementing ratings with preferences. In International Conference on Learning Representations, volume 2025, pages 56548–56575. find
  29. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024c. Helpsteer 2: Open-source dataset for training top-performing reward models. In Advances in Neural Information Processing Systems, volume 37, pages 1… find
  30. Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Ellie Evans, Daniel Egert, Hoo-Chang Shin, Felipe Soares, Yi Dong, and Oleksii Kuchaiev. 2025b. Rlbff: Binary flexible feedback to bridge between human feedback & verifiable rewards. Preprint, arXiv:2509.21319. — [arXiv:2509.21319](https://arxiv.org/abs/2509.21319) find
  31. Guofu Xie, Xiao Zhang, Ting Yao, and Yunsheng Shi. 2025. Bone soups: A seek-and-soup model merging approach for controllable multi-objective generation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 27237–27263, Vienna, Aust… find
  32. Nuoya Xiong and Aarti Singh. 2025. Projection optimization: A general framework for multi-objective and multi-group RLHF. In Forty-second International Conference on Machine Learning. find
  33. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  34. Kailai Yang, Zhiwei Liu, Qianqian Xie, Jimin Huang, Tianlin Zhang, and Sophia Ananiadou. 2024a. Metaaligner: Towards generalizable multi-objective alignment of language models. In Advances in Neural Information Processing Systems, volume 37, pages 34453–34486. Curran Associates, Inc. find
  35. Rui Yang, Ruomeng Ding, Yong Lin, Huan Zhang, and Tong Zhang. 2024b. Regularizing hidden states enables learning generalizable reward model for llms. In Advances in Neural Information Processing Systems, volume 37, pages 62279–62309. Curran Associates, Inc. find
  36. Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. 2024c. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machi… find
  37. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2024. SafetyBench: Evaluating the safety of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap…
  38. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 3…
  39. Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. 2024. Panacea: Pareto alignment via preference adaptation for llms. In Advances in Neural Information Processing Systems, volume 37, pages 75522–75558. Curran Associates, Inc. find
  40. Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. 2024. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics: ACL 2024, pages 10586–10613, Bangkok, Thailand. Associati… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-self-guided-alignment-adaptive-preference/