Elamur Research⌕
☰
home / papers / 2026 / Beyond Compromise: Pareto-Lenient Consensus for Efficient Multi-Preference LLM Alignment

Beyond Compromise: Pareto-Lenient Consensus for Efficient Multi-Preference LLM Alignment

Paper · 2026 · Renxuan Tan, Rongpeng Li, Zhifeng Zhao et al.

Summary

Aligning an LLM with several human values at once, such as helpfulness, harmlessness and humor, is usually handled by fixed weighted sums of rewards or by MGDA-style gradient projection. The authors argue that these rigid schemes refuse to let any single objective temporarily get worse. As a result, training often stalls at a conservative stationary point, which they call a "risk-averse equilibrium", even when a better Pareto solution lies nearby.

They propose Pareto-Lenient Consensus (PLC), which borrows the idea of leniency from multi-agent RL. Each preference is treated as a player in a cooperative game. When the remaining objectives jointly stand to gain enough, PLC uses an adaptive mask to tolerate a short-term loss on a minority objective. Training uses a multi-objective PPO-style setup with a multi-head critic. To keep the method cheap, the coalition's potential gain is estimated from per-objective advantages instead of being computed over the full gradient space.

The theory shows that PLC can escape such stalemates and converges asymptotically to a "Pareto consensus equilibrium". Experiments reportedly beat baselines on both fixed-preference alignment and overall Pareto frontier quality.

Key points

  • Formal setup: Generation is modeled as a Multi-Objective MDP with a K-dimensional reward and a KL penalty. A multi-head critic V_ϕ supplies a separate advantage for each objective.
  • Gradient split: At each token, objectives fall into a positive-advantage coalition P or a conflict set N, and the update is decomposed into v_P + v_N.
  • Equilibrium definitions: A risk-averse equilibrium is a Pareto stationary point where some objective k still has a large "latent coalition surplus" S_−k, meaning the other objectives could still gain substantially. A Pareto Consensus Equilibrium (PCE) is a stationary point where this surplus is near zero.
  • Efficient estimate: S_−k is approximated by the normalized weighted sum of the other objectives' advantages. This avoids an intractable optimization over the LLM's high-dimensional gradient space.
  • Availability and limitations: Code is released at github.com/lakjx/consensus-equilibra. The authors list three limitations: no standard evaluation protocol for multi-preference alignment, dependence on the quality of proxy reward models, and leniency that may persist in highly stochastic landscapes, which a time-based decay schedule could address.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🔬Zhejiang LabHangzhou · China7 works · 1,596 citations🎓Macau University of Science and TechnologyMacao · China3 works · 178 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 worksComputational Theory and Mathematics · 252 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Renxuan Tan, Rongpeng Li, Zhifeng Zhao and Honggang Zhang. 2026. Beyond Compromise: Pareto-Lenient Consensus for Efficient Multi-Preference LLM Alignment. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1879

@inproceedings{tan2026beyond,
  title = {{Beyond Compromise: Pareto-Lenient Consensus for Efficient Multi-Preference LLM Alignment}},
  author = {Renxuan Tan, Rongpeng Li, Zhifeng Zhao and Honggang Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1879},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1879}
}

References

71 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, and Zheng Wen. 2025. Multi-objective preference optimization: Improving human alignment of generative models. arXiv preprint arXiv:2505.10892. — [arXiv:2505.10892](https://arxiv.org/abs/2505.10892) find
  2. Johannes Bader and Eckart Zitzler. 2011. HypE: An algorithm for fast hypervolume-based many-objective optimization. Evolutionary Computation, 19:45–76. find
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, and 12 others. 202… find
  4. Xinye Cai, Yushun Xiao, Miqing Li, Han Hu, Hisao Ishibuchi, and Xiaoping Li. 2021. A grid-based inverted generational distance for multi/many-objective optimization. IEEE Transactions on Evolutionary Computation, 25(1):21–34. find
  5. Weiyu Chen, Baijiong Lin, Xiaoyuan Zhang, Xi Lin, Han Zhao, Qingfu Zhang, and James T. Kwok. 2025. Gradient-based multi-objective deep learning: Algorithms, theories, applications, and beyond. Preprint, arXiv:2501.10945. — [arXiv:2501.10945](https://arxiv.org/abs/2501.10945) find
  6. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Proceedings of the 31st International Conference on Neural Information Processing Systems, page 4302–4310. find
  7. Filippos Christianos, Georgios Papoudakis, and Stefano V. Albrecht. 2023. Pareto Actor-Critic for equilibrium selection in multi-agent reinforcement learning. Transactions on Machine Learning Research. find
  8. Iain D. Couzin, Christos C. Ioannou, Güven Demirel, Thilo Gross, Colin J. Torney, Andrew Hartnett, Larissa Conradt, Simon A. Levin, and Naomi E. Leonard. 2011. Uninformed individuals promote democratic consensus in animal groups. Science, 334(6062):1578–1580. find
  9. Elia Cunegatti, Leonardo Custode, and Giovanni Iacca. 2024. Many-objective evolutionary influence maximization: Balancing spread, budget, fairness, and time. arXiv preprint arXiv:arXiv:2403.18755. — [arXiv:2403.18755](https://arxiv.org/abs/2403.18755) find
  10. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2023. Safe RLHF: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773.
  11. Yann Dauphin, Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, Surya Ganguli, and Yoshua Bengio. 2014. Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. Preprint, arXiv:1406.2572. find
  12. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, and 245 others. 2025. DeepSeek-V3.2: Pushing the frontier of open…
  13. Jean-Antoine Désidéri. 2012. Multiple-gradient descent algorithm (mgda) for multiobjective optimization. Comptes Rendus Mathematique, 350(5):313–318. find
  14. Saeed Ghadimi and Guanghui Lan. 2013b. Stochastic first- and zeroth-order methods for nonconvex stochastic programming. arXiv preprint arXiv:1309.5549. — [arXiv:1309.5549](https://arxiv.org/abs/1309.5549) find
  15. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  16. Raghav Gupta, Ryan Sullivan, Yunxuan Li, Samrat Phatale, and Abhinav Rastogi. 2025. Robust multi-objective preference alignment with online DPO. arXiv preprint arXiv:2503.00295. — [arXiv:2503.00295](https://arxiv.org/abs/2503.00295) find
  17. Qiang He and Setareh Maghsudi. 2025. Pareto multi-objective alignment for language models. arXiv preprint arXiv:2508.07768. — [arXiv:2508.07768](https://arxiv.org/abs/2508.07768) find
  18. Daisuke Hirota. 2025. Rational adversaries and the maintenance of fragility: A game-theoretic theory of rational stagnation. arXiv preprint arXiv:2510.22232. — [arXiv:2510.22232](https://arxiv.org/abs/2510.22232) find
  19. Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, and Prithviraj Ammanabrolu. 2023. Personalized soups: Personalized large language model alignment via post-hoc parameter merging. arXiv preprint arXiv:2310.11564. find
  20. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beaver-Tails: Towards improved safety alignment of LLM via a human-preference dataset. arXiv preprint arXiv:2307.04657.
  21. Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M. Kakade, and Michael I. Jordan. 2017a. How to escape saddle points efficiently. Preprint, arXiv:1703.00887. — [arXiv:1703.00887](https://arxiv.org/abs/1703.00887) find
  22. Chi Jin, Rong Ge, Praneeth Netrapalli, Sham M. Kakade, and Michael I. Jordan. 2017b. How to escape saddle points efficiently. arXiv preprint arXiv:1703.00887. — [arXiv:1703.00887](https://arxiv.org/abs/1703.00887) find
  23. Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2023. Understanding the effects of RLHF on LLM generalisation and diversity. arXiv preprint arXiv:2310.06452.
  24. Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. 2015. Escaping from saddle points — online stochastic gradient for tensor decomposition. arXiv preprint arXiv:1503.02101. — [arXiv:1503.02101](https://arxiv.org/abs/1503.02101) find
  25. Kyungmin Lee, Xiaohang Li, Qifei Wang, Junfeng He, Junjie Ke, Ming-Hsuan Yang, Irfan Essa, Jinwoo Shin, Feng Yang, and Yinxiao Li. 2025. Calibrated multi-preference optimization for aligning diffusion models. arXiv preprint arXiv:2502.02588. — [arXiv:2502.02588](https://arxiv.org/abs/2502.02588) find
  26. Saeed Ghadimi and Guanghui Lan. 2013a. Accelerated gradient methods for nonconvex nonlinear and stochastic programming. arXiv preprint arXiv:1310.3787. — [arXiv:1310.3787](https://arxiv.org/abs/1310.3787) find
  27. Chengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue, Xiang Ao, and Qing He. 2025. Gradient-adaptive policy optimization: Towards multi-objective alignment of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics, pages 11214–11232. find
  28. Miqing Li and Xin Yao. 2019. Quality evaluation of solution sets in multiobjective optimisation: A survey. ACM Computing Surveys, 52(2). find
  29. Liang Lin, Zhihao Xu, Junhao Dong, Jian Zhao, Yuchen Yuan, Guibin Zhang, Miao Yu, Yiming Zhang, Zhengtao Yao, Huahui Yi, Dongrui Liu, Xinfeng Li, and Kun Wang. 2025. OrthAlign: Orthogonal subspace decomposition for non-interfering multi-objective alignment. arXiv preprint arXiv:2509.24610. — [arXiv:2509.24610](https://arxiv.org/abs/2509.24610) find
  30. Samuel Lippl and Jack W. Lindsey. 2024. Inductive biases of multi-task learning and finetuning: multiple regimes of feature reuse. Preprint, arXiv:2310.02396. — [arXiv:2310.02396](https://arxiv.org/abs/2310.02396) find
  31. Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. 2021. Conflict-averse gradient descent for multi-task learning. In Proceedings of the 35th International Conference on Neural Information Processing Systems. find
  32. Zixuan Liu, Xiaolin Sun, and Zizhan Zheng. 2024. Enhancing LLM safety via constrained direct preference optimization. arXiv preprint arXiv:2403.02475. — [arXiv:2403.02475](https://arxiv.org/abs/2403.02475) find
  33. Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Côme Fiegel, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, and Bilal Piot. 2024. Nash learni… find
  34. Daye Nam, Andrew Macvean, Vincent Hellendoorn, Bogdan Vasilescu, and Brad Myers. 2024. Using an LLM to help with code understanding. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages 1–13. find
  35. Gregory Palmer, Karl Tuyls, Daan Bloembergen, and Rahul Savani. 2018. Lenient multi-agent deep reinforcement learning. arXiv preprint arXiv:1707.04402. — [arXiv:1707.04402](https://arxiv.org/abs/1707.04402) find
  36. Liviu Panait, Keith Sullivan, and Sean Luke. 2006. Lenient learners in cooperative multiagent systems. In Proceedings of the Fifth International Joint Conference on Autonomous Agents and Multiagent Systems, page 801–803. find
  37. Bezalel Peleg. 2003. Introduction to the Theory of Cooperative Games. Introduction to the Theory of Cooperative Games. find
  38. Barna Pásztor, Thomas Kleine Buening, and Andreas Krause. 2025. Stackelberg learning from human feedback: Preference optimization as a sequential game. arXiv preprint arXiv:2512.16626. — [arXiv:2512.16626](https://arxiv.org/abs/2512.16626) find
  39. Alexandre Rame, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. 2023. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. In Proceedings of the 37th International Conference on Neural In… find
  40. Herbert Robbins and Sutton Monro. 1951. A Stochastic Approximation Method. The Annals of Mathematical Statistics, 22(3):400 – 407. find
  41. John Schulman, Sergey Levine, Philipp Moritz, Michael I. Jordan, and Pieter Abbeel. 2017a. Trust region policy optimization. arXiv preprint arXiv:1502.05477. find
  42. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017b. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  43. Ruizhe Shi, Yifang Chen, Yushi Hu, Alisa Liu, Hannaneh Hajishirzi, Noah A. Smith, and Simon S. Du. 2024. Decoding-time language model alignment with multiple objectives. arXiv preprint arXiv:2406.18853. — [arXiv:2406.18853](https://arxiv.org/abs/2406.18853) find
  44. Zhekun Shi, Kaizhao Liu, Qi Long, Weijie J. Su, and Jiancong Xiao. 2025. Fundamental limits of game-theoretic LLM alignment: Smith consistency and preference matching. arXiv preprint arXiv:2505.20627. — [arXiv:2505.20627](https://arxiv.org/abs/2505.20627) find
  45. Oliver Slumbers, David Henry Mguni, Stefano B. Blumberg, Stephen McAleer, Yaodong Yang, and Jun Wang. 2023. A game-theoretic framework for managing risk in multi-agent systems. In Proceedings of the 40th International Conference on Machine Learning. find
  46. Bertram I. Spector. 2006. Resiliency in negotiation: Bouncing back from impasse. International Negotiation, 11:273–286. find
  47. Yuhui Sun, Xiyao Wang, Zixi Li, Zhenlong Yuan, and Jinman Zhao. 2025. Multi-preference lambda-weighted listwise DPO for small-scale model alignment. arXiv preprint arXiv:2506.19780. — [arXiv:2506.19780](https://arxiv.org/abs/2506.19780) find
  48. Gokul Swamy, Christoph Dann, Rahul Kidambi, Zhiwei Steven Wu, and Alekh Agarwal. 2024. A minimaximalist approach to reinforcement learning from human feedback. In Proceedings of the 41st International Conference on Machine Learning. find
  49. Prashant Trivedi, Souradip Chakraborty, Avinash Reddy, Vaneet Aggarwal, Amrit Singh Bedi, and George K. Atia. 2025. Align-pro: a principled approach to prompt optimization for llm alignment. In Proceedings of the Thirty-Ninth AAAI Conference on Artificial Intelligence and Thirty-Seventh Conference o… find
  50. Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, Shengyi Huang, Kashif Rasul, and Quentin Gallouédec. 2020. TRL: Transformer reinforcement learning. https://github.com/huggingface/ trl. ↗
  51. Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. 2024a. Arithmetic control of LLMs for diverse user preferences: Directional preference alignment with multi-objective rewards. arXiv preprint arXiv:2402.18571.
  52. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024b. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. arXiv preprint arXiv:2406.12845.
  53. Tianze Wang, Dongnan Gui, Yifan Hu, Shuhang Lin, and Linjun Zhang. 2025. Mpo: An efficient post-processing framework for mixing diverse preference alignment. arXiv preprint arXiv:2502.18699. — [arXiv:2502.18699](https://arxiv.org/abs/2502.18699) find
  54. Xinran Wang, Qi Le, Ammar Ahmed, Enmao Diao, Yi Zhou, Nathalie Baracaldo, Jie Ding, and Ali Anwar. 2024c. MAP: Multi-human-value alignment palette. arXiv preprint arXiv:2410.19198. — [arXiv:2410.19198](https://arxiv.org/abs/2410.19198) find
  55. Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Mao, Sitaram Asur, and 1 others. 2024d. A comprehensive survey of LLM alignment techniques: RLHF, RLAIF, PPO, DPO and more. arXiv preprint arXiv:2407.16216. — [arXiv:2407.16216](https://arxiv.org/abs/2407.16216) find
  56. Ermo Wei and Sean Luke. 2016. Lenient learning in independent-learner stochastic cooperative games. Journal of Machine Learning Research, 17:84:1– 84:42. find
  57. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transfo…
  58. Fang Wu, Xu Huang, Weihao Xuan, Zhiwei Zhang, Yijia Xiao, Guancheng Wan, Xiaomin Li, Bing Hu, Peng Xia, Jure Leskovec, and Yejin Choi. 2025. Multiplayer Nash preference optimization. arXiv preprint arXiv:2509.23102. find
  59. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023a. Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01693.
  60. Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A. Smith, Mari Ostendorf, and Hannaneh Hajishirzi. 2023b. Fine-grained human feedback gives better rewards for language model training. In Proceedings of the 37th International Conference on Neural Information Proc…
  61. Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. 2023. Iterative preference learning from human feedback: Bridging theory and practice for RLHF under kl-constraint. arXiv preprint arXiv:2312.11456. find
  62. Jinluan Yang, Dingnan Jin, Anke Tang, Li Shen, Didi Zhu, Zhengyu Chen, Ziyu Zhao, Daixin Wang, Qing Cui, Zhiqiang Zhang, Jun Zhou, Fei Wu, and Kun Kuang. 2025. Mix data or merge models? balancing the helpfulness, honesty, and harmlessness of large language model via model merging. arXiv preprint arX… — [arXiv:2502.06876](https://arxiv.org/abs/2502.06876) find
  63. Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. 2024a. Rewards-in-context: multi-objective alignment of foundation models with dynamic preference adjustment. In Proceedings of the 41st International Conference on Machine Learning. find
  64. Xianjun Yang, Liangming Pan, Xuandong Zhao, Haifeng Chen, Linda Ruth Petzold, William Yang Wang, and Wei Cheng. 2024b. A survey on detection of LLMs-generated content. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9786–9805. find
  65. Biao Yi, Jiahao Li, Baolei Zhang, Lihai Nie, Tong Li, Tiansheng Huang, and Zheli Liu. 2025. Gradient surgery for safe LLM fine-tuning. arXiv preprint arXiv:2508.07172. — [arXiv:2508.07172](https://arxiv.org/abs/2508.07172) find
  66. Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. arXiv preprint arXiv:2001.06782.
  67. Fan Zhang, Chenglu Li, Owen Henkel, Wanli Xing, Sami Baral, Neil Heffernan, and Hai Li. 2025. Math-LLMs: AI cyberinfrastructure with pre-trained transformers for math education. International Journal of Artificial Intelligence in Education, 35(2):509–532. find
  68. Zeyu Zhao, Yueling Che, Kaichen Liu, Jian Li, and Jun-mei Yao. 2025. Multi-policy pareto front tracking based online and offline multi-objective reinforcement learning. arXiv preprint arXiv:2508.02217. — [arXiv:2508.02217](https://arxiv.org/abs/2508.02217) find
  69. Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. 2024. Panacea: Pareto alignment via preference adaptation for LLMs. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pages 75522–75558. find
  70. Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. 2024. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. arXiv preprint arXiv:2310.03708. find
  71. Siqi Zhu, David Zhang, Pedro Cisneros-Velarde, and Jiaxuan You. 2025. GTAlign: Game-theoretic alignment of LLM assistants for social welfare. arXiv preprint arXiv:2510.08872. — [arXiv:2510.08872](https://arxiv.org/abs/2510.08872) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tan-2026-beyond-compromise-pareto-lenient-consensus/