Elamur Research⌕
☰
home / papers / 2026 / Personalizing LLMs with Binary Feedback: A Preference-Calibrated Optimization Framework

Personalizing LLMs with Binary Feedback: A Preference-Calibrated Optimization Framework

Paper · 2026 · Xilai Ma, Liye Zhao, Weijun Yao et al.

Summary

Most LLM personalization methods learn from one user's history in isolation and ignore what sets that user apart from others. DPO-style alignment is also hard to apply, because user histories contain no paired better/worse answers to the same prompt. The authors propose C-BPO, which treats personalization as binary-feedback preference optimization. The target user's records act as "thumbs-up" examples, and records from a pool of auxiliary users act as implicit "thumbs-down" signals. Both are used to train a user-specific LoRA module.

Other users share task knowledge and common stylistic habits with the target user, so penalizing their data wholesale would suppress useful behavior. The paper calls this the preference overlap problem. To address it, the authors derive a loss from Positive-Unlabeled (PU) learning. This loss treats auxiliary data as an unlabeled mixture and subtracts an estimated "positive bias," weighted by a coefficient α. A separate EMA-based reference point keeps training stable when positive and negative sets are imbalanced.

Across five personalized generation tasks and several backbone LLMs, C-BPO consistently beats strong baselines. This suggests that unpaired data plus inter-user contrast can support fine-grained personalization.

Key points

  • C-BPO builds on binary-feedback methods KTO and BCO (Eq. 2–3), adapting their reference-point formulation to a setting where the "negatives" are noisy.
  • The PU-learning objective estimates the genuine negative risk by taking the total auxiliary risk and removing the expected risk contributed by target-like preferences concealed within the auxiliary data. α controls how much is removed.
  • In the analysis, users with less distinctive (Non-unique) histories need a higher α, while Unique users do best with a lower α. An embedding-based heuristic (Appendix E.3) gives an α estimate close to the empirical optimum.
  • Compared approaches include retrieval/profile prompting (RAG, PAG), PEFT methods, and inter-user methods such as DPL, DEP, CoPE and CFRAG. C-BPO avoids costly LLM reasoning and rejection sampling.
  • Stated limitations: sensitivity to α, the need for centralized raw data from other users (a privacy issue for federated or on-device use), and evaluation limited to generative benchmarks. The PDF title says "Preference-Corrected" rather than "Preference-Calibrated."

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China158 works · 28,841 citations🎓Nanyang Technological UniversitySingapore · Singapore156 works · 25,644 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Xilai Ma, Liye Zhao, Weijun Yao, Haibing Di, Wenya Wang and Jing Li. 2026. Personalizing LLMs with Binary Feedback: A Preference-Calibrated Optimization Framework. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1222

@inproceedings{ma2026personalizing,
  title = {{Personalizing LLMs with Binary Feedback: A Preference-Calibrated Optimization Framework}},
  author = {Xilai Ma, Liye Zhao, Weijun Yao, Haibing Di, Wenya Wang and Jing Li},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1222},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1222}
}

References

40 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jessa Bekker and Jesse Davis. 2020. Learning from positive and unlabeled data: A survey. Machine learning, 109(4):719–760. find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems (NeurIPS), 33:1877–1901.
  3. Hyungjune Bu, ChanJoo Jung, Minjae Kang, and Jaehyung Kim. 2025. Personalized LLM decoding via contrasting personal preference. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 33946– 33966. find
  4. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024a. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv preprint arXiv:2402.03216.
  5. Jin Chen, Zheng Liu, Xu Huang, Chenwang Wu, Qi Liu, Gangwei Jiang, Yuanhao Pu, Yuxuan Lei, Xiaolong Chen, Xingmei Wang, and 1 others. 2024b. When large language models meet personalization: Perspectives of challenges and opportunities. World Wide Web, 27(4):42. find
  6. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  7. Charles Elkan and Keith Noto. 2008. Learning classifiers from only positive and unlabeled data. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining (KDD), pages 213–220. find
  8. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Model alignment as prospect theoretic optimization. In Forty-first International Conference on Machine Learning (ICML). find
  9. Jian Guan, Junfei Wu, Jia-Nan Li, Chuanqi Cheng, and Wei Wu. 2025. A survey on personalized Alignment—The missing piece for large language models in real-world applications. In Findings of the Association for Computational Linguistics: ACL 2025, pages 5313–5333. Association for Computational Linguis… find
  10. Weiyang Guo, Zesheng Shi, Liye Zhao, Jiayuan Ma, Zeen Zhu, Junxian He, Min Zhang, and Jing Li. 2026a. E3-tir: Enhanced experience exploitation for tool-integrated reasoning. arXiv preprint arXiv:2604.09455. — [arXiv:2604.09455](https://arxiv.org/abs/2604.09455) find
  11. Weiyang Guo, Zesheng Shi, Zeen Zhu, Yuan Zhou, Min Zhang, and Jing Li. 2026b. Backdoors in rlvr: Jailbreak backdoors in llms from verifiable reward. arXiv preprint arXiv:2604.09748.
  12. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2021. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR).
  13. Antti Hyttinen, Frederick Eberhardt, and Patrik O Hoyer. 2013. Experiment selection for causal discovery. The Journal of Machine Learning Research, 14(1):3041– 3071. find
  14. Caglar Irmak, Beth Vallen, and Sankar Sen. 2010. You like what i like, but i don’t like what you like: Uniqueness motivations in product preferences. Journal of Consumer Research, 37(3):443–455. find
  15. Shantanu Jain, Justin Delano, Himanshu Sharma, and Predrag Radivojac. 2020. Class prior estimation with biased positives and unlabeled examples. In Proceedings of the AAAI conference on artificial intelligence (AAAI), volume 34, pages 4255–4263. find
  16. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  17. Seungjae Jung, Gunsoo Han, Daniel Wontae Nam, and Kyoung-Woon On. 2025. Binary classifier optimization for large language model alignment. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 1858– 1872.
  18. Ryuichi Kiryo, Gang Niu, Marthinus C Du Plessis, and Masashi Sugiyama. 2017. Positive-unlabeled learning with non-negative risk estimator. In Advances in neural information processing systems (NeurIPS), volume 30. find
  19. Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, and 1 others. 2024. Longlamp: A benchmark for personalized long-form text generation. arXiv preprint arXiv:2407.11016. — [arXiv:2407.11016](https://arxiv.org/abs/2407.11016) find
  20. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in neural information processing systems (NeurI…
  21. Zhuo Li, Guodong Du, Weiyang Guo, Yigeng Zhou, Xiucheng Li, Wenya Wang, Fangming Liu, Yequan Wang, Deheng Ye, Min Zhang, and 1 others. 2025. Multi-objective large language model alignment with hierarchical experts. arXiv preprint arXiv:2505.20925. — [arXiv:2505.20925](https://arxiv.org/abs/2505.20925) find
  22. Jiongnan Liu, Yutao Zhu, Shuting Wang, Xiaochi Wei, Erxue Min, Yu Lu, Shuaiqiang Wang, Dawei Yin, and Zhicheng Dou. 2025. Llms+ persona-plug= personalized llms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL), pages 9373–9385.
  23. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR). find
  24. Sheshera Mysore, Zhuoran Lu, Mengting Wan, Longqi Yang, Bahareh Sarrafzadeh, Steve Menezes, Tina Baghaee, Emmanuel Barajas Gonzalez, Jennifer Neville, and Tara Safavi. 2024. Pearl: Personalizing large language model writing assistants with generation-calibrated retrievers. In Proceedings of the 1st… find
  25. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. In Advances in neural information processing systems (NeurIPS), volu…
  26. Yilun Qiu, Tianhao Shi, Xiaoyan Zhao, Fengbin Zhu, Yang Zhang, and Fuli Feng. 2025a. Latent inter-user difference modeling for llm personalization. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 10610–10628.
  27. Yilun Qiu, Xiaoyan Zhao, Yang Zhang, Yimeng Bai, Wenjie Wang, Hong Cheng, Fuli Feng, and Tat-Seng Chua. 2025b. Measuring what makes you unique: Difference-aware user modeling for enhancing LLM personalization. In Findings of the Association for Computational Linguistics: ACL 2025, pages 21258– 21277… find
  28. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in neural information processing systems (NeurIPS), volume 36, pages 53728–53741.
  29. Xutao Wang, Hanting Chen, Tianyu Guo, and Yunhe Wang. 2023. Pue: Biased positive-unlabeled learning enhancement by causal inference. Advances in Neural Information Processing Systems (NeurIPS), 36:19783–19798. find
  30. Chris Richardson, Yao Zhang, Kellen Gillespie, Sudipta Kar, Arshdeep Singh, Zeynab Raeesy, Omar Zia Khan, and Abhinav Sethy. 2023. Integrating summarization and retrieval for enhanced personalization via large language models. arXiv preprint arXiv:2310.20081. — [arXiv:2310.20081](https://arxiv.org/abs/2310.20081) find
  31. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 40 others. 2024. Qwen2 technical report. arXiv preprint arXi… find
  32. Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2024. Lamp: When large language models meet personalization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 7370–7392.
  33. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  34. Teng Shi, Jun Xu, Xiao Zhang, Xiaoxue Zang, Kai Zheng, Yang Song, and Han Li. 2025. Retrieval augmented generation with collaborative filtering for personalized text generation. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR)… find
  35. Charles R Snyder and Howard L Fromkin. 2012. Uniqueness: The human pursuit of difference. find
  36. Zhaoxuan Tan, Zheyuan Liu, and Meng Jiang. 2024a. Personalized pieces: Efficient personalized large language models through collaborative efforts. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6459–6475.
  37. Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. 2024b. Democratizing large language models via personalized parameter-efficient fine-tuning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6476–6491.
  38. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  39. Yu-Min Tseng, Yu-Chao Huang, Teng-Yun Hsiao, Wei-Lin Chen, Chao-Wei Huang, Yu Meng, and Yun-Nung Chen. 2024. Two tales of persona in llms: A Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, and 1 others. 2025a. The landsc… — [arXiv:2509.02547](https://arxiv.org/abs/2509.02547) find
  40. Linhai Zhang, Jialong Wu, Deyu Zhou, and Yulan He. 2025b. Proper: A progressive learning framework for personalized large language models with group-level adaptation. In The 63rd Annual Meeting of the Association for Computational Linguistics (ACL).

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2026-personalizing-llms-binary-feedback-preference/