Elamur Research⌕
☰
home / papers / 2025 / Binary Classifier Optimization for Large Language Model Alignment

Binary Classifier Optimization for Large Language Model Alignment

Paper · 2025 · Seungjae Jung, Gunsoo Han, Daniel Wontae Nam et al.

Summary

Deployed chat assistants mostly gather simple thumbs-up/thumbs-down ratings. Standard alignment methods such as RLHF and DPO, however, need pairs of a preferred and a dispreferred response. Jung et al. (Kakao Corp) offer a theoretical bridge between the two. They show that a binary classifier whose logit is the implicit DPO reward, trained with binary cross-entropy, minimizes an upper bound on the DPO loss. To tighten that bound, they add a reward shift that subtracts a reference value δ from the rewards. Combining the two gives Binary Classifier Optimization (BCO).

The analysis also points to a weakness in KTO, the main earlier binary-feedback method: its reference point degenerates during training. Experiments use paired preference data (UltraFeedback, Capybara) and HelpSteer2, a Likert-5 dataset built on real user queries, with four Llama and Qwen base models. BCO matches DPO and beats KTO on paired data, and it outperforms both on HelpSteer2. This makes cheap binary feedback a credible signal for alignment.

Key points

  • Theory: the authors prove that log σ(a+b) is greater than log σ(a) + log σ(b). It follows that the BCE classifier loss bounds the DPO loss from above. The slack in this bound is smallest when δ equals the average of the chosen and rejected rewards.
  • KTO diagnosis: KTO's batch-estimated reference point z_ref collapses to 0 early in training. At that point KTO behaves much like plain BCE, and both perform similarly.
  • Models and benchmarks:
    • Base models are Llama-3.2-3B, Qwen2.5-3B, Llama-3.1-8B-Instruct and Qwen2.5-7B-Instruct.
    • Evaluation uses MT Bench, AlpacaEval 2.0 Length Controlled, and Arena-Hard (win rate against GPT-4-0314), with three random seeds.
  • Output length: BCO and DPO produce completions of similar length, and so do KTO and BCE. For example, on Capybara with Llama-3.2-3B, BCO averages about 417 tokens versus about 387 for DPO and about 323 for KTO.
  • Limitations:
    • No real-world benchmark with binary annotations exists for evaluation.
    • The method optimizes only an upper bound, leaving a gap to the true DPO objective.
    • Binary labels discard the comparative information that paired data contains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

Also on the byline: Kakao Corporation.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Seungjae Jung, Gunsoo Han, Daniel Wontae Nam and Kyoung-Woon On. 2025. Binary Classifier Optimization for Large Language Model Alignment. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.93

@inproceedings{jung2025binary,
  title = {{Binary Classifier Optimization for Large Language Model Alignment}},
  author = {Seungjae Jung, Gunsoo Han, Daniel Wontae Nam and Kyoung-Woon On},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.93},
  url = {https://doi.org/10.18653/v1/2025.acl-long.93}
}

References

33 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2023. Introducing claude. find
  2. Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. 2023. A general theoretical paradigm to understand learning from human preferences. ArXiv, abs/2310.12036.
  3. Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324– 345. find
  4. Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, and Jun Zhu. 2024a. Noise contrastive alignment of language models with explicit rewards. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  5. Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. 2024b. Self-play fine-tuning converts weak language models to strong language models. ArXiv, abs/2401.01335. find
  6. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. ArXiv, abs/2310.01377. find
  7. Luigi Daniele and Suphavadeeprasit. 2023. Amplify-instruct: Synthetically generated diverse multi-turn conversations for efficient llm training. arXiv preprint arXiv:(coming soon). find
  8. Tri Dao. 2024. FlashAttention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations (ICLR).
  9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  10. Yann Dubois, Percy Liang, and Tatsunori Hashimoto. 2024. Length-controlled alpacaeval: A simple debiasing of automatic evaluators. In First Conference on Language Modeling. find
  11. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Model alignment as prospect theoretic optimization. In Forty-first International Conference on Machine Learning. find
  12. Amelia Glaese, Nathan McAleese, Maja Trkebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, A. See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fri… — Improving alignment of dialogue agents via targeted human judgements
  13. Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. 2024. Rewardbench: Evaluating reward models for language modeling. Preprint, arXiv:2403.13787. find
  14. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. 2024. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. Preprint, arXiv:2406.11939.
  15. Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu. 2023. Statistical rejection sampling improves preference optimization. ArXiv, abs/2309.06657.
  16. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. In International Conference on Learning Representations. find
  17. OpenAI. 2022. Introducing chatgpt. find
  18. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Trai… — Training language models to follow instructions with human feedback
  19. Sundar Pichai and Demis Hassabis. 2023. Introducing gemini: our largest and most capable ai model. find
  20. Rafael Rafailov, Joey Hejna, Ryan Park, and Chelsea Finn. 2024. From $r$ to $q^*$: Your language model is secretly a q-function. In First Conference on Language Modeling. find
  21. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems.
  22. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. ArXiv, abs/1707.06347. find
  23. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008– 3021. find
  24. Yunhao Tang, Daniel Zhaohan Guo, Zeyu Zheng, Daniele Calandriello, Yuan Cao, Eugene Tarassov, Rémi Munos, Bernardo Ávila Pires, Michal Valko, Yong Cheng, et al. 2024. Understanding the performance gap between online and offline alignment algorithms. arXiv preprint arXiv:2405.08448. — [arXiv:2405.08448](https://arxiv.org/abs/2405.08448) find
  25. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  26. Amos Tversky and Daniel Kahneman. 1992. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5:297– 323. find
  27. Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. 2020. Trl: Transformer reinforcement learning. https://github.com/huggingface/trl. ↗
  28. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024. Helpsteer2: Open-source dataset for training top-performing reward models. Preprint, arXiv:2406.08673. find
  29. Shusheng Xu, Wei Fu, Jiaxuan Gao, Wenjie Ye, Weilin Liu, Zhiyu Mei, Guangju Wang, Chao Yu, and Yi Wu. 2024. Is DPO superior to PPO for LLM alignment? a comprehensive study. In Forty-first International Conference on Machine Learning. find
  30. Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. 2024. Self-rewarding language models. ArXiv, abs/2401.10020. find
  31. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 3…
  32. Daniel M. Ziegler, Nisan Stiennon, Jeff Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. ArXiv, abs/1909.08593. find
  33. RyokoAI. 2023. Ryokoai/sharegpt52k. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jung-2025-binary-classifier-optimization-large-language/