Elamur Research⌕
☰
home / papers / 2026 / DynamicFocalPO: Adaptive Focusing Strategy for Preference Optimization

DynamicFocalPO: Adaptive Focusing Strategy for Preference Optimization

Paper · 2026 · Shu Zhou, Junan Chen, Rui Ling et al.

Summary

Preference optimization methods such as DPO align LLMs on pairs of preferred and rejected answers. However, they are easily disturbed by pairs that the model initially ranks the wrong way. FocalPO reduces the influence of such pairs with a modulating factor p^γ, but it keeps γ constant for the entire run. The authors argue that the right amount of damping should shift as the model gets better. They propose DynamicFocalPO, which turns γ into a function of the training step.

The design follows curriculum-learning logic. A large γ at the start means updates come mostly from pairs the model already orders correctly. As γ shrinks, difficult pairs gain weight, until the objective is almost identical to DPO. The authors tested linear, exponential and cosine schedules, and cosine worked best.

Models were trained on UltraFeedback (Mistral-Base-7B) and Llama3-ultrafeedback-armorm (Llama-3-Instruct-8B). The method outperforms DPO, SimPO and a grid-tuned FocalPO on most Alpaca Eval 2.0 and Arena-Hard metrics. This is a cheap change that needs little tuning, and a gradient analysis explains why it works.

Key points

  • On Llama-3-Instruct-8B, the cosine schedule scores 56.4 LCWR on Alpaca Eval 2.0 and 36.0 WR on Arena-Hard. The best fixed-γ FocalPO (γ=0.08) scores 55.2 and 35.0, and DPO scores 48.2 and 33.1. Results are averaged over 3 seeds, and the gains are significant at p<0.05.
  • On Mistral-Base-7B, the method reaches 24.1 LCWR and 18.2 on Arena-Hard, compared with 20.6 and 16.4 for DPO. SimPO still has a slightly higher raw WR on Alpaca Eval 2.0 (21.4 vs 21.2).
  • A single setting, γ_max=0.1 and γ_min=0.01, was used in all experiments. An ablation grid on Llama-3 confirmed it as the best choice: a high γ_min or a low γ_max both reduce quality.
  • The share of gradient coming from correctly ranked pairs drops from 78% to 47% over training, approaching DPO's ~46%. In FocalPO this share stays at about 61%.
  • General ability on OpenLLM leaderboard tasks is preserved. The average is 66.4, against 65.7 for FocalPO and 65.4 for DPO. GSM8k is lower than DPO (73.8 vs 75.4), but the gap is not significant. Experiments cover only 7B/8B models and English.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢BaiduBeijing · China84 works · 17,681 citations🎓Nanjing UniversityNanjing · China99 works · 8,510 citations🎓Nanjing University of Finance and EconomicsNanjing · China4 works · 2 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Shu Zhou, Junan Chen, Rui Ling, Xin Wang, Tao Fan and Hao Wang. 2026. DynamicFocalPO: Adaptive Focusing Strategy for Preference Optimization. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1009

@inproceedings{zhou2026dynamicfocalpo,
  title = {{DynamicFocalPO: Adaptive Focusing Strategy for Preference Optimization}},
  author = {Shu Zhou, Junan Chen, Rui Ling, Xin Wang, Tao Fan and Hao Wang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1009},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1009}
}

References

39 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Afra Amini, Tim Viber, and Ryan Cotterell. 2024. Direct preference optimization with an offset. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. find
  2. Mohammad Gheshlaghi Azar, Mark Rowland, Bilal Piot, Daniel Guo, Daniele Calandriello, Michal Valko, and Rémi Munos. 2024. A general theoretical paradigm to understand learning from human preferences. In International Conference on Artificial Intelligence and Statistics, pages 4447–4455.
  3. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th annual international conference on machine learning, pages 41–48.
  4. Angelica Chen, Sadhika Malladi, Lily H Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, and Kyunghyun Cho. 2024. Preference learning algorithms do not learn preference rankings. Advances in Neural Information Processing Systems, 37:101928– 101968. find
  5. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. find
  6. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377. find
  7. Tri Dao. 2023. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint arXiv:2307.08691.
  8. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. find
  9. Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. 2024. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306. find
  10. Aditya Golatkar, Alessandro Achille, and Stefano Soatto. 2019. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. Advances in Neural Information Processing Systems, 32. find
  11. Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. find
  12. Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive growing of gans for improved quality, stability, and variation. In International Conference on Learning Representations. find
  13. M Kumar, Benjamin Packer, and Daphne Koller. 2010. Self-paced learning for latent variable models. volume 23. find
  14. Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. 2024. From crowdsourced data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406.11939.
  15. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. find
  16. Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2017. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980–2988.
  17. Tong Liu, Xiao Yu, Wenxuan Zhou, Jindong Gu, and Volker Tresp. 2025. FocalPO: Enhancing preference optimizing by focusing on correct preference rankings. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 256–267, Vienna, Austri… find
  18. Ilya Loshchilov and Frank Hutter. 2017. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations. find
  19. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734. find
  20. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  21. Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. 2024. Disentangling length from quality in direct preference optimization. arXiv preprint arXiv:2403.19159.
  22. Emmanouil Antonios Platanios, Otilia Stretcu, Graham Neubig, Barnabas Poczos, and Tom Mitchell. 2019. Competence-based curriculum learning for neural machine translation. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human langu… find
  23. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36.
  24. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 3505–3506. find
  25. Xin Wang, Yudong Chen, and Wenwu Zhu. 2021. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, 44(9):4555–4576. find
  26. Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation. arXiv preprint arXiv:2401.08417.
  27. Shu Zhou, Yuxuan Ao, Yunyang Xuan, Xin Wang, Tao Fan, and Hao Wang. 2026a. Inference scaling law for retrieval augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 40, pages 16522–16530. find
  28. Shu Zhou, Yufei Song, Jinman Leng, Xin Wang, Tao Fan, and Hao Wang. 2026b. Activation caching for retrieval-augmented generation. In Proceedings of the ACM Web Conference 2026, pages 8341–8344. find
  29. Shu Zhou, Yufei Song, Jinman Leng, Xin Wang, Tao Fan, and Hao Wang. 2026c. Cascaded verification framework: A progressive approach for mitigating hallucinations in large language models. In Proceedings of the ACM Web Conference 2026, pages 8333– 8336. find
  30. Shu Zhou, Xin Wang, Jingwen Qiu, Xiaomin Li, Bin Shi, and Hao Wang. 2025a. Losdf: a logical optimization and semantic decoupling framework for question answering in multi-party conversations. Information Processing & Management, 62(5):104200. find
  31. Corby Rosset, Ching-An Cheng, Arindam Mitra, Michael Santacroce, Ahmed Awadallah, and Tengyang Xie. 2024. Direct nash optimization: Teaching language models to self-improve with general preferences. arXiv preprint arXiv:2404.03715. find
  32. Shu Zhou, Yunyang Xuan, Yuxuan Ao, Xin Wang, Tao Fan, and Hao Wang. 2025b. Merit: Multi-agent collaboration for unsupervised time series representation learning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 24011–24028. find
  33. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  34. Shu Zhou, Rui Zhao, Zhengda Zhou, Haohan Yi, Xuhui Zheng, and Hao Wang. 2025c. Enhancing extractive question answering in multiparty dialogues with logical inference memory network. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8725–8738. find
  35. Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. 2017. Don’t decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489. find
  36. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008– 3021. find
  37. Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, and 1 others. 2023. Zephyr: Direct distillation of lm alignment. arXiv preprint arXiv:2310.16944. find
  38. Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, and Chenguang Zhu. 2024. Wpo: Enhancing rlhf with weighted preference optimization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. find
  39. Zhengda Zhou and Shu Zhou. 2025. Reasoning-guided prompt learning with historical knowledge injection for ancient chinese relation extraction. In CCF International Conference on Natural Language Processing and Chinese Computing, pages 172–184. Springer. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2026-dynamicfocalpo-adaptive-focusing-strategy-preference/