Elamur Research⌕
☰
home / papers / 2026 / Offline Preference Optimization via Maximum Marginal Likelihood Estimation

Offline Preference Optimization via Maximum Marginal Likelihood Estimation

Paper · 2026 · Saeed Najafi and Alona Fyshe

Summary

The authors tackle LLM alignment with human preferences without the heavy RLHF machinery of a learned reward model plus PPO. Even DPO, which drops the reward model, still inherits an entropy-promoting term from the RLHF objective. The paper introduces MMPO, which treats alignment as Maximum Marginal Likelihood estimation. It maximizes the marginal log-likelihood of a desirable output and approximates the marginalization using just the two responses in each offline preference pair.

When this objective is computed with a stable log-sum-exp, its gradient becomes a weighted update. The chosen response is pushed up more strongly than the rejected one, and the weights are sigmoids of their score gap. No explicit reward or entropy bonus is needed.

Experiments cover models from 135M to 8B parameters and compare MMPO with DPO and SimPO. MMPO is less sensitive to the β hyperparameter. It reaches comparable or better AlpacaEval-2 win rates and erodes general abilities on the LM Harness benchmark less than the baselines. This makes MMPO a simple, robust alternative for offline alignment.

Key points

  • Core theory (Theorem 3.2): on a pair (x, z_w, z_l), the MMPO gradient breaks into two log-probability gradients, one for each response. The preferred one is scaled by σ(s_w − s_l), the rejected one by σ(s_l − s_w). The scores s combine the policy log-probability, the reward and the β-weighted SFT reference log-probability.
  • Implementation: written in PyTorch on top of torch.logsumexp. It follows the coding conventions of the TRL library, which the DPO and SimPO baselines also use.
  • Ablations:
    • Adding token-level entropy maximization consistently hurt results on both LM Harness and AlpacaEval-2.
    • Removing the DPO-style auxiliary loss or in-batch normalization lowered peak LM Harness accuracy but slightly raised AlpacaEval-2 win rates. This points to a trade-off between the two benchmarks.
  • Baseline observation: the authors could not reproduce SimPO's reported edge. In every one of their runs, a well-tuned DPO beat SimPO.
  • Limitations:
    • Models above 1B parameters were trained only with memory-saving tricks such as LoRA or 8-bit optimizers, because GPU resources were limited.
    • Online variants of MMPO and comparisons with PPO or GRPO are left for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AlbertaEdmonton · Canada44 works · 3,236 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Saeed Najafi and Alona Fyshe. 2026. Offline Preference Optimization via Maximum Marginal Likelihood Estimation. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.318

@inproceedings{najafi2026offline,
  title = {{Offline Preference Optimization via Maximum Marginal Likelihood Estimation}},
  author = {Saeed Najafi and Alona Fyshe},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.318},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.318}
}

References

40 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. 2024. Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms. Preprint, arXiv:2402.14740. find
  2. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Gabriel Martín Blázquez, Guilherme Penedo, Lewis Tunstall, Andrés Marafioti, Hynek Kydlíček, Agustín Piqueres Lajarín, Vaibhav Srivastav, Joshua Lochner, Caleb Fahlgren, Xuan-Son Nguyen, Clémentine Fourrier, Ben Burtenshaw, Hugo Larcher, Haojun Zhao, C… — SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language Model
  3. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computation…
  4. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2019. Piqa: Reasoning about physical commonsense in natural language. Preprint, arXiv:1911.11641.
  5. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457.
  6. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. Preprint, arXiv:2310.01377. find
  7. Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 2022. 8-bit optimizers via block-wise quantization. Preprint, arXiv:2110.02861.
  8. Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. RAFT: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767.
  9. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. find
  10. Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. 2020. Implementation matters in deep policy gradients: A case study on ppo and trpo. Preprint, arXiv:2005.12729. find
  11. Leo Gao, Jonathan Tow, Bena Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Gold, Jeffrey Hsu, Kyle Leahy, et al. 2021. A framework for evaluating language models. In 2021 NeurIPS Workshop on Datasets and Benchmarks. find
  12. Kelvin Guu, Panupong Pasupat, Evan Liu, and Percy Liang. 2017. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1051–1062, Vancouver, C…
  13. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.
  14. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  15. Andrew Ilyas, Logan Engstrom, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. 2020. A closer look at deep policy gradients. Preprint, arXiv:1811.02553. — [arXiv:1811.02553](https://arxiv.org/abs/1811.02553) find
  16. Wouter Kool, Herke van Hoof, and Max Welling. 2019. Buy 4 REINFORCE samples, get a baseline for free! find
  17. Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785– 794, Copenhagen, Denmark. Association for Computational Lin…
  18. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval. ↗
  19. Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023. Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orca/OpenOrca. find
  20. Ilya Loshchilov and Frank Hutter. 2017. Sgdr: Stochastic gradient descent with warm restarts. Preprint, arXiv:1608.03983. find
  21. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. Preprint, arXiv:1711.05101. find
  22. Yu Meng, Mengzhou Xia, and Danqi Chen. 2024. Simpo: Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems (NeurIPS). find
  23. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391, Brussels, Belgium. Association fo… find
  24. Saeed Najafi and Alona Fyshe. 2024. RIFF: Learning to rephrase inputs for few-shot fine-tuning of language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1447–1466, Bangkok, Thailand. Association for Computational Linguistics. find
  25. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Trai… — Training language models to follow instructions with human feedback
  26. Andrew Patterson, Samuel Neumann, Martha White, and Adam White. 2024. Empirical design in reinforcement learning. Preprint, arXiv:2304.01315. — [arXiv:2304.01315](https://arxiv.org/abs/2304.01315) find
  27. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023a. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference on Neural Information Processing Systems.
  28. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023b. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc.
  29. Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In KDD, pages 3505– 3506. find
  30. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. Preprint, arXiv:1907.10641. find
  31. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2018. High-dimensional continuous control using generalized advantage estimation. Preprint, arXiv:1506.02438. find
  32. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. Preprint, arXiv:1707.06347. find
  33. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. Preprint, arXiv:2402.03300.
  34. Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  35. Yunhao Tang, Zhaohan Daniel Guo, Zeyu Zheng, Daniele Calandriello, R’emi Munos, Mark Rowland, Pierre Harvey Richemond, Michal Valko, Bernardo ’Avila Pires, and Bilal Piot. 2024. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749. — [arXiv:2402.05749](https://arxiv.org/abs/2402.05749) find
  36. Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen. 2023. Beyond reverse kl: Generalizing direct preference optimization with diverse divergence constraints. arXiv preprint arXiv:2309.16240. — [arXiv:2309.16240](https://arxiv.org/abs/2309.16240) find
  37. Junxiong Wang, Daniele Paliotta, Avner May, Alexander M Rush, and Tri Dao. 2024. The mamba in the llama: Distilling and accelerating hybrid models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  38. Teng Xiao, Yige Yuan, Huaisheng Zhu, Mingxiao Li, and Vasant G. Honavar. 2024. Cal-dpo: Calibrated direct preference optimization for language model alignment. arXiv preprint arXiv:2412.14516. — [arXiv:2412.14516](https://arxiv.org/abs/2412.14516) find
  39. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. RRHF: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302.
  40. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? Preprint, arXiv:1905.07830. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/najafi-2026-offline-preference-optimization-maximum-marginal/