Elamur Research⌕
☰
home / papers / 2023 / Aligning Language Models with Preferences through f-divergence Minimization

Aligning Language Models with Preferences through f-divergence Minimization

Paper · 2023 · Dongyoung Go, Tomasz Korbak, German Kruszewski et al.

Summary

The paper frames language-model alignment as fitting a model to a target distribution that encodes desired behaviour. It observes that earlier methods tie a particular target to a particular divergence. RLHF, which uses RL with a KL penalty, implicitly minimizes reverse KL to an implied target. GDC instead trains with DPG, which minimizes forward KL to an explicit energy-based target. The authors propose f-DPG, which can approximate any target that can be evaluated pointwise using any member of the f-divergence family. They derive a general gradient formula, add variance-reducing baselines and support conditional targets. They compare four variants (KL-DPG, RKL-DPG, TV-DPG, JS-DPG) across thirteen tasks. These include lexical constraints, gender and religion bias reduction, factual consistency in summarization and code compilability. No single divergence wins everywhere. Reverse KL favours alignment at the cost of diversity, while forward KL does the reverse. Jensen-Shannon usually gives the best balance and often beats forward KL even when judged by forward KL. This shows that the choice of divergence is an important design decision in alignment.

Key points

  • f-DPG unifies GDC/DPG and RLHF (RL with KL penalties) as special cases of f-divergence minimization toward an EBM target.
  • Evaluation covers 13 alignment tasks, 3 target types (binary, scalar, distributional preferences), 4 divergences and 8 metrics.
  • On the Pareto frontier, RKL-DPG shows high alignment but collapsed diversity, while KL-DPG shows the opposite. JS-DPG consistently lands on the frontier.
  • In scaling experiments from GPT-2 small up to xl (1.5B parameters), the alignment score rises with size, but the ranking between divergences stays the same.
  • An analysis of pseudo-rewards explains the behaviour. Forward KL is sensitive to sampling noise, reverse KL penalizes harshly, and TV becomes noisy near the optimum. JS gives smooth rewards in both directions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Dongyoung Go, Tomasz Korbak, German Kruszewski, Jos Rozen, Nahyeon Ryu and Marc Dymetman. 2023. Aligning Language Models with Preferences through f-divergence Minimization. arXiv:2302.08215.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2302.08215 · arXiv 2302.08215

@misc{go2023aligning,
  title = {{Aligning Language Models with Preferences through f-divergence Minimization}},
  author = {Dongyoung Go, Tomasz Korbak, German Kruszewski, Jos Rozen, Nahyeon Ryu and Marc Dymetman},
  year = {2023},
  doi = {10.48550/arxiv.2302.08215},
  eprint = {2302.08215},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.08215}
}

References

98 references, 38 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Proc. of NeurIPS, 20…
  2. Arjovsky, M., Chintala, S., and Bottou, L. Wasserstein generative adversarial networks. In Precup, D. and Teh, Y. W. (eds.), Proc. of ICML, volume 70 of Proceedings of Machine Learning Research, pp. 214–223. PMLR, 2017. URL http://proceedings.mlr.press/ v70/arjovsky17a.html. ↗
  3. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., DasSarma, N., et al. A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861, 2021. URL https://arxiv.org/abs/2112.00861. ↗
  4. Bahdanau, D., Brakel, P., Xu, K., Goyal, A., Lowe, R., Pineau, J., Courville, A. C., and Bengio, Y. An actor-critic algorithm for sequence prediction. In Proc. of ICLR. OpenReview.net, 2017. URL https://openreview.net/forum?id=SJDaqqveg. ↗
  5. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C.… find
  6. Cao, Y., Sotnikova, A., Daumé III, H., Rudinger, R., and Zou, L. Theory-grounded measurement of U.S. social stereotypes in English language models. In Proc. of NAACL-HLT, pp. 1276–1295, Seattle, United States, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.naacl-main.92. URL…
  7. Che, T., Li, Y., Jacob, A. P., Bengio, Y., and Li, W. Mode regularized generative adversarial networks. In Proc. of ICLR. OpenReview.net, 2017. URL https://openreview.net/forum?id=HJKkY35le. ↗
  8. Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fedus, W., Li, E., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finetuned language models. ArXiv preprint, abs/2210.11416, 2022. URL https://arxiv.org/abs/2210.11416. ↗
  9. Dathathri, S., Madotto, A., Lan, J., Hung, J., Frank, E., Molino, P., Yosinski, J., and Liu, R. Plug and play language models: A simple approach to controlled text generation. In Proc. of ICLR. OpenReview.net, 2020. URL https://openreview.net/forum? id=H1edEyBKDS. ↗
  10. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. ArXiv preprint, abs/2204.05862, 2022b. URL https://arxiv.org/ abs/2204.05862. ↗
  11. Dohan, D., Xu, W., Lewkowycz, A., Austin, J., Bieber, D., Lopes, R. G., Wu, Y., Michalewski, H., Saurous, R. A., Sohl-Dickstein, J., et al. Language model cascades. ArXiv preprint, abs/2207.10342, 2022. URL https://arxiv.org/abs/2207.10342. ↗
  12. Baxter, J. and Bartlett, P. L. Infinite-horizon policy-gradient estimation. Journal of Artificial Intelligence Research, 15:319–350, 2001. find
  13. Eikema, B., Kruszewski, G., Dance, C. R., Elsahar, H., and Dymetman, M. An approximate sampler for energy-based models with divergence diagnostics. Transactions of Machine Learning Research, 2022. URL https://openreview.net/forum?id=VW4IrC0n0M. ↗
  14. Berger, A. L., Della Pietra, S. A., and Della Pietra, V. J. A maximum entropy approach to natural language processing. Computational Linguistics, 22(1):39–71, 1996. URL https://aclanthology.org/J96-1002. ↗
  15. Black, S., Gao, L., Wang, P., Leahy, C., and Biderman, S. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, 2021. URL https://doi.org/10.5281/zenodo.5297715. — [doi:10.5281/zenodo.5297715](https://doi.org/10.5281/zenodo.5297715) ↗
  16. Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural story generation. In Proc. of ACL, pp. 889–898, Melbourne, Australia, 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-1082. URL https://aclanthology.org/P18-1082. ↗
  17. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. RealToxic… — RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models
  18. (eds.), Proc. of ICML, volume 70 of Proceedings of Machine Learning Research, pp. 1645–1654. PMLR, 2017. URL http://proceedings.mlr.press/ v70/jaques17a.html. ↗
  19. Ghasemipour, S. K. S., Zemel, R., and Gu, S. A divergence minimization perspective on imitation learning methods. In Kaelbling, L. P., Kragic, D., and Sugiura, K. (eds.), Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 1259–1277. PMLR, 202… find
  20. Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C., Lapedriza, À., Jones, N., Gu, S., and Picard, R. W. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. ArXiv preprint, abs/1907.00456, 2019. URL https://arxiv.org/ abs/1907.00456. — [arXiv:1907.00456](https://arxiv.org/abs/1907.00456) ↗
  21. Kappen, H. J., Gómez, V., and Opper, M. Optimal control as a graphical model inference problem. Machine learning, 87(2):159–182, 2012. find
  22. Glaese, A., McAleese, N., Trebacz, M., Aslanides, J., Firoiu, V., Ewalds, T., Rauh, M., Weidinger, L., Chadwick, M., Thacker, P., Campbell-Gillingham, L., Uesato, J., Huang, P., Comanescu, R., Yang, F., See, A., Dathathri, S., Greig, R., Chen, C., Fritz, D., Elias, J. S., Green, R., Mokrá, S., Fern… — Improving alignment of dialogue agents via targeted human judgements
  23. Kappen, H. J., Gómez, V., and Opper, M. Optimal control as a graphical model inference problem. In Borrajo, D., Kambhampati, S., Oddi, A., and Fratini, S. (eds.), Proceedings of the Twenty-Third International Conference on Automated Planning and Scheduling, ICAPS 2013, Rome, Italy, June 10-14, 2013… find
  24. AAAI, 2013. URL http://www.aaai.org/ocs/index.php/ ICAPS/ICAPS13/paper/view/6012. ↗
  25. Ke, L., Choudhury, S., Barnes, M., Sun, W., Lee, G., and Srinivasa, S. S. Imitation learning as f-divergence minimization. In LaValle, S. M., Lin, M., Ojala, T., Shell, D. A., and Yu, J. (eds.), Algorithmic Foundations of Robotics XIV, Proceedings of the Fourteenth Workshop on the Algorithmic Founda… find
  26. Springer, 2021. doi: 10.1007/978-3-030-66723-8\\ 19. URL https://doi.org/10.1007/978-3-030-66723-8_19. — [doi:10.1007/978-3-030-66723-8\](https://doi.org/10.1007/978-3-030-66723-8\) ↗
  27. Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Commun. ACM, 63(11): 139–144, 2020. ISSN 0001-0782. doi: 10.1145/3422622. URL https://doi.org/10.1145/3422622. — [doi:10.1145/3422622](https://doi.org/10.1145/3422622) ↗
  28. Goyal, K., Dyer, C., and Berg-Kirkpatrick, T. Exposing the implicit energy networks behind masked language models via metropolis–hastings. In Proc. of ICLR. OpenReview.net, 2022. URL https://openreview.net/ forum?id=6PvWo1kEvlT. ↗
  29. Khalifa, M., Elsahar, H., and Dymetman, M. A distributional approach to controlled text generation. In Proc. of ICLR. OpenReview.net, 2021. URL https://openreview.net/forum?id=jWkw45-9AbL. ↗
  30. HF Canonical Model Maintainers. distilbert-base-uncased-finetuned-sst-2-english (revision bfdd146), 2022. URL https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english. ↗
  31. Kingma, D. P. and Ba, J. Adam: A method for stochastic optimization. In Bengio, Y. and LeCun, Y. (eds.), Proc. of ICLR, 2015. URL http://arxiv.org/abs/1412. 6980. ↗
  32. Hiriart-Urruty, J.-B. and Lemaréchal, C. Convex analysis and minimization algorithms I: Fundamentals, volume 305. Springer science & business media, 2013. find
  33. Huszar, F. How (not) to train your generative model: Scheduled sampling, likelihood, adversary? ArXiv preprint, abs/1511.05101, 2015. URL https://arxiv.org/ abs/1511.05101. — [arXiv:1511.05101](https://arxiv.org/abs/1511.05101) ↗
  34. Korbak, T., Elsahar, H., Kruszewski, G., and Dymetman, M. Controlling conditional language models without catastrophic forgetting. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proc. of ICML, volume 162 of Proceedings of Machine Learning Research, pp. 11499… find
  35. Jaques, N., Gu, S., Bahdanau, D., Hernández-Lobato, J. M., Turner, R. E., and Eck, D. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In Precup, D. and Teh, Y. W. Korbak, T., Elsahar, H., Kruszewski, G., and Dymetman, M. On reinforcement learning and distribu… — TruthfulQA: Measuring How Models Mimic Human Falsehoods
  36. Liu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., and Pineau, J. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. In Proc. of EMNLP, pp. 2122–2132, Austin, Texas, 2016. Association for Computational Linguis…
  37. Korbak, T., Perez, E., and Buckley, C. L. RL with KL penalties is better viewed as bayesian inference. CoRR, abs/2205.11275, 2022c. doi: 10.48550/arXiv. 2205.11275. URL https://doi.org/10.48550/ arXiv.2205.11275. find
  38. Maas, A. L., Daly, R. E., Pham, P. T., Huang, D., Ng, A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Proc. of ACL, pp. 142–150, Portland, Oregon, USA, 2011. Association for Computational Linguistics. URL https://aclanthology.org/P11-1015. ↗
  39. Lebret, R., Grangier, D., and Auli, M. Neural text generation from structured data with application to the biography domain. In Proc. of EMNLP, pp. 1203–1213, Austin, Texas, 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16-1128. URL https://aclanthology.org/D16-1128. ↗
  40. Maynez, J., Narayan, S., Bohnet, B., and McDonald, R. On faithfulness and factuality in abstractive summarization. In Proc. of ACL, pp. 1906–1919, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.acl-main.173. URL https://aclanthology.org/2020.acl-main.173. ↗
  41. LeCun, Y., Chopra, S., Hadsell, R., Ranzato, M., and Huang, F. A tutorial on energy-based learning. Predicting structured data, 1(0), 2006. find
  42. Levine, S. Reinforcement learning and control as probabilistic inference: Tutorial and review. ArXiv preprint, abs/1805.00909, 2018. URL https://arxiv.org/ abs/1805.00909. — [arXiv:1805.00909](https://arxiv.org/abs/1805.00909) ↗
  43. Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., and McAleese, N. Teaching language models to support answers with verified quotes, 2022. URL https://arxiv.org/abs/ 2203.11147. ↗
  44. Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. A diversity-promoting objective function for neural conversation models. In Proc. of NAACL-HLT, pp. 110–119, San Diego, California, 2016a. Association for Computational Linguistics. doi: 10.18653/v1/N16-1014. URL https://aclanthology.org/N16-1… ↗
  45. Mescheder, L. M., Geiger, A., and Nowozin, S. Which training methods for gans do actually converge? In Dy, J. G. and Krause, A. (eds.), Proc. of ICML, volume 80 of Proceedings of Machine Learning Research, pp. 3478– 3487. PMLR, 2018. URL http://proceedings.mlr.press/v80/mescheder18a.html. ↗
  46. Li, J., Monroe, W., Ritter, A., Jurafsky, D., Galley, M., and Gao, J. Deep reinforcement learning for dialogue generation. In Proc. of EMNLP, pp. 1192–1202, Austin, Texas, 2016b. Association for Computational Linguistics. doi: 10.18653/v1/D16-1127. URL https://aclanthology.org/D16-1127. ↗
  47. Liese, F. and Vajda, I. On divergences and informations in statistics and information theory. IEEE Transactions on Information Theory, 52(10):4394–4412, 2006. find
  48. Lin, C.-Y. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74–81, Barcelona, Spain, 2004. Association for Computational Linguistics. URL https://aclanthology.org/W04-1013. ↗
  49. Miao, N., Zhou, H., Mou, L., Yan, R., and Li, L. CGMH: constrained sentence generation by metropolis-hastings sampling. In The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI… find
  50. AAAI Press, 2019. doi: 10.1609/ aaai.v33i01.33016834. URL https://doi.org/10. 1609/aaai.v33i01.33016834. find
  51. Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D., and Riedmiller, M. A. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013. URL http://arxiv.org/ abs/1312.5602. ↗
  52. Lin, S., Hilton, J., and Evans, O. TruthfulQA: Measuring how models mimic human falsehoods. In Proc. of ACL, pp. 3214–3252, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18653/v1/
  53. Nallapati, R., Zhou, B., dos Santos, C., Gulçehre, Ç., and Xiang, B. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pp. 280–290, Berlin, Germany, 2016. Association for Computational…
  54. Pasunuru, R. and Bansal, M. Reinforced video captioning with entailment rewards. In Proc. of EMNLP, pp. 979–985, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10.18653/v1/D17-1103. URL https://aclanthology.org/D17-1103. ↗
  55. Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. Pytorch: An imperative style, high-perfo… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  56. Nan, F., Nallapati, R., Wang, Z., Nogueira dos Santos, C., Zhu, H., Zhang, D., McKeown, K., and Xiang, B. Entity-level factual consistency of abstractive text summarization. In Proc. of EACL, pp. 2727–2733, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.eacl-main.235.…
  57. Ngo, H., Raterink, C., Araújo, J. G., Zhang, I., Chen, C., Morisot, A., and Frosst, N. Mitigating harm in language models with conditional-likelihood filtration. ArXiv preprint, abs/2108.07790, 2021. URL https://arxiv.org/abs/2108.07790. — [arXiv:2108.07790](https://arxiv.org/abs/2108.07790) ↗
  58. Norouzi, M., Bengio, S., Chen, Z., Jaitly, N., Schuster, M., Wu, Y., and Schuurmans, D. Reward augmented maximum likelihood for neural structured prediction. In Lee, D. D., Sugiyama, M., von Luxburg, U., Guyon, I., and Garnett, R. (eds.), Proc. of NeurIPS, pp. 1723– 1731, 2016. URL https://proceedin… find
  59. Nowozin, S., Cseke, B., and Tomioka, R. f-gan: Training generative neural samplers using variational divergence minimization. In Lee, D. D., Sugiyama, M., von Luxburg, U., Guyon, I., and Garnett, R. (eds.), Proc. of NeurIPS, pp. 271–279, 2016. URL https://proceedings.neurips.cc/paper/2016/hash/ ced… ↗
  60. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Gray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with h… — Training language models to follow instructions with human feedback
  61. Paulus, R., Xiong, C., and Socher, R. A deep reinforced model for abstractive summarization. In Proc. of ICLR. OpenReview.net, 2018. URL https://openreview.net/forum?id=HkAClQgA-. ↗
  62. Peters, J. and Schaal, S. Reinforcement learning by reward-weighted regression for operational space control. In Ghahramani, Z. (ed.), Proc. of ICML, volume 227 of ACM International Conference Proceeding Series, pp. 745–750. ACM, 2007. doi: 10.1145/ 1273496.1273590. URL https://doi.org/10. 1145/1273… find
  63. Polyanskiy, Y. f -divergences, 2019. https://people.lids.mit.edu/yp/ homepage/data/LN_fdiv.pdf. ↗
  64. URL Qin, L., Welleck, S., Khashabi, D., and Choi, Y. Cold decoding: Energy-based constrained text generation with langevin dynamics. ArXiv preprint, abs/2202.11705, 2022. URL https://arxiv.org/abs/2202. 11705. — [arXiv:2202.11705](https://arxiv.org/abs/2202.11705) ↗
  65. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  66. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P. J., et al. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67, 2020.
  67. Parshakova, T., Andreoli, J.-M., and Dymetman, M. Distributional reinforcement learning for energy-based sequential models. ArXiv preprint, abs/1912.08517, 2019. URL https://arxiv.org/abs/1912.08517. — [arXiv:1912.08517](https://arxiv.org/abs/1912.08517) ↗
  68. Ranzato, M., Chopra, S., Auli, M., and Zaremba, W. Sequence level training with recurrent neural networks. In Bengio, Y. and LeCun, Y. (eds.), Proc. of ICLR, 2016. URL http://arxiv.org/abs/1511.06732. ↗
  69. Raychev, V., Bielik, P., and Vechev, M. Probabilistic model for code with decision trees. ACM SIGPLAN Notices, 51 (10):731–747, 2016. find
  70. Agarwal, A., Power, A., Ray, A., Warstadt, A., Kocurek, A. W., Safaya, A., Tazarv, A., Xiang, A., Parrish, A., Nie, A., Hussain, A., Askell, A., Dsouza, A., Rahane, A., Iyer, A. S., Andreassen, A., Santilli, A., Stuhlmüller, A., Dai, A. M., La, A., Lampinen, A. K., Zou, A., Jiang, A., Chen, A., Vuo… — Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
  71. Rockafellar, R. T. Convex analysis, volume 18. Princeton university press, 1970. find
  72. Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., and Weston, J. Recipes for building an open-domain chatbot. In Proc. of EACL, pp. 300–325, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.eacl-main.24. UR…
  73. Sason, I. On f-divergences: Integral representations, local behavior, and inequalities. Entropy, 20(5):383, 2018. find
  74. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D. M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. Learning to summarize from human feedback. In Proc. of NeurIPS, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546. find
  75. Sason, I. and Verdú, S. f -divergence inequalities. IEEE Transactions on Information Theory, 62(11):5973–6006, 2016. find
  76. Tambwekar, P., Dhuliawala, M., Martin, L. J., Mehta, A., Harrison, B., and Riedl, M. O. Controllable neural story plot generation via reward shaping. In Kraus, S. (ed.), Proc. of IJCAI, pp. 5982–5988. ijcai.org, 2019. doi: 10. 24963/ijcai.2019/829. URL https://doi.org/10. 24963/ijcai.2019/829. find
  77. Saunders, W., Yeh, C., Wu, J., Bills, S., Ouyang, L., Ward, J., and Leike, J. Self-critiquing models for assisting human evaluators. ArXiv preprint, abs/2206.05802, 2022. URL https://arxiv.org/abs/2206.05802. ↗
  78. Scheurer, J., Campos, J. A., Chan, J. S., Chen, A., Cho, K., and Perez, E. Training language models with natural language feedback. ArXiv preprint, abs/2204.14146, 2022. URL https://arxiv.org/abs/2204.14146. ↗
  79. Theis, L., van den Oord, A., and Bethge, M. A note on the evaluation of generative models. In Bengio, Y. and LeCun, Y. (eds.), Proc. of ICLR, 2016. URL http://arxiv.org/abs/1511.01844. — [arXiv:1511.01844](https://arxiv.org/abs/1511.01844) ↗
  80. Schulman, J., Moritz, P., Levine, S., Jordan, M. I., and Abbeel, P. High-dimensional continuous control using generalized advantage estimation. In Bengio, Y. and LeCun, Y. (eds.), Proc. of ICLR, 2016. URL http://arxiv.org/abs/1506.02438. ↗
  81. Thoppilan, R., Freitas, D. D., Hall, J., Shazeer, N., Kulshreshtha, A., Cheng, H., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y., Lee, H., Zheng, H. S., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhou, Y., Chang, C., Kr… — LaMDA: Language Models for Dialog Applications
  82. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. ArXiv preprint, abs/1707.06347, 2017. URL https://arxiv.org/abs/1707.06347. ↗
  83. Sheng, E., Chang, K.-W., Natarajan, P., and Peng, N. The woman worked as a babysitter: On biases in language generation. In Proc. of EMNLP, pp. 3407–3412, Hong Kong, China, 2019. Association for Computational Linguistics. doi: 10.18653/v1/D19-1339. URL https://aclanthology.org/D19-1339. ↗
  84. Solaiman, I. and Dennison, C. Process for adapting language models to society (palms) with values-targeted datasets. Proc. of NeurIPS, 34:5861–5873, 2021. find
  85. Todorov, E. Linearly-solvable markov decision problems. In Schölkopf, B., Platt, J. C., and Hofmann, T. (eds.), Proc. of NeurIPS, pp. 1369–1376. MIT Press, 2006a. URL — Bot-Adversarial Dialogue for Safe Conversational Agents https://proceedings.neurips.cc/paper/2006/hash/ Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown,… ↗
  86. Todorov, E. Linearly-solvable markov decision problems. In Schölkopf, B., Platt, J. C., and Hofmann, T. (eds.), Proc. of NeurIPS, pp. 1369–1376. MIT Press, 2006b. URL https://proceedings.neurips.cc/paper/2006/hash/ d806ca13ca3449af72a1ea5aedbed26a-Abstract.html. ↗
  87. Wang, D., Liu, H., and Liu, Q. Variational inference with tail-adaptive f-divergence. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Proc. of NeurIPS, pp. 5742– 5752, 2018. URL https://proceedings.neurips.cc/paper/2018/hash/ 1cd138d0499a68f4bb7… ↗
  88. Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P., Cheng, M., Glaese, M., Balle, B., Kasirzadeh, A., Kenton, Z., Brown, S., Hawkins, W., Stepleton, T., Biles, C., Birhane, A., Haas, J., Rimell, L., Hendricks, L. A., Isaac, W. S., Legassick, S., Irving, G., and Gabriel, I. Ethic… — Ethical and social risks of harm from Language Models
  89. Zelikman, E., Wu, Y., Mu, J., and Goodman, N. STar: Bootstrapping reasoning with reasoning. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Proc. of NeurIPS, 2022. URL https://openreview.net/ forum?id=_3ELRdg2sgI. ↗
  90. Zhao, J., Khashabi, D., Khot, T., Sabharwal, A., and Chang, K.-W. Ethical-advice taker: Do language models understand natural language interventions? In Findings of ACL, pp. 4158–4164, Online, 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021.findings-acl. 364. URL https://aclan… find
  91. Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., and Yu, Y. Texygen: A benchmarking platform for text generation models. In Collins-Thompson, K., Mei, Q., Davison, B. D., Liu, Y., and Yilmaz, E. (eds.), Proc. of SIGIR, pp. 1097–1100. ACM, 2018. doi: 10. 1145/3209978.3210080. URL https://doi…
  92. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. ArXiv preprint, abs/1909.08593, 2019. URL https://arxiv.org/ abs/1909.08593. ↗
  93. Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., and Huang, P.-S. Challenges in detoxifying language models. In Findings of EMNLP, pp. 2447–2469, Punta Cana, Dominican Republic, 2021. Association for Computational Linguistics. doi:…
  94. Ziegler, D. M., Nix, S., Chan, L., Bauman, T., Schmidt-Nielsen, P., Lin, T., Scherlis, A., Nabeshima, N., Weinstein-Raun, B., de Haas, D., Shlegeris, B., and Thomas, N. Adversarial training for high-stakes reliability. CoRR, abs/2205.01663, 2022. doi: 10.48550/arXiv. 2205.01663. URL https://doi.org/… — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) find
  95. Williams, R. J. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3):229–256, 1992. find
  96. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State-of-the-art natural lan… — Transformers: State-of-the-Art Natural Language Processing
  97. Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., and Dinan, E. Bot-adversarial dialogue for safe conversational agents. In Proc. of NAACL-HLT, pp. 2950–2968, Online, 2021. Association for Computational Linguistics. doi:
  98. A. Complements on Formal Aspects and Proofs A.1. Equivalent definitions for f -divergences find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/go-2023-aligning-language-models-preferences-through/