Elamur Research⌕
☰
home / papers / 2023 / Pretraining Language Models with Human Preferences

Pretraining Language Models with Human Preferences

Summary

Standard pretraining teaches language models to copy web text, including toxic language, leaked personal data and messy code. Most alignment work fixes this only after pretraining, but large models tend to retain what they learned. Removing bad data entirely also weakens the model. The authors propose pretraining with human feedback (PHF): a reward model scores each segment of the training corpus, and the training objective uses those scores. The model still sees undesirable text but is steered away from reproducing it. They compare MLE with five PHF objectives on three tasks: toxicity, PII and PEP8-style Python. Conditional training was the best option. It labels each segment with a control token and, at generation time, conditions on the "good" token. It cut undesirable outputs by up to tenfold, held up better under adversarial prompting, and kept capabilities close to MLE. It also clearly beat MLE pretraining followed by feedback finetuning. The work suggests preferences should shape models from the start of training rather than being added afterwards.

Key points

  • Six objectives were benchmarked: MLE (baseline), dataset filtering, unlikelihood training, reward-weighted regression (RWR), advantage-weighted regression (AWR) and conditional training. Conditional training used segment-level <|good|>/<|bad|> tokens with a reward threshold.
  • The three tasks were toxicity avoidance, avoiding personally identifiable information (PII) and PEP8-compliant Python generation. For each task, the misalignment score is the negated reward.
  • Conditional training was on the alignment–capabilities Pareto frontier for all three tasks. It stayed competitive with MLE on zero-shot benchmarks and after GLUE finetuning, and it was more robust under automated red-teaming following Perez et al. (2022).
  • Feedback finetuning of MLE models covered 1.6B or 330M tokens of a 3.3B-token budget, and PHF matched or beat it in every case. On PII, PHF reached a misalignment score of 0.0013 versus 0.0018 for 1.6B-token finetuning. A PHF model needed about ten red-teaming rounds to reach the score a finetuned model hit after one.
  • The paper appeared at ICML 2023. Code and datasets are at github.com/tomekkorbak/pretraining-with-human-feedback.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

94 papers of our library cite it (of the 140884 whose bibliographies we read); 16 of them have a page here, 78 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2302.08582 · arXiv 2302.08582

@misc{korbak2023pretraining,
  title = {{Pretraining Language Models with Human Preferences}},
  author = {Korbak, Tomasz and Shi, Kejian and Chen, Angelica and Bhalerao, Rasika and Buckley, Christopher L. and Phang, Jason and Bowman, Samuel R. and Perez, Ethan},
  year = {2023},
  doi = {10.48550/arxiv.2302.08582},
  eprint = {2302.08582},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.08582}
}

References

86 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Askell, A., Bai, Y., Chen, A., Drain, D., Ganguli, D., Henighan, T., Jones, A., Joseph, N., Mann, B., Das-Sarma, N., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Kernion, J., Ndousse, K., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., and Kaplan, J. A general language assi… find
  2. Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das-Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C.… find
  3. Bar-Haim, R., Dagan, I., Dolan, B., Ferro, L., and Giampiccolo, D. The second pascal recognising textual entailment challenge. Proceedings of the Second PASCAL Challenges Workshop on Recognising Textual Entailment, 01 2006. find
  4. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramer, F., and Zhang, C. Quantifying memorization across neural language models, 2022. URL https://arxiv.org/ abs/2202.07646. ↗
  5. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 1–14, Vancouver, Canada, August 2017. Associ… find
  6. Bengio, Y., Ducharme, R., Vincent, P., and Janvin, C. A neural probabilistic language model. J. Mach. Learn. Res., 3(null):1137–1155, mar 2003. ISSN 1532-4435. find
  7. Bentivogli, L., Magnini, B., Dagan, I., Dang, H. T., and Giampiccolo, D. The fifth PASCAL recognizing textual entailment challenge. In Proceedings of the Second Text Analysis Conference, TAC 2009, Gaithersburg, Maryland, USA, November 16-17, 2009. find
  8. NIST, 2009. URL https://tac.nist.gov/publications/ 2009/additional.papers/RTE5_overview. proceedings.pdf. ↗
  9. Chen, A., Scheurer, J., Korbak, T., Campos, J. A., Chan, J. S., Bowman, S. R., Cho, K., and Perez, E. Improving code generation by training with natural language feedback, 2023. find
  10. Borkan, D., Dixon, L., Sorensen, J., Thain, N., and Vasserman, L. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Proceedings of The 2019 World Wide Web Conference, WWW ’19, pp. 491–500, New York, NY, USA, 2019. Association for Computing Machinery.… — [doi:10.1145/3308560](https://doi.org/10.1145/3308560) find
  11. Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. Decision transformer: Reinforcement learning via sequence modeling. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Process… find
  12. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, Chen, M., Tworek, J., Jun, H., Yuan, Q., de Oliveira Pinto, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., H. P., Kaplan, J., E… find
  13. Carlini, N., Liu, C., Erlingsson, U., Kos, J., and Song, D. 2021b. The secret sharer: Evaluating and testing unintended memorization in neural networks. In Proceedings of Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., the 28th USENIX Conference on Security Symposium, Fedus, W., Li, Y., Wang,… find our card
  14. Dagan, I., Glickman, O., and Magnini, B. The pascal recognising textual entailment challenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW’05, pp. 177–190,… find
  15. Dai, N., Liang, J., Qiu, X., and Huang, X. Style transformer: Unpaired text style transfer without disentangled latent representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5997–6007, Florence, Italy, July 2019. Association for Computational… find our card
  16. Gao, L., Biderman, S., Black, S., Golding, L., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The Pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. find
  17. Gehman, S., Gururangan, S., Sap, M., Choi, Y., and Smith, N. A. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3356–3369, Online, November 2020. Association for Computational Linguistics. doi… find our card
  18. Giampiccolo, D., Magnini, B., Dagan, I., and Dolan, B. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pp. 1–9, Prague, June 2007. Association for Computational Linguistics. URL https://aclanthology.org/W07… ↗
  19. Dettmers, T. and Zettlemoyer, L. The case for 4-bit precision: k-bit inference scaling laws, 2023. find
  20. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and… find
  21. Go, D., Korbak, T., Kruszewski, G., Rozen, J., Ryu, N., and Dymetman, M. Aligning language models with preferences through f-divergence minimization, 2023. find
  22. Hanu, L. and Unitary team. Detoxify. https://github.com/unitaryai/detoxify, 2020. ↗
  23. Dolan, W. B. and Brockett, C. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL https://aclanthology.org/I05-5002. ↗ our card
  24. Emmons, S., Eysenbach, B., Kostrikov, I., and Levine, S. Rvs: What is essential for offline RL via supervised learning? In International Conference on Learning Representations, 2022. URL https://openreview.net/ forum?id=S874XAIpkR-. ↗
  25. Github. Henderson, P., Sinha, K., Angelard-Gontier, N., Ke, N. R., Fried, G., Lowe, R., and Pineau, J. Ethical challenges in data-driven dialogue systems. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, pp. 123–129, New York, NY, USA, 2018. Association for Comput… — [doi:10.1145/3278721.3278777](https://doi.org/10.1145/3278721.3278777) find
  26. Hendrycks, D., Mazeika, M., Kadavath, S., and Song, D. Using self-supervised learning can improve model robustness and uncertainty. Advances in Neural Information Processing Systems (NeurIPS), 2019. find
  27. Fan, A., Lewis, M., and Dauphin, Y. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889–898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/v1/P18-… find
  28. Hendrycks, D., Liu, X., Wallace, E., Dziedzic, A., Krishnan, R., and Song, D. Pretrained transformers improve out-of-distribution robustness. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2744–2751, Online, July 2020. Association for Computational Li… find our card
  29. Ficler, J. and Goldberg, Y. Controlling linguistic style aspects in neural language generation. In Proceedings of the Workshop on Stylistic Variation, pp. 94–104, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.18653/v1/W17-4912. URL https://aclanthology.org/W… ↗ our card
  30. Hewitt, J. Initializing new word embeddings for pretrained language models, 2021. find
  31. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J. W., and Sifre, L.… find
  32. K. (eds.), Advances in Neural Information Processing Systems, 2022a. URL https://openreview.net/ forum?id=XvI6h-s4un. ↗
  33. Korbak, T., Perez, E., and Buckley, C. RL with KL penalties is better viewed as Bayesian inference. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 1083–1091, Abu Dhabi, United Arab Emirates, December 2022b. Association for Computational Linguistics. URL https://aclanth… find
  34. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum? id=rygGQyrFvH. ↗
  35. Kumar, A., Peng, X. B., and Levine, S. Reward-conditioned policies, 2019. URL https://arxiv.org/abs/ 1912.13465. ↗
  36. Honnibal, M., Montani, I., Van Landeghem, S., and Boyd, A. spaCy: Industrial-strength Natural Language Processing in Python. 2020. doi: 10.5281/zenodo.1212303. find
  37. Jang, Y., Lee, J., and Kim, K.-E. GPT-critic: Offline reinforcement learning for end-to-end task-oriented dialogue systems. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=qaxhBG1UUaS. ↗
  38. Janner, M., Li, Q., and Levine, S. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021. find
  39. Jaques, N., Shen, J. H., Ghandeharioun, A., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, R. Human-centric dialog training via offline reinforcement learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3985–4003, Online, Novem… find
  40. Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., and Socher, R. Ctrl: A conditional transformer language model for controllable generation, 2019. URL https://arxiv.org/abs/1909.05858. ↗ our card
  41. Kumar, A., Zhou, A., Tucker, G., and Levine, S. Conservative q-learning for offline reinforcement learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546. find
  42. Levesque, H. J. The winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning. AAAI, 2011. URL http://dblp.uni-trier.de/db/conf/ aaaiss/aaaiss2011-6.html\#Levesque11. ↗
  43. Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020. URL https://arxiv.org/ abs/2005.01643. ↗
  44. Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. A diversity-promoting objective function for neural conversation models. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110–119, San Diego,… find our card
  45. Khalifa, M., Elsahar, H., and Dymetman, M. A distributional approach to controlled text generation. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. find
  46. OpenReview.net, 2021. URL https://openreview.net/ forum?id=jWkw45-9AbL. ↗
  47. Lin, S., Hilton, J., and Evans, O. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214– 3252, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 1… find our card
  48. Korbak, T., Elsahar, H., Kruszewski, G., and Dymetman, M. On reinforcement learning and distribution matching for fine-tuning language models with no catastrophic forgetting. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis,… find
  49. Lu, X., Welleck, S., Hessel, J., Jiang, L., Qin, L., West, P., Ammanabrolu, P., and Choi, Y. QUARK: Controllable text generation with reinforced unlearning. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL https://openreview.ne… ↗ our card
  50. Perez, E., Huang, S., Song, F., Cai, T., Ring, R., Aslanides, J., Glaese, A., McAleese, N., and Irving, G. Red teaming language models with language models, 2022. URL https://arxiv.org/abs/2202.03286. ↗ our card
  51. Peters, J. and Schaal, S. Reinforcement learning by reward-weighted regression for operational space control. In Proceedings of the 24th International Conference on Machine Learning, ICML ’07, pp. 745–750, New York, NY, USA, 2007. Association for Computing Machinery. ISBN 9781595937933. doi: 10. 114… — [doi:10.1145/1273496.1273590](https://doi.org/10.1145/1273496.1273590) find
  52. Menick, J., Trebacz, M., Mikulik, V., Aslanides, J., Song, F., Chadwick, M., Glaese, M., Young, S., Campbell-Gillingham, L., Irving, G., and McAleese, N. Teaching language models to support answers with verified quotes, 2022. URL https://arxiv.org/abs/ 2203.11147. ↗ our card
  53. Radford, A. and Narasimhan, K. Improving language understanding by generative pre-training. 2018. find
  54. Mikolov, T. and Zweig, G. Context dependent recurrent neural network language model. In 2012 IEEE Spoken Language Technology Workshop (SLT), pp. 234–239, 2012. doi: 10.1109/SLT.2012.6424228. — [doi:10.1109/slt.2012.6424228](https://doi.org/10.1109/slt.2012.6424228) find
  55. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., and Sutskever, I. Language models are unsupervised multitask learners. 2019. find
  56. Nair, A., Gupta, A., Dalal, M., and Levine, S. Awac: Accelerating online reinforcement learning with offline datasets, 2020. URL https://arxiv.org/abs/ 2006.09359. ↗
  57. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383– 2392, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 1… find our card
  58. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Gray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J., and Lowe, R. Training language models to follow instructions with h… find
  59. Ramasesh, V. V., Lewkowycz, A., and Dyer, E. Effect of scale on catastrophic forgetting in neural networks. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum? id=GhVS8_yPeEa. ↗
  60. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, N. Q., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume… find our card
  61. Sanh, V., Webson, A., Raffel, C., Bach, S., Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Raja, A., Dey, M., Bari, M. S., Xu, C., Thakker, U., Sharma, S. S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N., Datta, D., Chang, J., Jiang, M. T.-J., Wang, H., Manica, M., Shen, S., Yong, Z. X.,… find our card
  62. Peng, N., Ghazvininejad, M., May, J., and Knight, K. Towards controllable story generation. In Proceedings of the First Workshop on Storytelling, pp. 43–49, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/W18-1505. URL https://aclanthology.org/W18-1505. ↗
  63. Sap, M., Card, D., Gabriel, S., Choi, Y., and Smith, N. A. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1668–1678, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653… find our card
  64. Peng, X. B., Kumar, A., Zhang, G., and Levine, S. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning, 2019. URL https://arxiv.org/abs/1910.00177. ↗
  65. Scheurer, J., Campos, J. A., Chan, J. S., Chen, A., Cho, K., and Perez, E. Training language models with language feedback, 2022. URL https://arxiv.org/abs/ 2204.14146. ↗
  66. Vu, T., Barua, A., Lester, B., Cer, D., Iyyer, M., and Constant, N. Overcoming catastrophic forgetting in zero-shot cross-lingual generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9279–9300, Abu Dhabi, United Arab Emirates, December 2022. Asso… find
  67. Scheurer, J., Campos, J. A., Korbak, T., Chan, J. S., Chen, A., Cho, K., and Perez, E. Training language models with language feedback at scale, 2023. find
  68. Schmidhuber, J. Reinforcement learning upside down: Don’t predict rewards – just map them to actions, 2019. URL https://arxiv.org/abs/1912.02875. — [arXiv:1912.02875](https://arxiv.org/abs/1912.02875) ↗
  69. Snell, C., Kostrikov, I., Su, Y., Yang, M., and Levine, S. Offline rl for natural language generation with implicit language q learning, 2022. URL https://arxiv.org/abs/2206.11871. — [arXiv:2206.11871](https://arxiv.org/abs/2206.11871) ↗
  70. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631– 1642, Seattle, Washington, USA, Oct… find
  71. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353– 355, Brussels, Belgium, Nove… find our card
  72. Wang, B., Ping, W., Xiao, C., Xu, P., Patwary, M., Shoeybi, M., Li, B., Anandkumar, A., and Catanzaro, B. Exploring the limits of domain-adaptive training for detoxifying large-scale language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processi… find
  73. Solaiman, I. and Dennison, C. Process for adapting language models to society (PALMS) with values-targeted datasets. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openreview.net/ forum?id=k-ghaB9VZBw. ↗
  74. Warstadt, A., Singh, A., and Bowman, S. R. Neural network acceptability judgments. arXiv preprint arXiv:1805.12471, 2018. find
  75. Welbl, J., Glaese, A., Uesato, J., Dathathri, S., Mellor, J., Hendricks, L. A., Anderson, K., Kohli, P., Coppin, B., and Huang, P.-S. Challenges in detoxifying language models. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2447–2469, Punta Cana, Dominican Republic, No… find our card
  76. Tang, R., Lu, Y., Liu, L., Mou, L., Vechtomova, O., and Lin, J. J. Distilling task-specific knowledge from bert into simple neural networks. ArXiv, abs/1903.12136, 2019. — [arXiv:1903.12136](https://arxiv.org/abs/1903.12136) find
  77. Tay, Y., Wei, J., Chung, H. W., Tran, V. Q., So, D. R., Shakeri, S., Garcia, X., Zheng, H. S., Rao, J., Chowdhery, A., Zhou, D., Metzler, D., Petrov, S., Houlsby, N., Le, Q. V., and Dehghani, M. Transcending scaling laws with 0.1 URL https://arxiv.org/abs/2210.11399. Welleck, S., Kulikov, I., Roller… ↗
  78. Tunstall, L., von Werra, L., and Wolf, T. Natural Language Processing with Transformers: Building Language Applications with Hugging Face. O’Reilly Media, Incorporated, 2022. ISBN 1098103246. URL https://books.google.ch/books?id=7hhyzgEACAAJ. van Rossum, G., Warsaw, B., and Coghlan, N. Style guide… ↗
  79. Villalobos, P., Sevilla, J., Heim, L., Besiroglu, T., Hobbhahn, M., and Ho, A. Will we run out of data? an analysis of the limits of scaling datasets in machine learning, 2022. URL https://arxiv.org/abs/2211.04325. ↗
  80. Williams, A., Nangia, N., and Bowman, S. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112… find
  81. Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Le Scao, T., Gugger, S., Drame, M., Lhoest, Q., and Rush, A. Transformers: State-of-the-art natural lan… find our card
  82. Xu, A., Pathak, E., Wallace, E., Gururangan, S., Sap, M., and Klein, D. Detoxifying language models risks marginalizing minority voices. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2390–2397, O… find
  83. Xu, J., Ju, D., Li, M., Boureau, Y.-L., Weston, J., and Dinan, E. Recipes for safety in open-domain chatbots, 2020. URL https://arxiv.org/abs/2010.07079. ↗
  84. Zhu, Y., Lu, S., Zheng, L., Guo, J., Zhang, W., Wang, J., and Yu, Y. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1097–1100, 2018. find our card
  85. Ziegler, D., Nix, S., Chan, L., Bauman, T., Schmidt-Nielsen, P., Lin, T., Scherlis, A., Nabeshima, N., Weinstein-Raun, B., de Haas, D., Shlegeris, B., and Thomas, N. Adversarial training for high-stakes reliability. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Info… find
  86. Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/korbak-2023-pretraining-language-models-human-preferences/