Elamur Research⌕
☰
home / papers / 2021 / Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets

Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets

Paper · 2021 · Irene Solaiman and Christy Dennison OpenAI)

Summary

Large pretrained language models such as GPT-3 can produce biased, toxic, or culturally inappropriate text. Retraining them from scratch for every desired behaviour is impractical, so the authors propose PALMS. PALMS is an iterative recipe for steering a pretrained model toward a chosen set of values. The steps are:

  • select sensitive topic categories;
  • write position statements describing the desired stance on each;
  • author question-answer prompts with carefully written completions;
  • fine-tune the model on this small values-targeted dataset;
  • evaluate it, then add new examples wherever weaknesses appear.

The values in this work are grounded in U.S. and international human-rights norms and are offered as an illustrative template, not a universal standard. The values-targeted models were compared with untouched base models and with control models tuned on neutral book and Wikipedia text. They scored better on human ratings of value adherence, on Perspective API toxicity, and on co-occurrence analyses of descriptive words. General capabilities stayed roughly intact, and the gains were largest for the biggest models. The work shows that a tiny, hand-curated dataset can meaningfully shift model behaviour, which makes human oversight of alignment practical.

Key points

  • The values-targeted dataset has only 80 prompt–completion pairs: 70 on broad topics and 10 aimed at weak categories. A professional writer produced the completions, each 40–340 words long, organised under eight sensitive-topic categories.
  • The control dataset contains 100 snippets from a private books/Wikipedia corpus. Their lengths are matched to the values-targeted data, which isolates the effect of the data's stance from the effect of its quality.
  • Experiments cover GPT-3 sizes from 125M to 175B parameters. PALMS has a growing effect with scale, which the authors read as suggesting that larger models need fewer examples.
  • The evaluation sets use five weakness-targeting prompts per category, with three samples per prompt per model (200 tokens, temperature 0.7). Each sample was scored by Perspective API and by blind Surge.AI raters.
  • Of the 40 test prompts, 34 had no close counterpart in training, which indicates generalisation. The work is limited to U.S. English and question-answer formats.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢OpenAISan Francisco · United States23 works · 27,874 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Irene Solaiman and Christy Dennison OpenAI). 2021. Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets. arXiv:2106.10328.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2106.10328 · arXiv 2106.10328

@misc{solaiman2021process,
  title = {{Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets}},
  author = {Irene Solaiman and Christy Dennison OpenAI)},
  year = {2021},
  doi = {10.48550/arxiv.2106.10328},
  eprint = {2106.10328},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2106.10328}
}

References

39 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 18 U.S.C. § 2241. Chapter 109a sexual abuse. URL https://uscode.house.gov/view. xhtml?path=/prelim@title18/part1/chapter109A&edition=prelim. ↗
  2. 18 U.S.C. § 2332b. U.S. Code 2332b - Acts of terrorism transcending national boundaries. URL https://www.law.cornell.edu/uscode/text/18/2332b. ↗
  3. A. Abid, M. Farooqi, and J. Zou. Persistent anti-muslim bias in large language models. CoRR, abs/2101.05783, 2021. URL https://arxiv.org/abs/2101.05783. ↗
  4. E. M. Bender, T. Gebru, A. McMillan-Major, and S. Shmitchell. On the dangers of stochastic parrots: Can language models be too big? . In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610623, New York, NY, USA, 2021. Association for Computing Ma… — [doi:10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922) find
  5. S. L. Blodgett, S. Barocas, H. D. III, and H. M. Wallach. Language (technology) is power: A critical survey of "bias" in NLP. CoRR, abs/2005.14050, 2020. URL https://arxiv.org/abs/2005.14050. ↗
  6. T. Bolukbasi, K. Chang, J. Y. Zou, V. Saligrama, and A. Kalai. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. CoRR, abs/1607.06520, 2016. URL http://arxiv.org/abs/1607.06520. ↗
  7. G. Branwen. GPT-3 Creative Fiction, 2021. URL https://www.gwern.net/GPT-3. ↗
  8. T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Ber… — Language Models are Few-Shot Learners
  9. CCal. Bus. & Prof. Code § 2052. California Legislative Information California Penal Code 2052. URL https://leginfo.legislature.ca.gov/faces/codes_ displaySection.xhtml?sectionNum=2052.&lawCode=BPC. ↗
  10. Centers for Disease Control and Prevention. Public Health Professionals Gateway vaccination laws. URL https://www.cdc.gov/phlp/publications/topic/ vaccinationlaws.html. ↗
  11. Children’s Bureau. Child Welfare Information Gateway state laws on child abuse and neglect, . URL https://www.childwelfare.gov/topics/systemwide/laws-policies/ can/. ↗
  12. Children’s Bureau. Child Welfare Information Gateway mandatory reporters of child abuse and neglect, . URL https://www.childwelfare.gov/pubPDFs/manda.pdf. ↗
  13. J. Dhamala, T. Sun, V. Kumar, S. Krishna, Y. Pruksachatkun, K. Chang, and R. Gupta. BOLD: dataset and metrics for measuring biases in open-ended language generation. CoRR, abs/2101.11718, 2021. URL https://arxiv.org/abs/2101.11718. — [arXiv:2101.11718](https://arxiv.org/abs/2101.11718) ↗
  14. L. Dixon, J. Li, J. Sorensen, N. Thain, and L. Vasserman. Measuring and mitigating unintended bias in text classification. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’18, page 6773, New York, NY, USA, 2018. Association for Computing Machinery. ISBN 9781450360128.… — [doi:10.1145/3278721.3278729](https://doi.org/10.1145/3278721.3278729) find
  15. Federal Bureau of Investigation. Reports and Publications terrorism 2002/2005. URL https://www.fbi.gov/stats-services/publications/terrorism-2002-2005. ↗
  16. Federal Trade Commission. No FEAR Act protections against discrimination and other prohibited practices, 2021. URL https://www.ftc.gov/site-information/ no-fear-act/protections-against-discrimination. ↗
  17. S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. CoRR, abs/2009.11462, 2020. URL https://arxiv.org/abs/2009.11462. ↗
  18. S. Gururangan, A. Marasovic, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith. Don’t stop pretraining: Adapt language models to domains and tasks. CoRR, abs/2004.10964, 2020. URL https://arxiv.org/abs/2004.10964. ↗
  19. A. Holtzman, J. Buys, M. Forbes, and Y. Choi. The curious case of neural text degeneration. CoRR, abs/1904.09751, 2019. URL http://arxiv.org/abs/1904.09751. ↗
  20. D. M. Jr., V. Prabhakaran, J. Kuhlberg, A. Smart, and W. S. Isaac. Extending the machine learning abstraction boundary: A complex systems approach to incorporate societal context. CoRR, abs/2006.09663, 2020. URL https://arxiv.org/abs/2006. 09663. — [arXiv:2006.09663](https://arxiv.org/abs/2006.09663) ↗
  21. Z. Kenton, T. Everitt, L. Weidinger, I. Gabriel, V. Mikulik, and G. Irving. Alignment of language agents. CoRR, abs/2103.14659, 2021. URL https://arxiv.org/abs/2103. 14659. ↗
  22. P. P. Liang, I. M. Li, E. Zheng, Y. C. Lim, R. Salakhutdinov, and L. Morency. Towards debiasing sentence representations. CoRR, abs/2007.08100, 2020. URL https://arxiv.org/abs/2007.08100. ↗
  23. A. Liu, M. Sap, X. Lu, S. Swayamdipta, C. Bhagavatula, N. A. Smith, and Y. Choi. On-the-fly controlled text generation with experts and anti-experts, 2021. find
  24. S. Mohamed, M. Png, and W. Isaac. Decolonial AI: decolonial theory as sociotechnical foresight in artificial intelligence. CoRR, abs/2007.04068, 2020. URL https://arxiv.org/abs/2007.04068. — [arXiv:2007.04068](https://arxiv.org/abs/2007.04068) ↗
  25. National Conference of State Legislatures. States With Religious and Philosophical Exemptions From School Immunization Requirements. URL https://www.ncsl.org/ research/health/school-immunization-exemption-state-laws.aspx. ↗
  26. A. Paullada, I. D. Raji, E. M. Bender, E. Denton, and A. Hanna. Data and its (dis)contents: A survey of dataset development and use in machine learning research. CoRR, abs/2012.05345, 2020. URL https://arxiv.org/abs/2012.05345. ↗
  27. Perspective. Perspective how it works, 2021. URL https://www.perspectiveapi.com/how-it-works. ↗
  28. Perspective. Perspective Developers attributes & languages, 2021. URL https://support.perspectiveapi.com/s/about-the-api-attributes-and-languages. ↗
  29. V. Sanh, T. Wolf, Y. Belinkov, and A. M. Rush. Learning from others’ mistakes: Avoiding dataset biases without modeling them. CoRR, abs/2012.01300, 2020. URL https://arxiv.org/abs/2012.01300. — [arXiv:2012.01300](https://arxiv.org/abs/2012.01300) ↗
  30. M. Sap, D. Card, S. Gabriel, Y. Choi, and N. A. Smith. The risk of racial bias in hate speech detection. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1668–1678, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v…
  31. Surge AI. Human Intelligence. On Demand., 2021. URL https://surgehq.ai. ↗
  32. United Nations. human-rights. Human Rights. URL https://www.un.org/en/global-issues/ ↗
  33. United Nations Office of the High Commissioner. Human Rights enhancing equality and countering discrimination, 2021. URL https://www.ohchr.org/EN/AboutUs/ ManagementPlan/Pages/equality.aspx. ↗
  34. U.S. Department of Health & Human Services. Statutory Rape: A Guide to State Laws and Reporting Requirements. Sexual Intercourse with Minors. URL https://aspe.hhs.gov/report/ statutory-rape-guide-state-laws-and-reporting-requirements-summary-current-state-laws/ sexual-intercourse-minors. ↗
  35. U.S. Department of Justice. The United States Department of Justice elder abuse and elder financial exploitation statutes. URL https://www.justice.gov/elderjustice/ prosecutors/statutes. ↗
  36. U.S. Food & Drug Administration. Buying & Using Medicine Safely medication health fraud. URL https://www.fda.gov/drugs/buying-using-medicine-safely/ medication-health-fraud. ↗
  37. B. Vidgen, A. Harris, D. Nguyen, R. Tromble, S. Hale, and H. Margetts. Challenges and frontiers in abusive content detection. In Proceedings of the Third Workshop on Abusive Language Online, pages 80–93, Florence, Italy, Aug. 2019. Association for Computational Linguistics. doi: 10.18653/v1/W19-3509…
  38. A. Xu, E. Pathak, E. Wallace, S. Gururangan, M. Sap, and D. Klein. Detoxifying language models risks marginalizing minority voices. CoRR, abs/2104.06390, 2021. URL https://arxiv.org/abs/2104.06390. ↗
  39. J. Xu, D. Ju, M. Li, Y. Boureau, J. Weston, and E. Dinan. Recipes for safety in open-domain chatbots. CoRR, abs/2010.07079, 2020. URL https://arxiv.org/abs/2010. 07079. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/solaiman-2021-process-adapting-language-models-society/