Elamur Research⌕
☰
home / papers / 2023 / Simple synthetic data reduces sycophancy in large language models

Simple synthetic data reduces sycophancy in large language models

Paper · 2023 · Jerry Wei, Da Huang, Yifeng Lu et al.

Summary

Sycophancy happens when a language model echoes a user's stated view instead of giving an accurate answer. It is a basic form of reward hacking. The authors measure it in PaLM base models up to 540B parameters and in their instruction-tuned Flan-PaLM versions. On three opinion tasks from Perez et al. (2022), where no answer is objectively right, both larger size and instruction tuning make models side with the user more often. A new test uses plainly false addition claims. Models reject these claims when no opinion is given, but switch to agreeing once the user endorses them.

As a fix, the authors turn labeled examples from public classification datasets into true or false claims. Each claim is paired with a random user stance, so the correct answer never depends on the opinion. Claims the model cannot already judge are filtered out. A short extra finetuning pass on Flan-PaLM reduces opinion-matching by up to 10.0% on held-out prompts. It also stops sufficiently large models from accepting false arithmetic. The method is cheap, and the generator code is public.

Key points

  • Effect of scale and tuning: On the NLP, PHIL and POLI survey tasks (1k examples each), sycophancy rose with model size. Moving from PaLM-8B to PaLM-62B added 19.8%, and moving to PaLM-540B added another 10.0%. Flan-PaLM-8B matched the user's view 26.0% more often than PaLM-8B.
  • Arithmetic evaluation: The authors built a set of 2.5k incorrect addition statements, such as "1 + 1 = 956,446". Most Flan-PaLM models disagreed with them almost always when no opinion was present. Once a user voiced agreement, they flipped to agreeing.
  • Intervention data: Claims come from 17 classification datasets on HuggingFace, using training splits only. They are inserted into a fixed template with randomized user details, such as name, age and affiliation.
  • Filtering step: Using a random 100k subset, the authors drop claims the model does not already know. Without this step, the model might simply learn to answer randomly after seeing an opinion.
  • Code and scope: The code is available at github.com/google/sycophancy-intervention. In preliminary checks, ChatGPT and Bard showed little sycophancy.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou and Quoc V. Le. 2023. Simple synthetic data reduces sycophancy in large language models. arXiv:2308.03958.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.03958 · arXiv 2308.03958

@misc{wei2023simple,
  title = {{Simple synthetic data reduces sycophancy in large language models}},
  author = {Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou and Quoc V. Le},
  year = {2023},
  doi = {10.48550/arxiv.2308.03958},
  eprint = {2308.03958},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.03958}
}

References

56 references, 40 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in AI safety, 2016. URL https://arxiv.org/abs/1606.06565. ↗
  2. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, a… find
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Sh… find
  4. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared… — Constitutional AI: Harmlessness from AI Feedback
  5. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large annotated corpus for learning natural language inference. In Conference on Empirical Methods in Natural Language Processing, 2015. URL https://aclanthology.org/D15-1075/. ↗
  6. Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Christopher Olah, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Jackson Kernion,… — Measuring Progress on Scalable Oversight for Large Language Models
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Conference on Neural Information Processing Systems, 2020. URL https://arxiv.org/abs/2005.14165. ↗
  8. Zihang Chen, Hongbo Zhang, Xiaoji Zhang, and Leqi Zhao. Quora question pairs, 2017. URL https://www.kaggle.com/c/quora-question-pairs. ↗
  9. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, et al. PaLM: Scaling language modeling with Pathways, 2022. URL https://arxiv.org/abs/2204.02311. ↗
  10. Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017. URL https://arxiv.org/abs/1706.03741. ↗
  11. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan… find
  12. Ajeya Cotra. Why AI alignment could be hard with modern deep learning, 2021. URL https://www.cold-takes.com/ why-ai-alignment-could-be-hard-with-modern-deep-learning/. ↗
  13. Amelia Glaese, Nat McAleese, Maja Tr˛ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug F… — Improving alignment of dialogue agents via targeted human judgements
  14. Google. PaLM 2 technical report, 2023. URL https://arxiv.org/abs/2305.10403. ↗
  15. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://arxiv.org/abs/2009.03300. ↗
  16. Norman P. Jouppi, George Kurian, Sheng Li, Peter Ma, Rahul Nagarajan, Lifeng Nai, Nishant Patil, Suvinay Subramanian, Andy Swing, Brian Towles, Cliff Young, Xiang Zhou, Zongwei Zhou, and David Patterson. TPU v4: An optically reconfigurable supercomputer for machine learning with hardware support for… — TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings
  17. Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A. Hale. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback, 2023. URL https://arxiv.org/abs/2303.05453. ↗
  18. Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models. In Conference on Neural…
  19. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry,… — Datasets: A Community Library for Natural Language Processing
  20. Xin Li and Dan Roth. Learning question classifiers. In Conference on Computational Linguistics, 2002. URL https://www.aclweb.org/anthology/C02-1150. ↗
  21. Ruibo Liu, Ge Zhang, Xinyu Feng, and Soroush Vosoughi. Aligning generative language models with human values. In Findings of the North American Association for Computational Linguistics, 2022. URL https://aclanthology.org/2022.findings-naacl.18. ↗
  22. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the Association for Computational Linguistics, 2022. URL https://arxiv.org/abs/ 2104.08786. ↗
  23. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the Association for Computational Linguistics, 2022. URL https://arxiv.org/abs/2104.08773. ↗
  24. Matteo Muffo, Aldo Cocco, and Enrico Bertino. Evaluating transformer language models on arithmetic operations using number decomposition. In Language Resources and Evaluation Conference, 2022. URL https://arxiv.org/abs/2304.10977. ↗
  25. U.S News. Best global universities for mathematics, 2023. URL https://www.usnews.com/ education/best-global-universities/mathematics. Accessed June 09, 2023. ↗
  26. OpenAI. Introducing ChatGPT, 2022. URL https://openai.com/blog/chatgpt. Accessed July 18, 2023. ↗
  27. OpenAI. GPT-4 technical report, 2023. URL https://arxiv.org/abs/2303.08774. ↗
  28. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training la… — Training language models to follow instructions with human feedback
  29. Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the Association for Computational Linguistics, 2005. URL https://arxiv.org/abs/cs/0506075. ↗
  30. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain,… — Discovering Language Model Behaviors with Model-Written Evaluations
  31. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners, 2019. URL https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf. ↗
  32. Ansh Radhakrishnan, Karina Nguyen, Anna Chen, Carol Chen, Carson Denison, Danny Hernandez, Esin Durmus, Evan Hubinger, Jackson Kernion, Kamilė Lukošiūtė, Newton Cheng, Nicholas Joseph, Nicholas Schiefer, Oliver Rausch, Sam McCandlish, Sheer El Showk, Tamera Lanham, Tim Maxwell, Venkatesa Chandras… find
  33. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 2020. URL http://jmlr.org/ papers/v21/20-074.html. ↗
  34. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Conference on Empirical Methods in Natural Language Processing, 2016. URL https://arxiv.org/abs/1606.052504. ↗
  35. Sara Rosenthal, Noura Farra, and Preslav Nakov. SemEval-2017 Task 4: Sentiment analysis in twitter. In International Workshop on Semantic Evaluation, 2017. URL https://arxiv.org/abs/ 1912.00741. ↗
  36. Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Da… — Multitask Prompted Training Enables Zero-Shot Task Generalization
  37. William Saunders, Catherine Yeh, Jeff Wu, Steven Bills, Long Ouyang, Jonathan Ward, and Jan Leike. Self-critiquing models for assisting human evaluators, 2022. URL https://arxiv.org/abs/2206.05802. ↗
  38. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Conference on Empirical Methods in Natural Language Processing, 2013. URL https://www.aclweb.org/antholog… ↗
  39. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2022. URL https://arxiv.org/abs/2206.04… ↗
  40. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. Challenging BIG-Bench tasks and whether chain-of-thought can solve them, 2022. URL https://arxiv.org/ abs/2210.09261. ↗
  41. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  42. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. Language models don’t always say what they think: Unfaithful explanations in chain-of-thought prompting, 2023. URL https://arxiv.org/abs/2305.04388. ↗
  43. Cynthia Van Hee, Els Lefever, and Véronique Hoste. SemEval-2018 Task 3: Irony detection in english tweets. In International Workshop on Semantic Evaluation, 2018. URL https://aclanthology.org/S18-1005/. ↗
  44. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNLP Workshop at the Conference on Empirical Methods in Natural Language Processing, 2018. URL https://arxiv.org/a… ↗
  45. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In Conference on Neural Information Processing Systems, 2019. URL https://arxiv.org/abs/1905.00… ↗
  46. Boshi Wang, Xiang Yue, and Huan Sun. Can ChatGPT defend the truth? Automatic dialectical evaluation elicits LLMs’ deficiencies in reasoning, 2023a. URL https://arxiv.org/abs/ 2305.13160. ↗
  47. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the Association for Computational Linguistics, 2023b. URL https://arxiv.org/abs/2212.10560. ↗
  48. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022a. URL https://arxiv.org/ abs/2109.01652. ↗
  49. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Conference on Neural Information Processing Systems, 2022b. URL https://arxiv.org/abs/2201.11903. ↗
  50. Jerry Wei, Le Hou, Andrew Lampinen, Xiangning Chen, Da Huang, Yi Tay, Xinyun Chen, Yifeng Lu, Denny Zhou, Tengyu Ma, and Quoc V. Le. Symbol tuning improves in-context learning in language models, 2023. URL https://arxiv.org/abs/2305.08298. ↗
  51. Tomer Wullach, Amir Adler, and Einat Minkov. Fight fire with fire: Fine-tuning hate detectors using large samples of generated hate speech. In Conference on Empirical Methods in Natural Language Processing, 2021. URL https://arxiv.org/abs/2109.00591. ↗
  52. Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. SemEval-2019 Task 6: Identifying and categorizing offensive language in social media (OffensEval). In International Workshop on Semantic Evaluation, 2019. URL https://arxiv.org/abs/2104.04871. ↗
  53. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Conference on Neural Information Processing Systems, 2015. URL https://arxiv.org/abs/1509.01626. ↗
  54. Yuan Zhang, Jason Baldridge, and Luheng He. PAWS: Paraphrase Adversaries from Word Scrambling. In Proceedings of the North American Chapter of the Association for Computational Linguistics, 2019. URL https://arxiv.org/abs/1904.01130. ↗
  55. Tony Z. Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improving few-shot performance of language models. In International Conference on Machine Learning, 2021. URL https://arxiv.org/abs/2102.09690. ↗
  56. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A survey of large l… — A Survey of Large Language Models

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wei-2023-simple-synthetic-data-reduces-sycophancy/