Elamur Research⌕
☰
home / papers / 2023 / Inverse Scaling: When Bigger Isn't Better

Inverse Scaling: When Bigger Isn't Better

Summary

Scaling laws predict that larger language models steadily lower training loss and usually improve on downstream tasks. This paper studies the reverse: tasks where accuracy falls as scale grows, called inverse scaling. To collect such cases systematically, the authors ran the Inverse Scaling Prize, a public two-round competition with cash awards.

Participants submitted text-completion datasets. These were tested zero-shot and few-shot on public and held-out model series from OpenAI, Anthropic and DeepMind, including instruction-tuned and RLHF variants. The held-out models guarded against tasks that only exploit the quirks of one model family.

Eleven tasks showed robust inverse scaling, often on problems that crowdworkers solved easily. From these tasks and earlier reports in the literature, the authors identify four causes:

  • strong memorized priors that override instructions,
  • imitation of undesirable training-data patterns,
  • easy distractor tasks,
  • misleading few-shot demonstrations.

The tasks also helped reveal U-shaped and inverted-U trends. This means extrapolating from small models is unreliable, and better training data and objectives are needed, not just more scale.

Key points

  • The contest received 99 submissions over two rounds. 11 winners were selected, and their datasets are released at inversescaling.com/data.
  • Models covered roughly 10^18 to 10^23 training FLOPs: GPT-2, GPT-3, OPT, Anthropic LM/Context Distilled/RLHF, GPT-3 FeedME, Gopher, Chinchilla, and PaLM.
  • Winning tasks include Memo Trap, Resisting Correction, Redefine, Prompt Injection, Modus Tollens, NeQA, Sig Figs, Pattern Match Suppression, Into the Unknown, Hindsight Neglect, and Repetitive Algebra.
  • Several tasks became U-shaped on PaLM, recovering at the largest scales. Prompt Injection showed inverted-U scaling, where performance first improved and then declined.
  • The prize structure offered up to ten $5,000 third prizes, five $20,000 second prizes, and a $100,000 grand prize.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Institutions

🏢GoogleMountain View · United States272 works · 255,532 citations🎓University of OxfordOxford · United Kingdom39 works · 153,213 citations🎓Johns Hopkins UniversityBaltimore · United States151 works · 84,111 citations🎓New York UniversityNew York · United States106 works · 82,819 citations🎓Boston UniversityBoston · United States7 works · 2,992 citations🏢AnthropicSan Francisco4 works · 1,415 citations

Cite

Cite as Ian R. McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, Andrew Gritsevskiy, Daniel Wurgaft, Derik Kauffman, Gabriel Recchia, Jiacheng Liu, Joe Cavanagh, Max Weiss, Sicong Huang, The Floating Droid, Tom Tseng, Tomasz Korbak, Xudong Shen, Yuhui Zhang, Zhengping Zhou, Najoung Kim, Samuel R. Bowman and Ethan Perez. 2023. Inverse Scaling: When Bigger Isn't Better. arXiv:2306.09479 · Transactions on Machine Learning Research (TMLR), 10/2023, https://openreview.net/forum?id=DwgRm72GQF.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.09479 · arXiv 2306.09479

@misc{mckenzie2023inverse,
  title = {{Inverse Scaling: When Bigger Isn't Better}},
  author = {Ian R. McKenzie, Alexander Lyzhov, Michael Pieler, Alicia Parrish, Aaron Mueller, Ameya Prabhu, Euan McLean, Aaron Kirtland, Alexis Ross, Alisa Liu, Andrew Gritsevskiy, Daniel Wurgaft, Derik Kauffman, Gabriel Recchia, Jiacheng Liu, Joe Cavanagh, Max Weiss, Sicong Huang, The Floating Droid, Tom Tseng, Tomasz Korbak, Xudong Shen, Yuhui Zhang, Zhengping Zhou, Najoung Kim, Samuel R. Bowman and Ethan Perez},
  year = {2023},
  doi = {10.48550/arxiv.2306.09479},
  eprint = {2306.09479},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.09479}
}

References

59 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. Introducing Claude. https://www.anthropic.com/index/introducing-claude, 2023. Accessed: 2023-03-15. ↗
  2. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova DasSarma, et al. A General Language Assistant as a Laboratory for Alignment, December 2021. URL http://arxiv.org/abs/2112.00861. arXiv:2112.00861 [cs]. ↗
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, et al. Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv preprint 2204.05862, 2022. URL https://arxiv.org/abs/2204. 05862. ↗
  4. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, pp. 610–623, New York, NY, USA, 2021. Associati… — [doi:10.1145/3442188.3445922](https://doi.org/10.1145/3442188.3445922) find
  5. Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the Opportunities and Risks of Foundation Models, July 2022. URL http://arxiv.org/abs/2108.07258. arXiv:2108.07258 [cs]. ↗ our card
  6. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, et al. Language Models are Few-Shot Learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing… find
  7. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf. ↗
  8. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramèr, and Chiyuan Zhang. Quantifying Memorization Across Neural Language Models. arXiv preprint 2202.07646, 2022. URL https://arxiv.org/abs/2202.07646. ↗
  9. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harrison Edwards, Yuri Burda, Nicholas Joseph, et al. Evaluating Large Language Models Trained on Code. arXiv preprint 2107.03374, 2021. URL https://arxiv.org/abs/2107.03374. ↗ our card
  10. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, et al. PaLM: Scaling Language Modeling with Pathways. arXiv preprint 2204.02311, 2022. URL https://arxiv.org/abs/2204.02311. ↗ our card
  11. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep Reinforcement Learning from Human Preferences. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/ d5e2c0adad503c… ↗
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum… our card
  13. Robert B. B. Durand, Fernando Patterson, and Corey Shank. Behavioral Biases in the NFL Gambling Market: Overreaction to News and the Recency Bias. SSRN Electronic Journal, 2021. doi: 10.2139/ssrn. 3861231. URL https://doi.org/10.2139/ssrn.3861231. — [doi:10.2139/ssrn](https://doi.org/10.2139/ssrn) ↗
  14. Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. Predictability and Surprise in Large Generative Models. In 2022 ACM Conference on Fairness, Accountability, and Transparency. ACM, jun 2022. doi: 10.1145/3… find
  15. Deep Ganguli, Amanda Askell, Nicholas Schiefer, Thomas I. Liao, Kamilė Lukošiūtė, Anna Chen, Anna Goldie, Azalia Mirhoseini, Catherine Olsson, Danny Hernandez, et al. The Capacity for Moral Self-Correction in Large Language Models, 2023. URL https://arxiv.org/abs/2302.07459. ↗ our card
  16. Leo Gao, John Schulman, and Jacob Hilton. Scaling Laws for Reward Model Overoptimization, October 2022. URL http://arxiv.org/abs/2210.10760. arXiv:2210.10760 [cs, stat]. ↗ our card
  17. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring Massive Multitask Language Understanding, January 2021. URL http://arxiv.org/abs/ 2009.03300. arXiv:2009.03300 [cs]. ↗ our card
  18. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, et al. Training Compute-Optimal Large Language Models. arXiv preprint 2203.15556, 2022. URL https://arxiv.org/abs/2203.15556. ↗ our card
  19. Evan Hubinger, Chris van Merwijk, Vladimir Mikulik, Joar Skalse, and Scott Garrabrant. Risks from Learned Optimization in Advanced Machine Learning Systems. arXiv preprint 1906.01820, 2021. URL https://arxiv.org/abs/1906.01820. — [arXiv:1906.01820](https://arxiv.org/abs/1906.01820) ↗
  20. Daniel Kahneman and Amos Tversky. On the Psychology of Prediction. Psychological Review, 80(4):237–251, July 1973. doi: 10.1037/h0034747. URL https://doi.org/10.1037/h0034747. — [doi:10.1037/h0034747](https://doi.org/10.1037/h0034747) ↗
  21. Nikhil Kandpal, Eric Wallace, and Colin Raffel. Deduplicating Training Data Mitigates Privacy Risks in Language Models. In ICML, 2022. URL https://proceedings.mlr.press/v162/kandpal22a/kandpal22a. pdf. ↗ our card
  22. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling Laws for Neural Language Models. arXiv:2001.08361 [cs, stat], January 2020. URL http://arxiv.org/abs/2001.08361. arXiv: 2001.08361. ↗ our card
  23. Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. Alignment of Language Agents, March 2021. URL http://arxiv.org/abs/2103.14659. arXiv:2103.14659 [cs]. ↗
  24. Najoung Kim, Tal Linzen, and Paul Smolensky. Uncontrolled Lexical Exposure Leads to Overestimation of Compositional Generalization in Pretrained Models, December 2022. URL http://arxiv.org/abs/ 2212.10769. arXiv:2212.10769 [cs]. — [arXiv:2212.10769](https://arxiv.org/abs/2212.10769) ↗
  25. Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, Christopher L. Buckley, Jason Phang, Samuel R. Bowman, and Ethan Perez. Pretraining Language Models with Human Preferences, February 2023. URL http://arxiv.org/abs/2302.08582. arXiv:2302.08582 [cs]. ↗ our card
  26. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. Holistic Evaluation of Language Models, November 2022. URL http://arxiv.org/abs/2211.09110. arXiv:2211.09110 [cs]. ↗
  27. Stephanie Lin, Jacob Hilton, and Owain Evans. TruthfulQA: Measuring How Models Mimic Human Falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214–3252, Dublin, Ireland, May 2022. Association for Computational Linguisti… our card
  28. Alex Lyzhov, Ian McKenzie, and Ethan Perez. Inverse Scaling Prize Ideas, Oct 2022. URL https://ethanperez.net/inverse-scaling-prize-ideas/. ↗
  29. Antonio Valerio Miceli-Barone, Fazl Barez, Ioannis Konstas, and Shay B. Cohen. The Larger They Are, the Harder They Fail: Language Models do not Recognize Identifier Swaps in Python, May 2023. URL http://arxiv.org/abs/2305.15507. arXiv:2305.15507 [cs]. ↗
  30. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381–2391, Brussels, Belgium, October-November 2018.… find
  31. Kanishka Misra, Julia Taylor Rayz, and Allyson Ettinger. COMPS: Conceptual Minimal Pair Sentences for testing Robust Property Knowledge and its Inheritance in Pre-trained Language Models, February 2023. URL http://arxiv.org/abs/2210.01963. arXiv:2210.01963 [cs]. ↗
  32. Richard Ngo, Lawrence Chan, and Sören Mindermann. The Alignment Problem From a Deep Learning Perspective, 2023. URL https://arxiv.org/abs/2209.00626. ↗
  33. Maxwell I. Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, et al. Show Your Work: Scratchpads for Intermediate Computation with Language Models. CoRR, abs/2112.00114, 2021. URL https://arxiv. org/abs/… our card
  34. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. In-context Learning and Induction Heads, 2022. find
  35. OpenAI. Model index for researchers. model-index-for-researchers, 2022. Accessed: 2023-04-06. https://platform.openai.com/docs/ ↗
  36. OpenAI. GPT-4 Technical Report, 2023. URL https://arxiv.org/abs/2303.08774. ↗ our card
  37. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback, March 2022. URL http://arxiv.org/abs/2203.02155. arXiv:2203.02155 [cs]. ↗ our card
  38. Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Comp… our card
  39. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: A Hand-Built Bias Benchmark for Question Answering. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 2086–2105, Dublin, Ireland, May 2022. As… our card
  40. Ethan Perez, Douwe Kiela, and Kyunghyun Cho. True Few-Shot Learning with Language Models. NeurIPS, 2021. URL https://arxiv.org/abs/2105.11447. ↗ our card
  41. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, et al. Discovering Language Model Behaviors with Model-Written Evaluations, 2022. URL https://arxiv.org/abs/2212.09251. ↗ our card
  42. Sundar Pichai. An important next step on our AI journey, 2023. URL https://blog.google/technology/ ai/bard-google-ai-search-updates/. ↗
  43. Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. Counterfactual Story Reasoning and Generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language P… find
  44. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners. Technical report, OpenAI, 2019. find
  45. Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, et al. Scaling Language Models: Methods, Analysis & Insights from Training Gopher. arXiv preprint 2112.11446, 2021. URL https://arxiv.org/abs/2112.11446. ↗
  46. John Schulman, Barret Zoph, Christina Kim, Jacob Hilton, Jacob Menick, Jiayi Weng, Juan Felipe Ceron Uribe, Liam Fedus, Luke Metza, Michael Pokorny, et al. Introducing ChatGPT. https://openai.com/ blog/chatgpt, 2022. Accessed: 2023-03-15. ↗
  47. Rohin Shah, Vikrant Varma, Ramana Kumar, Mary Phuong, Victoria Krakovna, Jonathan Uesato, and Zac Kenton. Goal Misgeneralization: Why Correct Specifications Aren’t Enough for Correct Goals, 2022. URL https://arxiv.org/abs/2210.01790. version 2. — [arXiv:2210.01790](https://arxiv.org/abs/2210.01790) ↗
  48. Irene Solaiman and Christy Dennison. Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P. S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 5861–5873. Curran Assoc… find
  49. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. arXiv preprint 2206.04615, 2022. URL ht… our card
  50. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to Summarize with Human Feedback. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems,… find
  51. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pp. 353–355, Brussel… our card
  52. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.… our card
  53. Tianlu Wang, Rohit Sridhar, Diyi Yang, and Xuezhi Wang. Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1719–1729, Seattle, United States, July 2022. Association for Computational L… find
  54. Peter Cathcart Wason. Reasoning about a Rule. Quarterly Journal of Experimental Psychology, 20(3): 273–281, 1968. doi: 10.1080/14640746808400161. URL https://journals.sagepub.com/doi/10.1080/ 14640746808400161. — [doi:10.1080/14640746808400161](https://doi.org/10.1080/14640746808400161) ↗
  55. Jason Wei, Najoung Kim, Yi Tay, and Quoc V. Le. Inverse Scaling Can Become U-shaped. Computing Research Repository, arXiv:2211.02011, 2022a. doi: 10.48550/ARXIV.2211.02011. URL https://arxiv.org/abs/2211.02011. ↗
  56. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent Abilities of Large Language Models, October 2022b. URL http://arxiv.org/abs/2206.07682. arXiv:2206.07682 [cs]. ↗ our card
  57. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of Thought Prompting Elicits Reasoning in Large Language Models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Proce… our card
  58. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, et al. OPT: Open Pre-trained Transformer Language Models. arXiv preprint 2205.01068, 2022. URL https://arxiv.org/abs/2205.01068. ↗ our card
  59. Yuhui Zhang, Michihiro Yasunaga, Zhengping Zhou, Jeff Z. HaoChen, James Zou, Percy Liang, and Serena Yeung. Beyond Positive Scaling: How Negation Impacts Scaling Trends of Language Models. In Findings of the Association for Computational Linguistics (ACL Findings), 2023. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mckenzie-2023-inverse-scaling-bigger-isn-t/