Elamur Research⌕
☰
home / papers / 2024 / How predictable is language model benchmark performance?

How predictable is language model benchmark performance?

Paper · 2024 · David Owen

Summary

David Owen (Epoch) asks whether a large language model's benchmark scores can be forecast before it is trained, a question relevant to anticipating capabilities, automation and policy. The study covers eleven recent model families, including GPT-3, PaLM, PaLM-2 and GPT-4, spanning about five orders of magnitude of training compute, and evaluates them on BIG-Bench, BIG-Bench Hard (BBH) and MMLU. The method has two steps. First, the Chinchilla scaling law estimates each model's loss from parameter count and training tokens, and that loss is re-expressed as equivalent "scaled compute." Second, simple curves are fitted to performance against this quantity and tested by holding out the largest models. Aggregate scores turn out to be reasonably predictable, with roughly 6 pp mean error when extrapolating BBH one order of magnitude ahead. Single tasks are much noisier but still beat chance and naive baselines. The work therefore supports compute-based forecasting of broad capabilities while warning that specific skills remain hard to anticipate.

Key points

  • Loss is estimated with the Chinchilla scaling law and expressed as scaled compute. The fitted forms are sigmoid, sigmoid with offset, clipped linear, and a constant-average baseline, all fitted by constrained nonlinear least squares and back-tested on held-out points.
  • Extrapolating one order of magnitude in compute gives about 6 pp mean absolute error on BBH, compared with about 18 pp (17.8 pp) on individual BIG-Bench tasks.
  • Per-task errors are fat-tailed: the median is 8.6 pp, but a few tasks miss badly. Errors also tend to be underestimates when early models scored poorly.
  • Prediction is worse for multiple-choice tasks (116 tasks) than for exact-string-match tasks (39 tasks).
  • Once data scaling and the S-shaped fit are accounted for, the apparent mismatch between GPT-3 and PaLM largely disappears. Remaining limitations include inconsistent task subsets across models and training-set sizes that span only one order of magnitude.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksEvaluation & benchmarks · 1,394 works

Cite

Cite as David Owen. 2024. How predictable is language model benchmark performance?. arXiv:2401.04757.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.04757 · arXiv 2401.04757

@misc{owen2024how,
  title = {{How predictable is language model benchmark performance?}},
  author = {David Owen},
  year = {2024},
  doi = {10.48550/arxiv.2401.04757},
  eprint = {2401.04757},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.04757}
}

References

34 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. [1] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 20…
  2. [2] Tom Davidson. What a compute-centric framework says about AI takeoff speeds - a draft report. Open Philanthropy, 2023. find
  3. [3] Lukas Finnveden. Extrapolating GPT-N performance. Alignment Forum, 2020. find
  4. [4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  5. [5] Lukas Finnveden. PaLM in "Extrapolating GPT-N performance". Alignment Forum, 2022. find
  6. [6] Lukas Finnveden. GPT-4 and PaLM-2 in "Extrapolating GPT-N performance". Alignment Forum, 2023. find
  7. [7] Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXiv:2206.07682, 2022.
  8. [10] Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. Predictability and surprise in large generative models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1747–1764, 2022.
  9. [11] Ibrahim M Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. Revisiting neural scaling laws in language and vision. Advances in Neural Information Processing Systems, 35:22300–22312, 2022. find
  10. [12] Ethan Caballero, Kshitij Gupta, Irina Rish, and David Krueger. Broken neural scaling laws. arXiv preprint arXiv:2210.14891, 2022. — [arXiv:2210.14891](https://arxiv.org/abs/2210.14891) find
  11. [13] Matthew Barnett. Twitter thread on retrodicting GPT-4 MMLU performance. Twitter, 2023. find
  12. [14] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022.
  13. [15] Nostalgebraist. Chinchilla’s wild implications. Alignment Forum, 2022. find
  14. [16] Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261–272, 2020. — SciPy 1.0 — Fundamental Algorithms for Scientific Computing in Python
  15. [17] Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. arXiv preprint arXiv:2210.09261, 2022.
  16. [18] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  17. [19] OpenAI. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  18. [8] Alyssa Vance. Is AI Progress Impossible to Predict? LessWrong, 2022. find
  19. [20] Gemini technical report. Technical report, Google DeepMind, 2023. find
  20. [9] Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. Are emergent abilities of large language models a mirage? arXiv preprint arXiv:2304.15004, 2023.
  21. [21] Ege Erdil and Tamay Besiroglu. Algorithmic progress in computer vision. arXiv preprint arXiv:2212.05153, 2022. — [arXiv:2212.05153](https://arxiv.org/abs/2212.05153) find
  22. [22] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. PaLM 2 technical report. arXiv preprint arXiv:2305.10403, 2023.
  23. [23] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  24. [24] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311.12022, 2023.
  25. [25] Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world GitHub issues? arXiv preprint arXiv:2310.06770, 2023.
  26. [26] Epoch. Parameter, compute and data trends in machine learning. https://epochai.org/data/pcd, 2022. ↗
  27. [27] Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446, 2021. find
  28. [28] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. BloombergGPT: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023.
  29. [29] Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. GPT-NeoX-20B: An open-source autoregressive language model. arXiv preprint arXiv:2204.06745, 2022.
  30. [30] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
  31. [31] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. BLOOM: A 176B-parameter open-access multilingual language model. preprint arXiv:2211.05100, 2022. arXiv [32] Yi Tay, Jason W… find
  32. [33] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485– 5551, 2020.
  33. [34] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMa: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  34. [35] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/owen-2024-predictable-language-model-benchmark-performance/