Elamur Research⌕
☰
home / papers / 2024 / Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

Summary

The Chinchilla scaling laws suggest how to divide a compute budget between the number of parameters and the amount of training data. However, they account only for training cost and ignore the cost of inference. The authors from Databricks MosaicML change this formulation. They fix a target model quality (pre-training loss) and the expected volume of requests over the model's entire lifetime, and then search for the model size and number of tokens that minimize the combined cost of training and inference. The problem has no analytical solution, so the optimum is found numerically using Newton's method. Costs are estimated twice: in FLOPs and in actual money, taking into account hardware utilization (MFU) and GPU prices. The conclusion: if around a billion requests or more are expected, it is more cost-effective to train a smaller model for longer than Chinchilla recommends. To test this in practice, the authors trained 47 models. Quality kept improving even with extremely long training, and they observed no signs of saturation. The work explains why models such as Llama 2 and Llama 3 were trained on far more tokens than is "optimal" according to Chinchilla.

Key points

  • Experiments: 47 models ranging from 150M to 6B parameters. Between 10 and 10,000 tokens were used per parameter, and data were not repeated (a single epoch). Quality was evaluated on the MosaicML Evaluation Gauntlet (MMLU, ARC, PIQA, COPA, etc.).
  • Example in FLOPs: a developer plans a 13B model and expects 2 trillion inference tokens. If a 7B model is trained on more data instead, total compute decreases by 1.7×10²² FLOPs, i.e. by roughly 17%. For quality at the level of 30B-Chinchilla and 10¹³ inference tokens, the optimum is a 13.6B model with 2.84× more data; this saves 28% of FLOPs.
  • Example in money: for quality at the level of 30B-Chinchilla and 1.5B requests, a 16B model trained on 3.35T tokens is 17% cheaper.
  • Why money and FLOPs diverge: for Chinchilla-70B with 2T inference tokens, the overspend in FLOPs is only 1.3%, but in money it is already 36%. The reason is that generating output tokens has roughly 50 times lower MFU than training.
  • Ablation: if the Chinchilla coefficients are fitted only on typical ratios (≤100 tokens per parameter), the law overestimates the benefit of additional data for very long training.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics: Language models & pretraining

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.00448 · arXiv 2401.00448

@misc{sardana2024beyond,
  title = {{Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws}},
  author = {Sardana, Nikhil and Portes, Jacob and Doubov, Sasha and Frankle, Jonathan},
  year = {2024},
  doi = {10.48550/arxiv.2401.00448},
  eprint = {2401.00448},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.00448}
}

References

69 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abnar, S., Dehghani, M., Neyshabur, B., and Sedghi, H. Exploring the limits of large scale pre-training. arXiv preprint arXiv:2110.02095, 2021. — [arXiv:2110.02095](https://arxiv.org/abs/2110.02095) find
  2. AI@Meta. Llama 3 model card, 2024. URL https://github.com/meta-llama/llama3/blob/main/MODEL CARD.md. ↗
  3. Ainslie, J., Lee-Thorp, J., de Jong, M., Zemlyanskiy, Y., Lebrón, F., and Sanghai, S. Gqa: Training generalized multi-query transformer models from multi-head checkpoints, 2023. find our card
  4. Amini, A., Gabriel, S., Lin, S., Koncel-Kedziorski, R., Choi, Y., and Hajishirzi, H. Mathqa: Towards interpretable math word problem solving with operation-based formalisms. CoRR, abs/1905.13319, 2019. URL http://arxiv.org/abs/1905.13319. ↗ our card
  5. Barton, T. Calibrating the mosaic evaluation gauntlet. https://www.databricks.com/blog/calibrating-mosaic-evaluation-gauntlet, April 2024. Blog post. ↗
  6. Besiroglu, T., Erdil, E., Barnett, M., and You, J. Chinchilla scaling: A replication attempt. arXiv preprint arXiv:2404.10102, 2024. find our card
  7. Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432–7439, 2020. find our card
  8. Bordelon, B., Atanasov, A., and Pehlevan, C. A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01092, 2024. — [arXiv:2402.01092](https://arxiv.org/abs/2402.01092) find
  9. Caballero, E., Gupta, K., Rish, I., and Krueger, D. Broken neural scaling laws. arXiv preprint arXiv:2210.14891, 2022. — [arXiv:2210.14891](https://arxiv.org/abs/2210.14891) find
  10. Chen, X., Liang, C., Huang, D., Real, E., Wang, K., Liu, Y., Pham, H., Dong, X., Luong, T., Hsieh, C.-J., Lu, Y., and Le, Q. V. Symbolic discovery of optimization algorithms, 2023. find
  11. Clark, C., Lee, K., Ming-Wei Chang, T. K., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. In NAACL, 2019. find our card
  12. Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv:1803.05457v1, 2018. find
  13. Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C., and Schulman, J. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021. find our card
  14. De Vries, H. Go smol or go home, 2023. https://www.harmdevries.com/post/model-size-vs-compute-overhead/. ↗
  15. URL Dey, N., Gosal, G., Zhiming, Chen, Khachane, H., Marshall, W., Pathria, R., Tom, M., and Hestness, J. Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster, 2023. find
  16. Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre-trained transformers, 2023. find
  17. Gadre, S. Y., Smyrnis, G., Shankar, V., Gururangan, S., Wortsman, M., Shao, R., Mercat, J., Fang, A., Li, J., Keh, S., et al. Language models scale reliably with over-training and on downstream tasks. arXiv preprint arXiv:2403.08540, 2024. find
  18. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020. find our card
  19. Henighan, T., Kaplan, J., Katz, M., Chen, M., Hesse, C., Jackson, J., Jun, H., Brown, T. B., Dhariwal, P., Gray, S., et al. Scaling laws for autoregressive generative modeling. arXiv preprint arXiv:2010.14701, 2020. find our card
  20. Hernandez, D., Kaplan, J., Henighan, T., and McCandlish, S. Scaling laws for transfer. arXiv preprint arXiv:2102.01293, 2021. find our card
  21. Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically. arXiv preprint arXiv:1712.00409, 2017. find our card
  22. Hestness, J., Ardalani, N., and Diamos, G. Beyond human-level accuracy: Computational challenges in deep learning. In Proceedings of the 24th symposium on principles and practice of parallel programming, pp. 1–14, 2019. find
  23. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoc, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Rae, J. W., Vinyals, O., and Sifre, L.… find our card
  24. Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models, 2020. find our card
  25. Knight, W. Openai’s ceo says the age of giant ai models is already over. Wired, 2023. ISSN 1059-1028. URL https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/. ↗
  26. Korthikanti, V., Casper, J., Lym, S., McAfee, L., Andersch, M., Shoeybi, M., and Catanzaro, B. Reducing activation recomputation in large transformer models, 2022. find our card
  27. Krajewski, J., Ludziejewski, J., Adamczewski, K., Pióro, M., Krutul, M., Antoniak, S., Ciebiera, K., Król, K., Odrzygóźdź, T., Sankowski, P., et al. Scaling laws for fine-grained mixture of experts. arXiv preprint arXiv:2402.07871, 2024. — [arXiv:2402.07871](https://arxiv.org/abs/2402.07871) find
  28. Labs, L. Gpu cloud - vms for deep learning. https://lambdalabs.com/service/gpu-cloud, 2023. Accessed 2023-10-02. ↗
  29. Levesque, H., Davis, E., and Morgenstern, L. The winograd schema challenge. In Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning. Citeseer, 2012. find
  30. Michaud, E., Liu, Z., Girit, U., and Tegmark, M. The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36, 2024. find our card
  31. Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Conference on Empirical Methods in Natural Language Processing, 2018. URL https://api.semanticscholar.org/CorpusID:52183757. ↗
  32. Mikami, H., Fukumizu, K., Murai, S., Suzuki, S., Kikuchi, Y., Suzuki, T., Maeda, S.-i., and Hayashi, K. A scaling law for synthetic-to-real transfer: How much is your pre-training effective? arXiv preprint arXiv:2108.11018, 2021. — [arXiv:2108.11018](https://arxiv.org/abs/2108.11018) find
  33. MosaicML. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. URL www.mosaicml.com/blog/mpt-7b. Accessed: 2024-01-28. find
  34. MosaicML NLP Team. Llm evaluation scores. https://www.mosaicml.com/llm-evaluation, 2023. Accessed: 2023-09-28. ↗
  35. Muennighoff, N., Rush, A. M., Barak, B., Scao, T. L., Piktus, A., Tazi, N., Pyysalo, S., Wolf, T., and Raffel, C. Scaling data-constrained language models, 2023. find
  36. NVIDIA. Nvidia a100 datasheet, 2021. URL https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf. ↗
  37. OpenAI and Pilipiszyn, A. Gpt-3 powers the next generation of apps, Mar 2021. URL https://openai.com/blog/gpt-3-apps. ↗
  38. Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bernardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. find our card
  39. Paquette, E., Paquette, C., Xiao, L., and Pennington, J. 4+ 3 phases of compute-optimal neural scaling laws. arXiv preprint arXiv:2405.15074, 2024. — [arXiv:2405.15074](https://arxiv.org/abs/2405.15074) find
  40. Patel, A., Bhattamishra, S., and Goyal, N. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2080–2094, Online, June 2021. Association f… find our card
  41. Pope, R., Douglas, S., Chowdhery, A., Devlin, J., Bradbury, J., Levskaya, A., Heek, J., Xiao, K., Agrawal, S., and Dean, J. Efficiently scaling transformer inference, 2022. find
  42. Porian, T., Wortsman, M., Jitsev, J., Schmidt, L., and Carmon, Y. Resolving discrepancies in compute-optimal scaling of language models. arXiv preprint arXiv:2406.19146, 2024. find our card
  43. Press, O., Smith, N. A., and Lewis, M. Train short, test long: Attention with linear biases enables input length extrapolation, 2022. find
  44. Rae, J. W., Borgeaud, S., Cai, T., Millican, K., Hoffmann, J., Song, F., Aslanides, J., Henderson, S., Ring, R., Young, S., Rutherford, E., Hennigan, T., Menick, J., Cassirer, A., Powell, R., van den Driessche, G., Hendricks, L. A., Rauh, M., Huang, P.-S., Glaese, A., Welbl, J., Dathathri, S., Huang… find
  45. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. find
  46. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv e-prints, art. arXiv:1606.05250, 2016. find our card
  47. Reddy, S., Chen, D., and Manning, C. D. CoQA: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249–266, 2019. doi: 10.1162/tacl a 00266. URL https://aclanthology.org/Q19-1016. ↗ our card
  48. Roemmele, M., Beja, C. A., and Gordon, A. S. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. Papers from the 2011 AAAI Spring Symposium, 2011. find
  49. Rosenfeld, J. S., Rosenfeld, A., Belinkov, Y., and Shavit, N. A constructive prediction of the generalization error across scales. arXiv preprint arXiv:1909.12673, 2019. find
  50. Ruan, Y., Maddison, C. J., and Hashimoto, T. Observational scaling laws and the predictability of language model performance. arXiv preprint arXiv:2405.10938, 2024. find our card
  51. Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. WINOGRANDE: an adversarial winograd schema challenge at scale. CoRR, abs/1907.10641, 2019. URL http://arxiv.org/abs/1907.10641. ↗
  52. Sap, M., Rashkin, H., Chen, D., LeBras, R., and Choi, Y. Socialiqa: Commonsense reasoning about social interactions. arXiv preprint arXiv:1904.09728, 2019. find our card
  53. Sardana, N. and Frankle, J. Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.00448, 2023. — [arXiv:2401.00448](https://arxiv.org/abs/2401.00448) find
  54. Shazeer, N. and Freitas, D. d. Introducing character, Dec 2022. URL https://blog.character.ai/introducing-character/. ↗
  55. Sorscher, B., Geirhos, R., Shekhar, S., Ganguli, S., and Morcos, A. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35:19523–19536, 2022. find our card
  56. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. find our card
  57. Talmor, A., Herzig, J., Lourie, N., and Berant, J. Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937, 2018. find our card
  58. Tay, Y., Dehghani, M., Abnar, S., Chung, H. W., Fedus, W., Rao, J., Narang, S., Tran, V. Q., Yogatama, D., and Metzler, D. Scaling laws vs model architectures: How does inductive bias influence scaling? arXiv preprint arXiv:2207.10551, 2022. find our card
  59. The Mosaic ML Team. composer. https://github.com/mosaicml/composer/, 2021. ↗
  60. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lample, G. Llama: Open and efficient foundation language models, 2023a. find our card
  61. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hossei… find
  62. Tow, J., Bellagente, M., Mahan, D., and Riquelme, C. Stablelm 3b 4e1t, 2023. URL https://huggingface.co/stabilityai/stablelm-3b-4e1t. ↗
  63. Villalobos, P. Scaling laws literature review, 2023. URL https://epochai.org/blog/scaling-laws-literature-review. ↗
  64. Villalobos, P. and Atkinson, D. Trading off compute in training and inference, 2023. URL https://epochai.org/blog/trading-off-compute-in-training-and-inference. Accessed: 2023-9-26. ↗
  65. Wolfe, T., Tunstall, L., and von Platen, P. Jeopardy dataset on hugging face hub. https://huggingface.co/datasets/jeopardy, 2022. ↗
  66. Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models, 2023. find
  67. Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. find
  68. Zheng, L., Chiang, W.-L., Sheng, Y., Li, T., Zhuang, S., Wu, Z., Zhuang, Y., Li, Z., Lin, Z., Xing, E. P., Gonzalez, J. E., Stoica, I., and Zhang, H. Lmsys-chat-1m: A large-scale real-world llm conversation dataset, 2023. find our card
  69. Zhong, W., Cui, R., Guo, Y., Liang, Y., Lu, S., Wang, Y., Saied, A., Chen, W., and Duan, N. Agieval: A human-centric benchmark for evaluating foundation models, 2023. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sardana-2024-chinchilla-optimal-accounting-inference-language/