Elamur Research⌕
☰
home / papers / 2024 / Chinchilla Scaling: A replication attempt

Chinchilla Scaling: A replication attempt

Summary

Hoffmann et al. (2022), the Chinchilla paper, estimated compute-optimal scaling for language models in three ways. The third, Approach 3, fits a parametric loss L(N, D) = E + A/N^α + B/D^β, where N is parameter count and D is training tokens. Epoch AI researchers checked whether this fit holds up. The original data were never released, so they recovered about 240 training runs from the SVG version of the paper's Figure 4. Point positions gave model size and compute, and the colour bar gave loss. They then refit the formula using Hoffmann et al.'s own procedure (Huber loss with δ = 10⁻³ and a grid of initialisations) and bootstrapped the standard errors. Their estimates differ significantly from the published ones, especially for E and β. The original parameters describe the recovered data poorly, and the reported confidence intervals are far too narrow for the number of experiments. The authors trace these problems to an optimizer that stopped before converging and to rounding of the printed coefficients. Their refit agrees with Hoffmann et al.'s other two approaches. This matters because widely used models, including Google's Gemini, rely on these scaling laws.

Key points

  • Refit vs. Hoffmann et al.:
    • Refit: E = 1.8172, A = 482.01, B = 2085.43, α = 0.3478, β = 0.3658.
    • Hoffmann et al. (TeX-source values): E = 1.6934, A = 406.4, B = 410.7, α = 0.3392, β = 0.2849.
    • A χ² test on bootstrapped covariance gives p < 10⁻⁵¹, and a likelihood ratio test gives p < 10⁻²³⁵.
  • Implausible confidence intervals: Hoffmann et al.'s intervals would require over 600,000 experiments, but they likely ran fewer than 500.
  • Rounding bias: Rounding β from 0.2849 to 0.28 biases the data term upward by about 13% at D = 10¹¹ tokens.
  • Inconsistent tokens-per-parameter ratio:
    • The published fit implies about 70 tokens per parameter, which contradicts the ~20 actually used for the 70B Chinchilla model.
    • The refit is consistent with roughly 20 (25.6 in the variant that keeps five outlier runs).
    • At 1e26 FLOP and above, the refit remains compatible with anywhere from 4 to 40 tokens per parameter.
  • Code availability: Analysis code is published at epochai.org/code/analyzing-chinchilla-repo.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Cite

Cite as Tamay Besiroglu, Ege Erdil, Matthew Barnett and Josh You. 2024. Chinchilla Scaling: A replication attempt. arXiv:2404.10102.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.10102 · arXiv 2404.10102

@misc{besiroglu2024chinchilla,
  title = {{Chinchilla Scaling: A replication attempt}},
  author = {Tamay Besiroglu, Ege Erdil, Matthew Barnett and Josh You},
  year = {2024},
  doi = {10.48550/arxiv.2404.10102},
  eprint = {2404.10102},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.10102}
}

References

9 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anil, R., Dai, A. M., Firat, O., Johnson, M., Lepikhin, D., Passos, A., Shakeri, S., Taropa, E., Bailey, P., Chen, Z., et al. (2023). Palm 2 technical report. arXiv preprint arXiv:2305.10403. our card
  2. Bi, X., Chen, D., Chen, G., Chen, S., Dai, D., Deng, C., Ding, H., Dong, K., Du, Q., Fu, Z., et al. (2024). Deepseek llm: Scaling open-source language models with longtermism. arXiv preprint arXiv:2401.02954. find
  3. Bordelon, B., Atanasov, A., and Pehlevan, C. (2024). A dynamical model of neural scaling laws. arXiv preprint arXiv:2402.01092. — [arXiv:2402.01092](https://arxiv.org/abs/2402.01092) find
  4. Borgeaud, S. (2024). Twitter. Twitter thread involving discussion on the paper’s methodologies and data replication issues. find
  5. Google, G. T. (2023). Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. our card
  6. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556. our card
  7. Michaud, E., Liu, Z., Girit, U., and Tegmark, M. (2024). The quantization model of neural scaling. Advances in Neural Information Processing Systems, 36. our card
  8. Sardana, N. and Frankle, J. (2023). Beyond chinchilla-optimal: Accounting for inference in language model scaling laws. arXiv preprint arXiv:2401.00448. our card
  9. Wilks, S. S. (1938). The large-sample distribution of the likelihood ratio for testing composite hypotheses. The annals of mathematical statistics, 9(1):60–62. find

This page in Markdown → https://md.research.elamur.ai/free/en/papers/besiroglu-2024-chinchilla-scaling-replication-attempt/