Elamur Research⌕
☰
home / authors / Tom Henighan

Tom Henighan

Tom Henighan studies how the behaviour of neural networks changes as they grow. Henighan's papers here centre on scaling laws: empirical relationships between test loss and model size, dataset size and compute budget. The usual method is to train families of transformer models across a wide span of scales and fit simple curves to the results. Recurring questions include how far these regularities extend beyond text, what pre-training is worth for a downstream task, and how data quality shapes learning.

The most cited of these papers is Scaling Laws for Neural Language Models. It trained a broad series of Transformers on WebText2, varying size, data, training length, batch size and shape. It found that cross-entropy loss falls as a smooth power law in each of parameters, data and compute, as long as the other factors are not limiting. Scaling Laws for Autoregressive Generative Modeling carried this question over to images, video, caption–image pairs and generated mathematics. It reported power-law-plus-constant trends in every domain and read the constant as an estimate of the data's own entropy.

The later work moves from describing scaling to explaining specific effects. Scaling Laws for Transfer measures the benefit of language-model pre-training on low-data tasks and expresses it as an equivalent amount of extra data. Scaling Laws and Interpretability of Learning from Repeated Data asks why repeating even a small share of training tokens damages models so much. It combines scaling analysis with mechanistic interpretability and finds a marked double-descent pattern.

The listed current affiliations are SLAC National Accelerator Laboratory and Stanford University. The papers here came out of research groups at OpenAI and later Anthropic, within the area of language models and pretraining. Frequent co-authors include Jared Kaplan, Danny Hernandez, Sam McCandlish and Tom B. Brown.

Tom Henighan has 4 papers in our library, in 2020–2022, most of them on Language models & pretraining, Interpretability; often with Jared Kaplan, Danny Hernandez, Sam McCandlish.

4 papers in our library · 2020–2022

OpenAlex · ORCID · h-index 16 · 36 works · 1,277 citations OpenAlex, 2026-10-14

📄 4 works in our library · 🔗 4,429 citations in our library · 3 more in our library without a page yet · ⭐ Paper Star rank #1146

The profile is the library's match of the name; two people with one name are not always told apart.

Elsewhere: Google Scholar · Semantic Scholar · dblp · ACL Anthology

 Affiliations

🔬SLAC National Accelerator LaboratoryMenlo Park · United States2015–2023 · 14 paperscurrent🎓Stanford UniversityStanford · United States2016–2023 · 9 paperscurrent🎓The Ohio State UniversityColumbus · United States2009–2012 · 6 papers🎓Delft University of TechnologyDelft · Netherlands2011–2014 · 4 papers

From the bylines of their papers (OpenAlex, 2026-10-14), ORCID and Wikidata; institutions on papers are matched by machine. 2 more institutions on single bylines are not listed.

 Topics

 Language models & pretraining (2) Interpretability (1)

 Works by year

2020s

20221 paper20211 paper20202 papers

 Papers

The most cited first (OpenAlex and our library).

  1. Scaling Laws for Neural Language Models (2020 · 1,814 citations)
    The authors from OpenAI study how the quality of a language model depends on its scale. They measure quality by the cross-entropy loss on test data. To this end, they trained a large series of Transformer models on the WebText2 corpus, ranging from thousands to roughly a billion parameters, excluding embeddings. In…
  2. Scaling Laws for Autoregressive Generative Modeling (2020 · 115 citations)
    The authors from OpenAI examine whether the scaling laws previously found for language models also hold for other data types.
  3. Scaling Laws for Transfer (2021 · 102 citations)
    This paper asks how much language-model pre-training is worth when the target task has little data. It answers in units of data. The authors trained transformers across roughly four orders of magnitude in both non-embedding parameters and dataset size, using three curricula:
  4. Scaling Laws and Interpretability of Learning from Repeated Data (2022 · 62 citations)
    Large language models often see the same data several times. Sometimes this is done deliberately, to upweight high-quality sources, and sometimes it results from incomplete deduplication. The authors from Anthropic investigate why even a small fraction of repeated data harms models so severely. They train a family of…

Also in our library

3 more works of theirs are in our library without a page here yet; they will appear as we publish them.

 Co-authors here

12 of 19 co-authors with a page, the most papers together first.

 Cited by

Papers of our library cite their works 4,429 times (of the 140,884 papers whose bibliographies we read); 321 citing papers with a page here →.

This page in Markdown → https://md.research.elamur.ai/free/en/authors/tom-henighan/