TinyStories: How Small Can Language Models Be and Still Speak Coherent English?

What it is about

The paper introduces TinyStories, a synthetic corpus of short stories written by GPT-3.5 and GPT-4 with only words that a 3–4-year-old would know (about 1,500 basic words). Each story is generated from three random words (noun, verb, adjective) and a random set of features such as dialogue or a twist, to force diversity. Small GPT-Neo models trained on it, below 10M parameters or with a single transformer block, write fluent multi-paragraph stories with almost perfect grammar. An evaluation framework asks GPT-4 to grade completions on grammar, creativity, consistency and plot. Grammar appears first in small models; consistency and creativity need larger hidden size; one-layer models fail at following instructions. A 28M-parameter model with eight layers already beats the 1.5B GPT2-XL on some of the paper's factual, reasoning and consistency prompts. Training takes under 30 hours on one V100 GPU. Memorisation is ruled out by n-gram overlap with the training set.

Ideas

I1. A closed-vocabulary parse test: generate sentences only from the child-vocabulary word list, and measure LAS and UAS on that subset with en parse-eval. A closed vocabulary separates syntax from lexical coverage.
I2. Score Mova's school answers on separate axes, like the paper's rubric, so that grammar and consistency gains are reported apart. Keep the counters in world/src/knowledge.rs.
I3. An entity-tracking test: story openings where the continuation must keep names, places and objects. Check in world/src/story.rs that a mention after a gap refers to the same entity. The paper's one-layer failures show where the hard cases are.
I4. Overlap check for any generated text: report the n-gram overlap with the lesson corpus next to each output, as the paper does before claiming diversity.

For Mova

Chapters: languages/en/19-lexicon-mwe-register.md (the basic vocabulary) and languages/en/35-reference-and-coreference.md (entity tracking). Code: en/src/parse.rs, world/src/story.rs, world/src/knowledge.rs. Measure: closed-vocabulary LAS/UAS and entity-tracking accuracy, run twice.

Library summary

The Research library's own summary of the paper, in our words — not the paper's text.

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2305.07759

Summary

Small language models (around 125M parameters, such as GPT-Neo or GPT-2 small) trained on broad web corpora rarely stay coherent for more than a few words. That leaves an open question: does fluent English need a large model, or is the training data simply too broad for a small one? Eldan and Li address this with TinyStories, a synthetic corpus of short stories written by GPT-3.5 and GPT-4. The stories use only vocabulary a 3–4-year-old child would know. Some models trained on it have under 10M parameters or a single transformer block, yet they write multi-paragraph stories that are varied and almost free of grammar errors. They also show simple reasoning and factual knowledge, and a 28M model's continuation compares favourably with GPT2-XL. The authors also propose an evaluation method in which GPT-4 grades generated stories the way a teacher would, with separate scores for grammar, creativity, consistency and instruction-following. Training takes under a day on one GPU, so the setup makes scaling, architecture and interpretability studies cheap. It also suggests that narrowly focused synthetic data could help in low-resource domains.

Key points

Released resources (Hugging Face): the TinyStories dataset and the models TinyStories-1M/3M/9M/28M/33M, TinyStories-1Layer/2Layer and TinyStories-Instruct-∗. The models use the GPT-Neo architecture with a 512-token context, a 256-token local attention window, and the GPT-Neo tokenizer cut down to its 10K most frequent tokens.
Comparison with a much larger model: the authors compare a 28M TinyStories model with GPT2-XL (1.5B parameters), using prompts checked to have no 6-gram overlap with the training data. Even a 2.5M model completes factual prompts sensibly. A 33M model with 4 layers solves a simple either/or reasoning prompt (a dog is refused, so she asks for a cat), but GPT2-XL does not.
Example GPT-4 grades (hidden size 768): a 2-layer model with 8 heads reaches eval loss 1.33, grammar 8.25 and consistency 8.16. 1-layer models score lower, with consistency of 6.38–7.02 and loss of 1.54–1.58. More attention heads generally help.
Order of emergence and model shape: grammar appears first, then consistency, then creativity. Width seems to matter more for factual knowledge, and depth more for tracking context.
Interpretability: in small models, attention heads split clearly into local and semantic types, and MLP neurons fire on tokens with a shared role in the sentence.

Cite

The paper itself, not this card. BibTeX from the library's record of the work. arXiv 2305.07759

@misc{eldan2023tinystories,
  title = {{TinyStories: How Small Can Language Models Be and Still Speak Coherent English?}},
  author = {Eldan, Ronen and Li, Yuanzhi},
  year = {2023},
  eprint = {2305.07759},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.07759}
}

References

33 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Common crawl. Accessed: 2019. find
Zeyuan Allen-Zhu and Yuanzhi Li. Towards understanding ensemble, knowledge distillation and self-distillation in deep learning. arXiv preprint arXiv:2012.09816, 2020. ↗
Sid Black, Leo Gao, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. find
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020. find
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023. ↗ our card
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert’s attention. arXiv preprint arXiv:1906.04341, 2019. ↗
John H Flavell, Eleanor R Flavell, Frances L Green, and Louis J Moses. Young children’s understanding of fact beliefs versus value beliefs. Child development, 61(4):915–928, 1990. ↗
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv preprint arXiv:1803.03635, 2018. ↗
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027, 2020. ↗
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531, 2015. ↗ our card
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15556, 2022. ↗
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. ↗
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):6869–6898, 2017. find
Samy Jelassi, Michael Sander, and Yuanzhi Li. Vision transformers provably learn spatial structure. Advances in Neural Information Processing Systems, 35:37822–37836, 2022. ↗
Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. ↗ our card
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. ↗
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012. find
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. Visualizing and understanding neural models in nlp. arXiv preprint arXiv:1506.01066, 2015. ↗ our card
Yuchen Li, Yuanzhi Li, and Andrej Risteski. How do transformers learn topic structure: Towards a mechanistic understanding. arXiv preprint arXiv:2303.04245, 2023. ↗
Wick Miller and Susan Ervin. The development of grammar in child language. Monographs of the Society for Research in Child Development, pages 9–34, 1964. ↗
OpenAI. Gpt-4 technical report, 2023. find our card
Denis Paperno, Germán Kruszewski, Angeliki Lazaridou, Quan Ngoc Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernández. The lambada dataset: Word prediction requiring a broad discourse context. arXiv preprint arXiv:1606.06031, 2016. ↗ our card
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019. find
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485–5551, 2020. find
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019. ↗
Michael Santacroce, Zixin Wen, Yelong Shen, and Yuanzhi Li. What matters in the structured pruning of generative language models? arXiv preprint arXiv:2302.03773, 2023. ↗
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615, 2022. ↗ our card
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: a compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02984, 2020. ↗ our card
Wilson L Taylor. “cloze procedure”: A new tool for measuring readability. Journalism quarterly, 30(4):415–433, 1953. find
Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418, 2019. ↗ our card
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. Ccnet: Extracting high quality monolingual datasets from web crawl data. arXiv preprint arXiv:1911.00359, 2019. ↗ our card
Terry Winograd. Understanding natural language. Cognitive psychology, 3(1):1–191, 1972. ↗
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103, 2017. ↗