Elamur Research⌕
☰
home / papers / 2026 / MiniGPT: Rebuilding GPT from First Principles

MiniGPT: Rebuilding GPT from First Principles

Paper · 2026 · Jibin Joseph

Summary

This report rebuilds a GPT-style language model from scratch. The goal is to make the full decoder-only pipeline easy to read and rerun, not to invent a new architecture. The author studied Karpathy's nanoGPT as a design reference, then wrote the code independently in PyTorch inside one Jupyter notebook. The notebook covers:

  • learned token and position embeddings
  • masked multi-head attention
  • pre-LayerNorm blocks with residual paths and MLPs
  • next-character cross-entropy training
  • validation tracking and checkpoint choice
  • sampling

The test bed is Tiny Shakespeare with character-level tokens, so no external tokenizer is needed. A small baseline first confirms that the whole pipeline works. A larger configuration then shows how model capacity and context size change the results. The larger model learns local Shakespearean form: speaker names, line breaks, punctuation and dialogue layout. Over longer stretches, its text loses meaning. The work matters mainly as a transparent reference implementation, with an honest list of limits: tiny data, character tokens, small scale and only qualitative evaluation.

Key points

  • The baseline has 826,433 parameters (context 128, 4 layers, 4 heads, width 128) and reaches validation loss 1.7236 after 3000 iterations.
  • The stronger model has 10.77M parameters (context 256, 6 layers, 6 heads, width 384) and reaches a best validation loss of 1.4780.
  • The best checkpoint of the stronger run comes at step 1750. After that, training loss keeps falling while validation loss rises, which is overfitting handled by early stopping.
  • A 256-character context covers only a few dozen words, which limits speaker consistency and long-range coherence.
  • The code is at github.com/jibin10/MiniGPT as one Colab notebook, with timings measured on an A100. The seed is fixed at 42, but the author notes that GPU nondeterminism can still cause small differences.

Interesting ideas

  • I1. Train a deliberately tiny model first, only to prove the full pipeline runs end to end, and scale up only after that check passes. This separates bugs in the pipeline from limits of capacity.
  • I2. Pick the saved checkpoint by the lowest validation loss rather than by the final step, because training loss keeps improving while held-out performance gets worse.
  • I3. Character-level tokens remove any dependence on an external tokenizer and keep preprocessing fully inspectable, at the cost of much longer sequences.
  • I4. Treat sampling temperature as an explicit, reported dial: low values give stable but repetitive text, and high values give varied but less coherent output.
  • I5. Package model, training, results and samples in one rerunnable artifact with a fixed seed, and state openly which sources of nondeterminism remain.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (tree): Before long gparse runs, train a miniature gparse on a small treebank slice to confirm that the data flow, LAS scoring and decoding are correct.
  • I2 (learn): Keep the best-on-dev snapshot when training on silver data agreed by witness parsers, since memorizing silver noise shows up as dev degradation.
  • I3 (tokens): A character-level fallback model in crate en could guess the morphology and lemmas of out-of-vocabulary words without needing a learned subword vocabulary.
  • I4 (back): Record the decoding setting alongside each generated sentence in the graph → text step, so that fluency and faithfulness trade-offs stay explainable and comparable.
  • I5 (learn): Each evaluation run, including negative controls, could ship as a single reproducible script that logs its seed and its known variance sources.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of Texas at AustinAustin · United States109 works · 56,942 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Jibin Joseph. 2026. MiniGPT: Rebuilding GPT from First Principles. arXiv:2605.17398.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2605.17398 · arXiv 2605.17398

@misc{joseph2026minigpt,
  title = {{MiniGPT: Rebuilding GPT from First Principles}},
  author = {Jibin Joseph},
  year = {2026},
  doi = {10.48550/arxiv.2605.17398},
  eprint = {2605.17398},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2605.17398}
}

References

6 references, 4 of them with a card of ours. ↗ leads to the work.

  1. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. doi:10.1038/nature14539. Deep learning. Nature, 521(7553):436–444, 2015. ↗
  2. Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks, 61:85–117, 2015. doi:10.1016/j.neunet.2014.09.003. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. OpenAI technical report, 2018. URL https://cdn.openai.com/research-covers/ language-unsupervised/language_understanding_paper.pdf. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI technical report, 2019. URL https://cdn.openai.com/ better-language-models/language_models_are_unsupervised_multitask_learners.pdf. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877–1901, 2020. Andrej Karpathy. mingpt: A minimal pytorch re-implementation of gpt. https://github.com/karpathy/minGPT, 2020. GitHub repository. Andrej Karpathy. nanogpt: The simplest, fastest repository for training/finetuning medium-sized gpts. https://github.com/karpathy/nanoGPT, 2022. GitHub repository. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155, 2003. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 1715–1725, 2016. doi:10.18653/v1/P16-1162. Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. Proceedings of the AAAI Conference on Artificial Intelligence, 33(1):3159–3166, 2019. doi:10.1609/aaai.v33i01.33013159. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38–45. Association for Computational Linguistics, 2020. doi:10.18653/v1/2020.emnlp-demos.6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770–778, 2016. doi:10.1109/CVPR.2016.90. Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016. Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tie-Yan Liu. On layer normalization in the transformer architecture. In Proceedings of the 37th International Conference on Machine Learning, pages 10524–10533. ↗
  3. PMLR, 2020. Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929–1958, 2014. URL https://jmlr.org/papers/v15/srivastava14a.html. Ofir Press and Lior Wolf. Using the output embedding to improve language models. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, pages 157–163. Association for Computational Linguistics, 2017. doi:10.18653/v1/E17-2025. Andrej Karpathy. char-rnn: Multi-layer recurrent neural networks for character-level language models. https://github.com/karpathy/char-rnn, 2015. GitHub repository. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7. ↗
  4. Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015. URL https://arxiv.org/abs/1412.6980. Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=Skq89Scxx. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In Proceedings of the 30th International Conference on Machine Learning, pages 1310–1318. ↗
  5. PMLR, 2013. URL https://proceedings.mlr.press/v28/pascanu13.html. Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. In International Conference on Learning Representations, 2018. URL https://openreview.net/forum?id=r1gs9JgRZ. Lutz Prechelt. Early stopping—but when? In Genevieve B. Orr and Klaus-Robert Müller, editors, Neural Networks: Tricks of the Trade, volume 1524 of Lecture Notes in Computer Science, pages 55–69. ↗
  6. Springer, 1998. doi:10.1007/3-540-49430-8_3. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id= rygGQyrFvH. Jibin Joseph. Minigpt: Rebuilding gpt from first principles. https://github.com/jibin10/MiniGPT, 2026. GitHub repository. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361, 2020. URL https://arxiv.org/abs/2001.08361. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre. Training compute-optimal large language models. In Advances in Neural Information Processing Systems, volume 35, pages 30016–30030, 2022. URL https://arxiv.org/abs/2203.15556. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32, pages 8024–8035, 2019. Thomas Kluyver, Benjamin Ragan-Kelley, Fernando Pérez, Brian Granger, Matthias Bussonnier, Jonathan Frederic, Kyle Kelley, Jessica Hamrick, Jason Grout, Sylvain Corlay, Paul Ivanov, Damián Avila, Safia Abdalla, and Carol Willing. Jupyter notebooks—a publishing format for reproducible computational workflows. In Positioning and Power in Academic Publishing: Players, Agents and Agendas, pages 87–90. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/joseph-2026-minigpt-rebuilding-gpt-first-principles/