Elamur Research
☰
home / papers / 2025 / gpt-oss-120b & gpt-oss-20b Model Card

gpt-oss-120b & gpt-oss-20b Model Card

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2508.10925

Summary

This model card from OpenAI describes gpt-oss-120b and gpt-oss-20b. They are text-only reasoning models with open weights under the Apache 2.0 license. They are built for agentic workflows: web search, Python execution, function calling and Structured Outputs. Users can set how much effort the model spends on reasoning, and the full chain-of-thought is visible. Both models are Mixture-of-Experts transformers derived from GPT-2/GPT-3. They were pretrained on trillions of text tokens focused on STEM, code and general knowledge, with hazardous biosecurity material filtered out. Post-training used chain-of-thought reinforcement learning similar to OpenAI o3, together with a new "harmony" chat format that defines message roles and channels. On math, science, coding and tool-use benchmarks, the larger model beats o3-mini and gets close to o4-mini. The smaller model stays competitive at roughly one sixth the size. Much of the card is about safety. OpenAI fine-tuned the 120b model adversarially on biology and cyber tasks to simulate an attacker, and it still stayed below the "High" capability threshold of the Preparedness Framework. This matters because it brings strong reasoning to modest hardware and gives a worst-case misuse assessment for releasing open weights.

Key points

  • Sizes: gpt-oss-120b has 36 layers, 116.8B total parameters, 5.1B active per token and 128 experts. gpt-oss-20b has 24 layers, 20.9B total parameters, 3.6B active and 32 experts. Both models route each token to its top 4 experts.
  • Memory: the MoE weights are quantized to MXFP4 (about 4.25 bits per parameter). This lets the 120b model fit on one 80GB GPU and the 20b model run in 16GB of memory.
  • Architecture: attention alternates between dense layers and 128-token banded-window layers, with GQA (8 KV heads). Context reaches 131,072 tokens via YaRN. The tokenizer is o200k_harmony, with 201,088 tokens.
  • Training: the 120b model took about 2.1M H100-hours, and the 20b model needed nearly 10× less. Knowledge cutoff is June 2024.
  • Instruction priority: when instructions conflict, the models follow System > Developer > User > Assistant > Tool. Developers deploying the models may need to add their own safeguards.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

552 papers of our library cite it (of the 140673 whose bibliographies we read); 5 of them have a page here, 547 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2508.10925 · arXiv 2508.10925

@misc{agarwal2025gpt,
  title = {{gpt-oss-120b \& gpt-oss-20b Model Card}},
  author = {Agarwal, OpenAI: Sandhini and Ahmad, Lama and Ai, Jason and Altman, Sam and Applebaum, Andy and Arbus, Edwin and Arora, Rahul K. and Bai, Yu and Baker, Bowen and Bao, Haiming and Barak, Boaz and Bennett, Ally and Bertao, Tyler and Brett, Nivedita and Brevdo, Eugene and Brockman, Greg and Bubeck, Sebastien and Chang, Che and Chen, Kai and Chen, Mark and Cheung, Enoch and Clark, Aidan and Cook, Dan and Dukhan, Marat and Dvorak, Casey and Fives, Kevin and Fomenko, Vlad and Garipov, Timur and Georgiev, Kristian and Glaese, Mia and Gogineni, Tarun and Goucher, Adam and Gross, Lukas and Guzman, Katia Gil and Hallman, John and Hehir, Jackie and Heidecke, Johannes and Helyar, Alec and Hu, Haitang and Huet, Romain and Huh, Jacob and Jain, Saachi and Johnson, Zach and Koch, Chris and Kofman, Irina and Kundel, Dominik and Kwon, Jason and Kyrylov, Volodymyr and Le, Elaine Ya and Leclerc, Guillaume and Lennon, James Park and Lessans, Scott and Lezcano-Casado, Mario and Li, Yuanzhi and Li, Zhuohan and Lin, Ji and Liss, Jordan and Liu, Lily and Liu, Jiancheng and Lu, Kevin and Lu, Chris and Martinovic, Zoran and McCallum, Lindsay and McGrath, Josh and McKinney, Scott and McLaughlin, Aidan and Mei, Song and Mostovoy, Steve and Mu, Tong and Myles, Gideon and Neitz, Alexander and Nichol, Alex and Pachocki, Jakub and Paino, Alex and Palmie, Dana and Pantuliano, Ashley and Parascandolo, Giambattista and Park, Jongsoo and Pathak, Leher and Paz, Carolina and Peran, Ludovic and Pimenov, Dmitry and Pokrass, Michelle and Proehl, Elizabeth and Qiu, Huida and Raila, Gaby and Raso, Filippo and Ren, Hongyu and Richardson, Kimmy and Robinson, David and Rotsted, Bob and Salman, Hadi and Sanjeev, Suvansh and Schwarzer, Max and Sculley, D. and Sikchi, Harshit and Simon, Kendal and Singhal, Karan and Song, Yang and Stuckey, Dane and Sun, Zhiqing and Tillet, Philippe and Toizer, Sam and Tsimpourlas, Foivos and Vyas, Nikhil and Wallace, Eric and Wang, Xin and Wang, Miles and Watkins, Olivia and Weil, Kevin and Wendling, Amy and Whinnery, Kevin and Whitney, Cedric and Wong, Hannah and Yang, Lin and Yang, Yu and Yasunaga, Michihiro and Ying, Kristen and Zaremba, Wojciech and Zhan, Wenting and Zhang, Cyril and Zhang, Brian and Zhang, Eddie and Zhao, Shengjia},
  year = {2025},
  doi = {10.48550/arxiv.2508.10925},
  eprint = {2508.10925},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2508.10925}
}

References

35 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Proceedings of Advances in Neural Information Processing Systems, 2017. find
  2. N. Shazeer, A. Mirhoseini, K. Maziarz, A. Davis, Q. Le, G. Hinton, and J. Dean, “Outrageously large neural networks: The sparsely-gated mixture-of-experts layer,” 2017. find
  3. D. Lepikhin, H. Lee, Y. Xu, D. Chen, O. Firat, Y. Huang, M. Krikun, N. Shazeer, and Z. Chen, “Gshard: Scaling giant models with conditional computation and automatic sharding,” arXiv preprint arXiv:2006.16668, 2020. find
  4. N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, et al., “Glam: Efficient scaling of language models with mixture-of-experts,” in International conference on machine learning, pp. 5547–5569, PMLR, 2022. find our card
  5. O. C. Project, “OCP Microscaling Formats (MX) Specification Version 1.0,” technical report, Open Compute Project, Sept. 2023. find
  6. B. Zhang and R. Sennrich, “Root mean square layer normalization,” 2019. find
  7. R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and T.-Y. Liu, “On layer normalization in the transformer architecture,” 2020. find
  8. A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., “Language models are unsupervised multitask learners,” OpenAI blog, 2019. find
  9. N. Shazeer, “GLU variants improve transformer,” arXiv preprint arXiv:2002.05202, 2020. find
  10. R. Child, S. Gray, A. Radford, and I. Sutskever, “Generating long sequences with sparse transformers,” arXiv preprint arXiv:1904.10509, 2019. find
  11. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” NeurIPS, 2020. find
  12. J. Ainslie, J. Lee-Thorp, M. de Jong, Y. Zemlyanskiy, F. Lebrón, and S. Sanghai, “GQA: Training generalized multi-query transformer models from multi-head checkpoints,” 2023. find our card
  13. N. Shazeer, “Fast transformer decoding: One write-head is all you need,” arXiv preprint arXiv:1911.02150, 2019. find
  14. J. Su, M. Ahmed, Y. Lu, S. Pan, W. Bo, and Y. Liu, “Roformer: Enhanced transformer with rotary position embedding,” Neurocomputing, 2024. find
  15. B. Peng, J. Quesnelle, H. Fan, and E. Shippole, “YaRN: Efficient context window extension of large language models,” arXiv preprint arXiv:2309.00071, 2023. find
  16. E. Miller, “Attention is off by one (2023),” URL https://www.evanmiller.org/attention-is-off-by-one.html. ↗
  17. G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis, “Efficient streaming language models with attention sinks,” arXiv preprint arXiv:2309.17453, 2023. find
  18. A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford, et al., “GPT-4o system card,” arXiv preprint arXiv:2410.21276, 2024. find our card
  19. A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., “Pytorch: An imperative style, high-performance deep learning library,” Advances in neural information processing systems, vol. 32, 2019. find our card
  20. P. Tillet, H.-T. Kung, and D. Cox, “Triton: an intermediate language and compiler for tiled neural network computations,” in Proceedings of the 3rd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages, pp. 10–19, 2019. find our card
  21. T. Dao, D. Y. Fu, S. Ermon, A. Rudra, and C. Ré, “FlashAttention: Fast and memory-efficient exact attention with IO-awareness,” 2022. find
  22. OpenAI, “Chowdhury, neil and aung, james and shern, chan jun and jaffe, oliver and sherburn, dane and starace, giulio and mays, evan and dias, rachel and aljubeh, marwan and glaese, mia and jimenez, carlos e and yang, john and ho, leyton and patwardhan, tejal and liu, kevin and madry, aleksander.” h… find
  23. S. Yao, N. Shinn, P. Razavi, and K. Narasimhan, “τ -bench: A benchmark for tool-agent-user interaction in real-world domains,” arXiv preprint arXiv:2406.12045, 2024. find our card
  24. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman, “GPQA: A graduate-level google-proof QA benchmark,” in COLM, 2024. find
  25. D. Hendrycks, C. Burns, S. Basart, A. Zou, M. Mazeika, D. Song, and J. Steinhardt, “Measuring massive multitask language understanding,” arXiv preprint arXiv:2009.03300, 2020. find our card
  26. L. Phan, A. Gatti, Z. Han, N. Li, J. Hu, H. Zhang, C. B. C. Zhang, M. Shaaban, J. Ling, S. Shi, et al., “Humanity’s last exam,” arXiv preprint arXiv:2501.14249, 2025. find our card
  27. N. Chowdhury, J. Aung, C. J. Shern, O. Jaffe, D. Sherburn, G. Starace, E. Mays, R. Dias, M. Aljubeh, M. Glaese, C. E. Jimenez, J. Yang, L. Ho, T. Patwardhan, K. Liu, and A. Madry, “Introducing SWE-bench Verified,” OpenAI, 2024. find
  28. R. K. Arora, J. Wei, R. S. Hicks, P. Bowman, J. Quiñonero-Candela, F. Tsimpourlas, M. Sharman, M. Shah, A. Vallone, A. Beutel, et al., “HealthBench: Evaluating large language models towards improved human health,” arXiv preprint arXiv:2505.08775, 2025. find
  29. M. Y. Guan, M. Joglekar, E. Wallace, S. Jain, B. Barak, A. Helyar, R. Dias, A. Vallone, H. Ren, J. Wei, H. W. Chung, S. Toyer, J. Heidecke, A. Beutel, and A. Glaese, “Deliberative alignment: Reasoning enables safer language models,” arXiv preprint arXiv:2412.16339, 2024. find
  30. E. Wallace, K. Xiao, R. Leike, L. Weng, J. Heidecke, and A. Beutel, “The instruction hierarchy: Training LLMs to prioritize privileged instructions,” arXiv preprint arXiv:2404.13208, 2024. find
  31. A. Souly, Q. Lu, D. Bowen, T. Trinh, E. Hsieh, S. Pandey, P. Abbeel, J. Svegliato, S. Emmons, O. Watkins, et al., “A strongreject for empty jailbreaks,” arXiv preprint arXiv:2402.10260, 2024. find
  32. A. Parrish, A. Chen, N. Nangia, V. Padmakumar, J. Phang, J. Thompson, P. M. Htut, and S. R. Bowman, “BBQ: A hand-built bias benchmark for question answering,” arXiv preprint arXiv:2110.08193, 2021. find our card
  33. T. Patwardhan, K. Liu, T. Markov, N. Chowdhury, D. Leet, N. Cone, C. Maltbie, J. Huizinga, C. Wainwright, S. Jackson, S. Adler, R. Casagrande, and A. Madry, “Building an early warning system for LLM-aided biological threat creation,” OpenAI, 2023. find
  34. J. M. Laurent, J. D. Janizek, M. Ruzo, M. M. Hinks, M. J. Hammerling, S. Narayanan, M. Ponnapati, A. D. White, and S. G. Rodriques, “LAB-Bench: Measuring capabilities of language models for biology research,” 2024. find
  35. G. Starace, O. Jaffe, D. Sherburn, J. Aung, J. S. Chan, L. Maksin, R. Dias, E. Mays, B. Kinsella, W. Thompson, J. Heidecke, A. Glaese, and T. Patwardhan, “PaperBench: Evaluating ai’s ability to replicate ai research.” https://openai.com/index/paperbench/, 2025. ↗