Elamur Research⌕
☰
home / papers / 2025 / OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discovery

OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discovery

Paper · 2025 · Vignesh Prabhakar, Md Amirul Islam, Adam Atanas et al.

Summary

General-purpose LLMs often lack the specialized vocabulary and multi-step reasoning that scientific problems require. Training a science model from scratch is very costly. The authors, from SES AI and NVIDIA, instead adapt LLaMA 3.1 70B in three stages:

  • Domain adaptive pretraining. They continue pretraining on about 35B tokens of scientific text, including research articles, preprints and textbooks. The result is OmniScience Base.
  • Instruction tuning. Supervised tuning on a mix of science tasks and general chat yields OmniScience Chat.
  • Reasoning distillation. Fine-tuning on reasoning traces originally produced by DeepSeek-R1 transfers stepwise thinking and gives OmniScience Reasoning.

The final model reaches 0.72 on GPQA Diamond. That is the best score among similarly sized models and is competitive with much larger reasoning systems. Ablations indicate that both the pretraining stage and the distillation stage are necessary. In a battery-research agent that grades candidate electrolyte solvents, the model clearly outperforms the original LLaMA. The work suggests that strong mid-sized domain-expert models can be built at modest extra compute.

Key points

  • Pretraining corpus. The corpus holds 35B tokens from arXiv, ChemRxiv, PubChem, Semantic Scholar, peer-reviewed papers and textbooks. PDFs were converted to text with Unstructured, near-duplicates were removed with MinHash plus LSH, and the text was filtered with NeMo Curator.

  • Training data and cost. Alignment used roughly 160k task instructions and 90k chat instructions. Reasoning distillation used s1K-1.1, a math and science set. The authors estimate the extra pretraining compute at about 1% of the original.

  • Benchmark result. OmniScience Reasoning scores 0.72 on GPQA Diamond, the top result among models of comparable size.

  • Battery agent setup. OmniScience Reasoning acts as the generator and GPT-o1 as the critic, supported by RAG over textbooks and a short-term memory. The models graded about 700 molecules, and the ranking was judged on how well it placed 71 known good electrolyte solvents.

  • Battery agent results.

    ModelMean GradeMean RankHits@50
    LLaMA 3.1 70B6.4/10166.7012/50
    OmniScience Chat7/10153.4615/50
    OmniScience Reasoning8/1081.2632/50

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 worksMaterials Science · 58 worksMaterials Chemistry · 58 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Vignesh Prabhakar, Md Amirul Islam, Adam Atanas, Yao-Ting Wang, Joah Han, Aastha Jhunjhunwala, Rucha Apte, Robert Clark, Kang Xu, Zihan Wang and Kai Liu. 2025. OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discovery. arXiv:2503.17604.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.17604 · arXiv 2503.17604

@misc{prabhakar2025omniscience,
  title = {{OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discovery}},
  author = {Vignesh Prabhakar, Md Amirul Islam, Adam Atanas, Yao-Ting Wang, Joah Han, Aastha Jhunjhunwala, Rucha Apte, Robert Clark, Kang Xu, Zihan Wang and Kai Liu},
  year = {2025},
  doi = {10.48550/arxiv.2503.17604},
  eprint = {2503.17604},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.17604}
}

References

49 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716–23736, 2022.
  2. Anthropic. Claude 3.5 sonnet. https://www.anthropic.com/, 2023. Large language model. Accessed: 2025-03-17. ↗
  3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023.
  4. Elliot Bolton, Abhinav Venigalla, Michihiro Yasunaga, David Hall, Betty Xiong, Tony Lee, Roxana Daneshjou, Jonathan Frankle, Percy Liang, Michael Carbin, et al. Biomedlm: A 2.7 b parameter language model trained on biomedical text. arXiv preprint arXiv:2403.18421, 2024.
  5. Sébastien Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Lisa Anne Hendricks, Catalina Dadarla… find
  6. Andrei Z Broder. On the resemblance and containment of documents. In Proceedings. Compression and Complexity of SEQUENCES 1997 (Cat. No. 97TB100171), pp. 21–29. find
  7. Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. Chemberta: large-scale self-supervised pretraining for molecular property prediction. arXiv preprint arXiv:2010.09885, 2020. find
  8. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.
  9. Colin B. Clement, Matthew Bierbaum, Kevin P. O’Keeffe, and Alexander A. Alemi. On the use of arxiv as a dataset, 2019. find
  10. Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 35:16344–16359, 2022. find
  11. José Cassio dos Santos Junior, Rachel Hu, Richard Song, and Yunfei Bai. Domain-driven llm development: Insights into rag and fine-tuning practices. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 6416–6417, 2024. find
  12. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  13. Yuyuan Feng, Guosheng Hu, and Zhihong Zhang. Gpt4battery: An llm-driven framework for adaptive state of health estimation of raw li-ion batteries. arXiv preprint arXiv:2402.00068, 2024. — [arXiv:2402.00068](https://arxiv.org/abs/2402.00068) find
  14. Nathan Frey, Ryan Soklaski, Simon Axelrod, Siddharth Samsi, Rafael Gomez-Bombarelli, Connor Coley, and Vijay Gadepally. Chemgpt: Neural scaling of deep chemical models. ChemRxiv, 2022. doi: 10.26434/chemrxiv-2022-3s512. URL https://chemrxiv.org/engage/chemrxiv/ article-details/63c6e3a2d53b6c4e8b7e6e… — [doi:10.26434/chemrxiv-2022-3s512](https://doi.org/10.26434/chemrxiv-2022-3s512) ↗
  15. Aristides Gionis, Piotr Indyk, and Rajeev Motwani. Similarity search in high dimensions via hashing. In Proceedings of the 25th International Conference on Very Large Data Bases (VLDB), pp. 518–529. VLDB Endowment, 1999. find
  16. Frédéric Guillaume and Jacques Rougemont. Nemo: an evolutionary and population genetics programming framework. Bioinformatics, 22(20):2556–2557, 2006. find
  17. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
  18. Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. Don’t stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964, 2020. find
  19. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  20. Gautier Izacard, Lucas Hosseini, Fabio Petroni, Patrick Lewis, Ledell Wu, Wen tau Yih, Sonal Gupta, Nicola De Cao, Antoine Bordes, Stuart Shieber, Sebastian Riedel, and Douwe Kiela. Atlas: Few-shot learning with retrieval augmented language models, 2022. arXiv preprint arXiv:2208.03299.
  21. L. L. Kiessling, L. E. Fernandez, A. P. Alivisatos, and P. S. Weiss. Chemrxiv: A chemistry preprint server. ACS Nano, 10(10):9053–9054, 2016. find
  22. Sunghwan Kim, Paul A Thiessen, Evan E Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A Shoemaker, et al. Pubchem substance and compound databases. Nucleic acids research, 2016. find
  23. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
  24. Oleksii Kuchaiev, Jason Li, Huyen Nguyen, Oleksii Hrinchuk, Ryan Leary, Boris Ginsburg, Samuel Kriman, Stanislav Beliaev, Vitaly Lavrukhin, Jack Cook, et al. Nemo: a toolkit for building ai applications using neural modules. arXiv preprint arXiv:1909.09577, 2019.
  25. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks, 2020. arXiv preprint arXiv:2005.11401.
  26. Juncai Li and Xiaofei Jiang. Mol-bert: An effective molecular representation with bert for molecular property prediction. Complexity, pp. 1–11, 2021. doi: 10.1155/2021/7181815. URL https://doi.org/10.1155/2021/7181815. — [doi:10.1155/2021/7181815](https://doi.org/10.1155/2021/7181815) ↗
  27. Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, et al. Chipnemo: Domain-adapted llms for chip design. arXiv preprint arXiv:2311.00176, 2023. — [arXiv:2311.00176](https://arxiv.org/abs/2311.00176) find
  28. Wei Liu, Mark Johnson, Li Zhang, Raj Kumar, and Xiaoming Chen. Grok 3: A 2700b parameter model for advanced reasoning. https://example.com/grok3, 2025. ↗
  29. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4969–4983, Online, July 2020. Association for Computational Linguistics. doi: 10.1865…
  30. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393, 2025.
  31. OpenAI. ChatGPT: Optimizing language models for dialogue. chatgpt/, November 2022. [Accessed: 2023-03-04]. https://openai.com/blog/ ↗
  32. OpenAI. ChatGPT: Optimizing language models for dialogue. chatgpt/, 2024. https://openai.com/blog/ ↗
  33. OpenAI. ChatGPT: Optimizing language models for dialogue. chatgpt/, 2025. https://openai.com/blog/ ↗
  34. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
  35. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99–106, 2021. find
  36. Junhong Shen, Neil Tenenholtz, James Brian Hall, David Alvarez-Melis, and Nicolo Fusi. Tag-llm: Repurposing general-purpose llms for specialized domains. arXiv preprint arXiv:2402.05140, 2024. — [arXiv:2402.05140](https://arxiv.org/abs/2402.05140) find
  37. Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-lm: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv:1909.08053, 2019. find
  38. Yingheng Tang, Wenbin Xu, Jie Cao, Jianzhu Ma, Weilu Gao, Steve Farrell, Benjamin Erichson, Michael W Mahoney, Andy Nonaka, and Zhi Yao. Matterchat: A multi-modal llm for material science. arXiv preprint arXiv:2502.13107, 2025. — [arXiv:2502.13107](https://arxiv.org/abs/2502.13107) find
  39. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022.
  40. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023.
  41. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2025.
  42. Unstructured.IO. Unstructured: A python library for processing unstructured data. https://github.com/Unstructured-IO/unstructured, 2022. Accessed: February 27, 2025. ↗
  43. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. Helpsteer2: Open-source dataset for training top-performing reward models, 2024.
  44. Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gideon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023.
  45. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115, 2024. find
  46. Aybars Yunusoglu, Dexter Le, Karn Tiwari, Murat Isik, and I Dikmen. Battery state of health estimation using llm framework. arXiv preprint arXiv:2501.18123, 2025. — [arXiv:2501.18123](https://arxiv.org/abs/2501.18123) find
  47. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019. find
  48. Di Zhang, Yutong Li, Yixuan Li, et al. Chemllm: A chemical large language model. https://example.com/chemllm, 2024. Preprint or manuscript in preparation. ↗
  49. Qiang Zhang, Keyan Ding, Tianwen Lv, Xinda Wang, Qingyu Yin, Yiwen Zhang, Jing Yu, Yuhao Wang, Xiaotong Li, Zhuoyi Xiang, et al. Scientific large language models: A survey on biological & chemical domains. ACM Computing Surveys, 57(6):1–38, 2025.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/prabhakar-2025-omniscience-domain-specialized-llm-scientific/