Elamur Research⌕
☰
home / papers / 2023 / DARWIN Series: Domain Specific Large Language Models for Natural Science

DARWIN Series: Domain Specific Large Language Models for Natural Science

Paper · 2023 · Tong Xie, Yuwei Wan, Wei Huang et al.

Summary

General-purpose instruction-tuned LLMs perform poorly on natural science. Fine-tuned GPT models are closed and must be adapted on OpenAI's paid servers. Open alternatives are usually tuned on instructions written by GPT-3.5/GPT-4, which can reduce factual accuracy. The authors address this with DARWIN, a family of open models built on LLaMA-7B for physics, chemistry and materials science.

The training data combine three sources:

  • the SciQ exam set,
  • question–answer pairs drawn from scientific literature,
  • public FAIR datasets of material and molecular properties.

To create literature-based instructions cheaply, they introduce a Scientific Instruction Generation (SIG) model. It turns paper text into instructions without manual extraction or knowledge graphs.

Training happens in stages. A knowledge-oriented model (DARWIN-BASE) comes first. A material and device prediction model (DARWIN-MDP), tuned on many tasks jointly, follows. DARWIN-BASE reaches 96.9% on SciQ. DARWIN-MDP wins most regression and classification benchmarks and is roughly on par with specialised ML tools.

The work shows that a compact open model grounded in curated science data can compete with closed or task-specific systems while keeping costs low.

Key points

  • Model and data: the base model is LLaMA-7B, tuned on more than 60,000 instruction data points. Code is released at github.com/MasterAI-EAM/Darwin.
  • Literature corpus:
    • About 6 million English full-text papers from Web of Science in materials-science-related fields, from publishers such as Elsevier, Springer Nature, ACS and RSC.
    • Highly cited papers were fetched with SciCrawler. Papers with overly similar topics were filtered out using cosine similarity of MatBert embeddings.
    • Long inputs are split with "To be continued" tokens to work around LLaMA's input length limit.
  • SciQ (13,679 questions): DARWIN-BASE scores 96.9%. For comparison, Pythia 6.9B scores 95.2% five-shot, and zero-shot scores are 89.2% for BLOOM-7B and 90.1% for OPT-6.7B.
  • DARWIN-MDP results:
    • F1 of 97.2 on Pei (gptchem: 79.2) and 93.3 on waterStability.
    • MAE of 0.54 on OPV.
    • MODNet still does better on steels (MAE 88 vs 125.97) and matbench_glass (F1 97.5 vs 88.0).
  • Training findings: joint multi-task tuning (LLaMA-all) beats per-task tuning (LLaMA-single), and an earlier science-knowledge stage adds further gains.
  • Stated limitations: few QA pairs, use of several separate models instead of one, and an evaluation limited to multiple-choice questions.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓The University of MelbourneMelbourne · Australia102 works · 22,618 citations🎓The University of SydneySydney · Australia49 works · 68,350 citations🎓City University of Hong KongHong Kong · China59 works · 3,480 citations🎓UNSW SydneySydney · Australia36 works · 1,509 citations

Also on the byline: Cooperative Trials Group for Neuro-Oncology.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksMaterials Science · 31 worksMaterials Chemistry · 31 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Tong Xie, Yuwei Wan, Wei Huang, Zhenyu Yin, Yixuan Liu, Shaozhou Wang, Qingyuan Linghu, Chunyu Kit, Clara Grazian, Wenjie Zhang, Imran Razzak and Bram Hoex. 2023. DARWIN Series: Domain Specific Large Language Models for Natural Science. arXiv:2308.13565.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.13565 · arXiv 2308.13565

@misc{xie2023darwin,
  title = {{DARWIN Series: Domain Specific Large Language Models for Natural Science}},
  author = {Tong Xie, Yuwei Wan, Wei Huang, Zhenyu Yin, Yixuan Liu, Shaozhou Wang, Qingyuan Linghu, Chunyu Kit, Clara Grazian, Wenjie Zhang, Imran Razzak and Bram Hoex},
  year = {2023},
  doi = {10.48550/arxiv.2308.13565},
  eprint = {2308.13565},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.13565}
}

References

48 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. L Weston, V Tshitoyan, J Dagdelen, O Kononova, A Trewartha, K A Persson, G Ceder, and A Jain. Named Entity Recognition and Normalization Applied to Large-Scale Information Extraction from the Materials Science Literature. Journal of Chemical Information and Modeling, 59(9):3692–3702, sep 2019. find
  2. Amalie Trewartha, Nicholas Walker, Haoyan Huo, Sanghoon Lee, Kevin Cruse, John Dagdelen, Alexander Dunn, Kristin A Persson, Gerbrand Ceder, and Anubhav Jain. Quantifying the advantage of domain-specific pre-training on named entity recognition tasks in materials science. Patterns, 3(4):100488, 2022. find
  3. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, and Alex Ray. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  4. Tong Xie, Yuwei Wan, Wei Huang, Yufei Zhou, Yixuan Liu, Qingyuan Linghu, Shaozhou Wang, Chunyu Kit, Clara Grazian, Wenjie Zhang, and Bram Hoex. Large language models as master key: Unlocking the secrets of materials science with gpt, 2023. find
  5. Kevin Maik Jablonka, Philippe Schwaller, and Andres Ortega-guerrero. Is GPT-3 all you need for low-data discovery in chemistry ? ChemRxiv, pages 1–32, 2023. find
  6. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, and Faisal Azhar. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
  7. Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, and Kranthi Kiran GV. RWKV: Reinventing RNNs for the Transformer Era. arXiv preprint arXiv:2305.13048, 2023. find
  8. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-following llama model. https://github.com/ tatsu-lab/stanford_alpaca, 2023. ↗
  9. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. find
  10. OpenAI. Gpt-4 technical report, 2023.
  11. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-Instruct: Aligning Language Model with Self Generated Instructions. arXiv preprint arXiv:2212.10560, 2022. find
  12. Haochun Wang, Chi Liu, Nuwa Xi, Zewen Qiang, Sendong Zhao, Bing Qin, and Ting Liu. Huatuo: Tuning llama model with chinese medical knowledge. arXiv preprint arXiv:2304.06975, 2023.
  13. Mark D Wilkinson, Michel Dumontier, IJsbrand Jan Aalbersberg, Gabrielle Appleton, Myles Axton, Arie Baak, Niklas Blomberg, Jan-Willem Boiten, Luiz Bonino da Silva Santos, Philip E Bourne, Jildau Bouwman, Anthony J Brookes, Tim Clark, Mercè Crosas, Ingrid Dillo, Olivier Dumon, Scott Edmunds, Chris T… find
  14. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
  15. Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  16. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  17. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv e-prints, pages arXiv–2204, 2022. find
  18. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Super-naturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. arXiv preprint arXiv:2204.07705,… find
  19. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. find
  20. Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse of recursion: Training on generated data makes models forget. arXiv preprint arxiv:2305.17493, 2023.
  21. Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676, 2019.
  22. Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23, 2021.
  23. Amalie Trewartha, Nicholas Walker, Haoyan Huo, Sanghoon Lee, Kevin Cruse, John Dagdelen, Alexander Dunn, Kristin A Persson, Gerbrand Ceder, and Anubhav Jain. Quantifying the advantage of domain-specific pre-training on named entity recognition tasks in materials science. Patterns, 3(4), 2022. find
  24. Yanrong Ji, Zhihan Zhou, Han Liu, and Ramana V Davuluri. Dnabert: pre-trained bidirectional encoder representations from transformers model for dna-language in genome. Bioinformatics, 37(15):2112–2120, 2021. find
  25. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, and Amanda Askell. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  26. Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Pmc-llama: Further finetuning llama on medical papers. arXiv preprint arXiv:2304.14454, 2023.
  27. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al. Large language models encode clinical knowledge. arXiv preprint arXiv:2212.13138, 2022.
  28. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085, 2022.
  29. Andres M Bran, Sam Cox, Andrew D White, and Philippe Schwaller. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376, 2023.
  30. Renqian Luo, Liai Sun, Yingce Xia, Tao Qin, Sheng Zhang, Hoifung Poon, and Tie-Yan Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23(6), 2022. find
  31. Peter Clark. Elementary school science and math tests as a driver for ai: Take the aristo challenge! In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, pages 4019–4021, 2015. find
  32. Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209, 2017.
  33. Matthias Scheffler, Martin Aeschlimann, Martin Albrecht, Tristan Bereau, Hans-Joachim Bungartz, Claudia Felser, Mark Greiner, Axel Groß, Christoph T Koch, Kurt Kremer, et al. Fair data enabling new horizons for materials research. Nature, 604(7907):635–642, 2022. find
  34. Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-instructions: A large-scale biomolecular instruction dataset for large language models. arXiv preprint arXiv:2306.08018, 2023.
  35. Alexander Dunn, Qi Wang, Alex Ganose, Daniel Dopp, and Anubhav Jain. Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm. npj Computational Materials, 6(1):138, 2020. find
  36. Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega-Guerrero, and Berend Smit. Is gpt-3 all you need for low-data discovery in chemistry? 2023. find
  37. Anna Gaulton, Louisa J Bellis, A Patricia Bento, Jon Chambers, Mark Davies, Anne Hersey, Yvonne Light, Shaun McGlinchey, David Michalovich, and Bissan Al-Lazikani. ChEMBL: a large-scale bioactivity database for drug discovery. Nucleic acids research, 40(D1):D1100–D1107, 2012. find
  38. Xixi Qin, Xiaochen Du, Sampreeti Bhattacharya, Connor Clayton, Jun Hu, Manoj Jana, Svenja Janke, Rebecca Lau, Raul Laasner, and Andrew Levin. HybriD 3: Materials Property Database for Hybrid Organic-Inorganic Perovskites. In APS March Meeting Abstracts, volume 2021, pages X54–007, 2021. find
  39. John S Delaney. ESOL: estimating aqueous solubility directly from molecular structure. Journal of chemical information and computer sciences, 44(3):1000–1005, 2004. find
  40. Osbel Almora, Derya Baran, Guillermo C Bazan, Carlos I Cabrera, Sule Erten-Ela, Karen Forberich, Fei Guo, Jens Hauch, Anita W Y Ho-Baillie, and T Jesper Jacobsson. Device performance of emerging photovoltaic materials (Version 3). Advanced Energy Materials, 13(1):2203313, 2023. find
  41. Masahiro Sakurai, Renhai Wang, Timothy Liao, Chao Zhang, Huaijun Sun, Yang Sun, Haidi Wang, Xin Zhao, Songyou Wang, and Balamurugan Balasubramanian. Discovering rare-earth-free magnetic materials through the development of a database. Physical review materials, 4(11):114408, 2020. find
  42. Ya Zhuo, Aria Mansouri Tehrani, and Jakoah Brgoch. Predicting the band gaps of inorganic solids by machine learning. The journal of physical chemistry letters, 9(7):1668–1673, 2018. find
  43. Yoshiyuki Kawazoe, J-Z Yu, A-P Tsai, and T Masumoto. Nonequilibrium phase diagrams of ternary amorphous alloys, 1997. find
  44. Seyed Mohamad Moosavi, Aditya Nandy, Kevin Maik Jablonka, Daniele Ongari, Jon Paul Janet, Peter G Boyd, Yongjin Lee, Berend Smit, and Heather J Kulik. Understanding the diversity of the metal-organic framework ecosystem. Nature communications, 11(1):1–10, 2020. find
  45. Shinji Nagasawa, Eman Al-Naamani, and Akinori Saeki. Computer-aided screening of conjugated polymers for organic solar cell: classification by random forest. The Journal of Physical Chemistry Letters, 9(10):2639–2646, 2018. find
  46. Zongrui Pei, Junqi Yin, Jeffrey A Hawk, David E Alman, and Michael C Gao. Machine-learning informed prediction of high-entropy solid solution formation: Beyond the Hume-Rothery rules. npj Computational Materials, 6(1):50, 2020. find
  47. Daniel Hickox-Young, Danilo Puggioni, and James M Rondinelli. Polar metals taxonomy for materials classification and discovery. Physical Review Materials, 7(1):10301, 2023. find
  48. Rohit Batra, Carmen Chen, Tania G Evans, Krista S Walton, and Rampi Ramprasad. Prediction of water stability of metal–organic frameworks using machine learning. Nature Machine Intelligence, 2(11):704–710, 2020. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xie-2023-darwin-series-domain-specific-large/