Elamur Research⌕
☰
home / papers / 2024 / ChemLLM: A Chemical Large Language Model

ChemLLM: A Chemical Large Language Model

Paper · 2024 · Di Zhang, Wei Liu, Qian Tan et al.

Summary

General-purpose LLMs are increasingly used for chemistry, but none had been built specifically for the field. The authors identify two main obstacles. Chemical knowledge mostly sits in structured databases that do not suit conversational training. There was also no fair benchmark covering the main chemistry tasks. To address this, they built ChemData, about 7M synthetic instruction Q&A pairs. These were produced with templates that turn records from resources such as PubChem, ChEMBL and USPTO into dialogue. They also built ChemBench, a multiple-choice test of nine molecule- and reaction-related tasks. Multiple-choice scoring was chosen so that output style cannot inflate results the way BLEU or ROUGE can. ChemLLM starts from InternLM2-Base-7B and is tuned in two stages: first on general dialogue data, then on chemistry data mixed with the general data. In 5-shot ChemBench evaluation it clearly beats open models of similar size and GPT-3.5, and edges out GPT-4 on most tasks. It also keeps strong general skills. The work offers a reusable recipe for turning structured scientific knowledge into conversational domain models.

Key points

  • ChemData: roughly 7M instruction Q&A pairs drawn from PubChem, ChEMBL, ChEBI, ZINC, USPTO, ORDerly, LibreTexts Chemistry, Wikipedia, Wikidata and others. The data is split into three groups:
    • molecule tasks: Name Conversion, Caption2Mol, Mol2Caption and property prediction;
    • reaction tasks: retrosynthesis, product, yield, temperature and solvent prediction;
    • other domain-specific knowledge.
  • ChemBench: 4,100 single-answer multiple-choice questions across nine tasks. It has been contributed to the OpenCompass evaluation platform.
  • Training pipeline:
    • Stage 1 tunes InternLM2-Base-7B on Multi-Corpus, 1.7M general Q&A pairs from Hugging Face, producing InternLM2-Chat-7B.
    • Stage 2 tunes on ChemData mixed with Multi-Corpus, so chemistry is added without losing general ability.
  • Chemistry results: ChemLLM scores above GPT-4 on six of nine ChemBench tasks and just below it on the other three. LLaMA-2 scores about 25 per task, close to random guessing.
  • General results: evaluated on MMLU, C-Eval, GSM8K and C-MHChem, a new set of school-level chemistry questions. ChemLLM outperforms similarly sized models on MMLU and C-Eval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China213 works · 22,907 citations🎓Shanghai Jiao Tong UniversityShanghai · China232 works · 20,305 citations🎓Chinese University of Hong KongHong Kong · China176 works · 17,536 citations🔬Shanghai Artificial Intelligence LaboratoryShanghai · China164 works · 13,544 citations🎓Hong Kong Polytechnic UniversityHong Kong · China109 works · 5,346 citations🎓Wuhan UniversityWuhan · China52 works · 1,456 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 worksMaterials Science · 58 worksMaterials Chemistry · 58 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Wanli Ouyang, Dongzhan Zhou, Shufei Zhang, Mao Su, Han-Sen Zhong and Yuqiang Li. 2024. ChemLLM: A Chemical Large Language Model. arXiv:2402.06852.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.06852 · arXiv 2402.06852

@misc{zhang2024chemllm,
  title = {{ChemLLM: A Chemical Large Language Model}},
  author = {Di Zhang, Wei Liu, Qian Tan, Jingdan Chen, Hang Yan, Yuliang Yan, Jiatong Li, Weiran Huang, Xiangyu Yue, Wanli Ouyang, Dongzhan Zhou, Shufei Zhang, Mao Su, Han-Sen Zhong and Yuqiang Li},
  year = {2024},
  doi = {10.48550/arxiv.2402.06852},
  eprint = {2402.06852},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.06852}
}

References

8 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 4 5 13 14 15 16 18 19 25 26 28 29 30 37 38 39 40 Devlin, J., Chang, M.-W., Lee, K. et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Preprint at https://doi.org/10.48550/arXiv.1810.04805 (2019). Brown, T., Mann, B., Ryder, N. et al. Language models are few-shot… ↗
  2. Nat. Med. (2024). Webb, T., Holyoak, K. J. & Lu, H. Emergent analogical reasoning in large language models. Nat. Hum. Behav. 7, 1526-1541 (2023). Truhn, D., Reis-Filho, J. S. & Kather, J. N. Large language models should be used as scientific reasoning engines, not knowledge databases. Nat. Med. 29,… find
  3. Kim, S., Chen, J., Cheng, T. et al. PubChem 2023 update. Nucleic Acids Research 51, D1373-D1380 (2023). Mendez, D., Gaulton, A., Bento, A. P. et al. ChEMBL: towards direct deposition of bioassay data. Nucleic Acids Research 47, D930–D940 (2019). Weininger, D. SMILES, a chemical language and informat… find
  4. Nat. Med. (2023). Singhal, K., Azizi, S., Tu, T. et al. Large language models encode clinical knowledge. Nature 620, 172-180 (2023). Hendrycks, D., Burns, C., Basart, S. et al. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations. (2021). Huang,…
  5. Marco, A. C., Graham, S. J. H. & Apple, K. The USPTO Patent Assignment Dataset: Descriptions and Analysis. SSRN Electronic Journal (2015). Wigh, D., Arrowsmith, J., Pomberger, A. et al. ORDerly: Datasets and benchmarks for chemical reaction data. In Advances in Neural Information Processing Systems.… find
  6. Reiter, E. A Structured Review of the Validity of BLEU. Computational Linguistics 44, 393-401 (2018). Lin, C.-Y. ROUGE: A Package for Automatic Evaluation of Summaries. In Association for Computational Linguistics. 74– 81 (2004). Guo, T., Guo, K., Nan, B. et al. What can Large Language Models do in…
  7. NeurIPS (2023). Contributors, O. OpenCompass: A Universal Evaluation Platform for Foundation Models. https://github.com/opencompass/opencompass (2023). Cai, Z., Cao, M., Chen, H. et al. InternLM2 Technical Report. Preprint at https://doi.org/10.48550/arXiv.2403.17297 (2024). https://huggingface.co/.… ↗
  8. ChemLLM: A Chemical Large Language Model Zhang et al. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2024-chemllm-chemical-large-language-model/