Leveraging large language models for predictive chemistry

Paper · 2024 · Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega‐Guerrero et al.

Summary

Chemistry datasets are usually small, so researchers tend to build bespoke machine learning models with hand-crafted chemical descriptors. This requires specialist skill. Jablonka and colleagues test a simpler route. They take GPT-3, a general-purpose LLM not trained specifically on chemistry, and fine-tune it on chemical questions written as plain text and paired with their answers.

They benchmark this approach on three kinds of data:

molecules: HOMO–LUMO gaps, water solubility, photovoltaic performance
materials: alloys, metal–organic frameworks, polymers
reactions: yields of cross-coupling reactions

The tasks include classification, regression and inverse design. For inverse design, the question is reversed so that the model proposes a structure with a target property.

With little training data, the fine-tuned model usually equals or beats dedicated baselines and often needs fewer examples. Conventional models tend to catch up as datasets grow. Results stay strong across different molecular representations.

The authors argue that querying a fine-tuned LLM could become a cheap default baseline and a quick way to start new chemistry projects. Future specialized models would then need to outperform it.

Key points

On high-entropy alloy phase prediction (single vs. multi-phase), GPT-3 trained on about 50 examples performed on par with a random forest from prior work that used roughly 1,126 training points. Automatminer and CrabNet were also compared.
All main results use the smallest GPT-3 variant (ada) via the OpenAI API with one fixed setting: 8 epochs and a learning rate multiplier of 0.02. Prompts and hyperparameters were not optimized.
IUPAC names, SMILES and SELFIES all gave good results, and IUPAC names were frequently the best. This makes the approach accessible to non-specialists.
For inverse design, iterative self-fine-tuning on generated molecules (evaluated with xTB, starting from QMugs) shifted HOMO–LUMO gaps toward a 4.0 eV target within four rounds.
In-context learning without fine-tuning also worked, especially with the largest GPT-3 models and GPT-4. The open-source GPT-J-6B could be fine-tuned on consumer hardware using 8-bit quantization and LoRA. The code is released in the gptchem and chemlift repositories.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Institutions

🎓École Polytechnique Fédérale de LausanneLausanne ·

Switzerland56 works · 8,189 citations 🎓Friedrich Schiller University JenaJena ·

Germany4 works · 579 citations

Also on the byline: Helmholtz Institute Jena.

Fields, topics and concepts

📚 Shelves 6Computer Science · 19,503 works Computational Theory and Mathematics · 496 works Materials Science · 58 works Materials Chemistry · 58 works Chemistry · 6 works Physical and Theoretical Chemistry · 5 works

🧭 Topics 1Language models & pretraining · 4,145 works

Cite

Cite as Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega‐Guerrero and Berend Smit. 2024. Leveraging large language models for predictive chemistry. Nature Machine Intelligence.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1038/s42256-023-00788-1

@inproceedings{jablonka2024leveraging,
  title = {{Leveraging large language models for predictive chemistry}},
  author = {Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega‐Guerrero and Berend Smit},
  booktitle = {Nature Machine Intelligence},
  year = {2024},
  doi = {10.1038/s42256-023-00788-1},
  url = {https://doi.org/10.1038/s42256-023-00788-1}
}

References

5 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. 2. 3. Bommasani, R. et al. On the opportunities and risks of foundation models. Preprint at https://arxiv.org/abs/2108.07258 (2021). Vaswani, A. et al. Attention is all you need. Adv. Neural Inf. Process. Syst. https://proceedings.neurips.cc/paper/2017/file/ 3f5ee243547dee91fbd053c1c4a845aa-Paper… ↗
Hoffmann, J. et al. An empirical analysis of compute-optimal large language model training. Adv. Neural Inf. Process. Syst. 35, 30016–30030 (2022). 5. Brown, T. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020). 6. Edwards, C. N., Lai, T., Ros, K., Ho… — Galactica: A Large Language Model for Science
ICLR 2024 Conference, OpenReview https://openreview.net/forum?id=X7nz6ljg9Y (2023). 27. Schwaller, P. et al. Molecular transformer: a model for uncertainty-calibrated chemical reaction prediction. ACS Cent. Sci. 5, 1572–1583 (2019). 28. Winter, B., Winter, C., Schilling, J. & Bardow, A. A smile is a… ↗
RDKit: Open-source Cheminformatics; (2023) http://www.rdkit.org 64. Preuer, K., Renz, P., Unterthiner, T., Hochreiter, S. & Klambauer, G. Fréchet ChemNet distance: a metric for generative models for molecules in drug discovery. J. Chem. Inf. Model. 58, 1736–1741 (2018). 65. Probst, D. & Reymond, J.-… ↗
Author contributions K.M.J. developed the machine learning approach with feedback from P.S. and B.S. K.M.J. and B.S. wrote the article. A.O.-G. contributed to the density functional theory calculations. Funding Open access funding provided by EPFL Lausanne. Competing interests The authors declare no… — [doi:10.1038/s42256-023-00788-1](https://doi.org/10.1038/s42256-023-00788-1) find

Updated 3 November 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jablonka-2024-leveraging-large-language-models-predictive/