Elamur Research⌕
☰
home / papers / 2024 / LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation

LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation

Paper · 2024 · Yuan Chiang, Elvis Hsieh, Chia-Hong Chou et al.

Summary

LLMs often produce confident but false statements, which is a serious obstacle to using them in materials science. Fine-tuning them on domain data is hard to reproduce and cannot keep pace with new results. LLaMP is a retrieval-augmented generation (RAG) framework that grounds model answers in the Materials Project (MP) database without any fine-tuning. A GPT-4 "supervisor" ReAct agent routes each request to specialised "assistant" ReAct agents. Each assistant handles a single MP API endpoint or tool, such as arXiv or Wikipedia search, a Python REPL, or atomate2 simulation workflows. This split keeps context-window use low, and assistants can repair their own failed API calls by learning from the errors returned.

The authors also introduce a self-consistency metric that combines uncertainty and confidence estimates. On this metric, LLaMP reduces the systematic errors vanilla LLMs make on bulk moduli, bandgaps, formation energies and magnetic ordering. The system can also edit crystal structures and run molecular dynamics from plain-language instructions, which makes computational materials work more accessible and less prone to hallucination.

Key points

  • Two-level architecture: a supervisor ReAct agent on GPT-4 delegates to tool-specific assistant agents, built on a standard LangChain interface.
    • Assistant data sources include MP endpoints for structures, thermodynamics, electronic, magnetic, elastic and dielectric/piezoelectric properties.
    • They also cover robocrystallographer descriptions (about 130K compounds) and synthesis recipes.
  • Crystal editing test: both systems were asked to insert a Li atom into diamond-cubic Si.
    • Vanilla GPT-3.5 produced a strongly distorted cell, with Si–Si bond length off by roughly 15% and cell volume by roughly 66%.
    • LLaMP's retrieved MP structure kept the lattice parameters correct. However, because of MP's coordinate convention, the Li atom ended up at a hexagonal rather than a tetrahedral interstitial site.
  • Language-driven simulation: LLaMP fetched a structure from MP, built a supercell and ran annealing MD (300 K → 800 K → 300 K) with MACE-MP-0 through atomate2. It then plotted the resulting temperature profile.
  • Limitations:
    • Results depend on the backbone LLM's function-calling ability; for example, it sometimes misreads the sign convention in sort_fields.
    • Results also depend on MP data quality: GGA underestimates bandgaps, antiferromagnetic configurations are incomplete, and structure coverage is not exhaustive.
  • Availability: code and a demo are available at github.com/chiang-yuan/llamp.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States175 works · 227,440 citations🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations🔬Lawrence Berkeley National LaboratoryBerkeley · United States3 works · 39,160 citations

Also on the byline: Foothill College.

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksMaterials Science · 26 worksMaterials Chemistry · 26 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksInformation retrieval & search · 473 worksEfficient models & compression · 242 works

Cite

Cite as Yuan Chiang, Elvis Hsieh, Chia-Hong Chou and Janosh Riebesell. 2024. LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation. arXiv:2401.17244.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.17244 · arXiv 2401.17244

@misc{chiang2024llamp,
  title = {{LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation}},
  author = {Yuan Chiang, Elvis Hsieh, Chia-Hong Chou and Janosh Riebesell},
  year = {2024},
  doi = {10.48550/arxiv.2401.17244},
  eprint = {2401.17244},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.17244}
}

References

45 references, 11 of them with a card of ours. ↗ leads to the work.

  1. Y. Bang et al. A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity, Nov. 2023. URL http://arxiv.org/abs/2302.04023. arXiv:2302.04023 [cs]. ↗
  2. I. Batatia et al. A foundation model for atomistic materials chemistry, Dec. 2023. URL http://arxiv.org/abs/2401.00096. arXiv:2401.00096 [cond-mat, physics:physics]. — [arXiv:2401.00096](https://arxiv.org/abs/2401.00096) ↗
  3. D. A. Boiko, R. MacKnight, B. Kline, and G. Gomes. Autonomous chemical research with large language models. Nature, 624(7992):570–578, Dec. 2023. ISSN 1476-4687. doi: 10.1038/s41586-023-06792-0. URL https://www.nature.com/articles/ s41586-023-06792-0. Number: 7992 Publisher: Nature Publishing Group. ↗
  4. P. Borlido et al. Exchange-correlation functionals for band gaps of solids: benchmark, reparametrization and machine learning. npj Computational Materials, 6(1):1–17, July 2020. ISSN 2057-3960. doi: 10.1038/s41524-020-00360-0. URL https://www.nature.com/ articles/s41524-020-00360-0. Number: 1 Publis… — [doi:10.1038/s41524-020-00360-0](https://doi.org/10.1038/s41524-020-00360-0) ↗
  5. K. Cobbe et al. Training Verifiers to Solve Math Word Problems, Nov. 2021. URL http://arxiv.org/abs/2110.14168. arXiv:2110.14168 [cs]. ↗
  6. J. Dagdelen et al. Structured information extraction from scientific text with large language models. Nature Communications, 15(1):1418, Feb. 2024. ISSN 2041-1723. doi: 10.1038/ s41467-024-45563-x. URL https://www.nature.com/articles/s41467-024-45563-x. Publisher: Nature Publishing Group. ↗
  7. N. Frey et al. Neural Scaling of Deep Chemical Models, May 2022. URL https://chemrxiv.org/engage/chemrxiv/article-details/627bddd544bdd532395fb4b5. ↗
  8. A. M. Ganose and A. Jain. Robocrystallographer: automated crystal structure text descriptions and analysis. MRS Communications, 9(3):874–881, Sept. 2019. ISSN 2159-6867. doi: 10. 1557/mrc.2019.94. URL https://doi.org/10.1557/mrc.2019.94. — [doi:10.1557/mrc.2019.94](https://doi.org/10.1557/mrc.2019.94) ↗
  9. N. Gruver et al. Fine-Tuned Language Models Generate Stable Inorganic Materials as Text. Nov. 2023. URL https://openreview.net/forum?id=0r5DE2ZSwJ. ↗
  10. T. Gupta, M. Zaki, N. M. A. Krishnan, and Mausam. MatSciBERT: A materials domain language model for text mining and information extraction. npj Computational Materials, 8(1):1–11, May 2022. ISSN 2057-3960. doi: 10.1038/s41524-022-00784-w. URL https://www.nature.com/articles/s41524-022-00784-w. Numbe… ↗
  11. D. Hendrycks et al. Measuring Massive Multitask Language Understanding, Jan. 2021. URL http://arxiv.org/abs/2009.03300. arXiv:2009.03300 [cs]. ↗
  12. K. M. Jablonka, P. Schwaller, A. Ortega-Guerrero, and B. Smit. Leveraging large language models for predictive chemistry. Nature Machine Intelligence, 6(2):161–169, Feb. 2024. ISSN 2522-5839. doi: 10.1038/s42256-023-00788-1. URL https://www.nature.com/articles/ s42256-023-00788-1. Publisher: Nature… ↗
  13. A. Jain et al. Commentary: The Materials Project: A materials genome approach to accelerating materials innovation. APL Materials, 1(1):011002, July 2013. ISSN 2166-532X. doi: 10.1063/1.4812323. URL https://doi.org/10.1063/1.4812323. — [doi:10.1063/1.4812323](https://doi.org/10.1063/1.4812323) ↗
  14. O. Kononova et al. Text-mined dataset of inorganic materials synthesis recipes. Scientific Data, 6(1):203, Oct. 2019. ISSN 2052-4463. doi: 10.1038/s41597-019-0224-1. URL https://www.nature.com/articles/s41597-019-0224-1. Number: 1 Publisher: Nature Publishing Group. — [doi:10.1038/s41597-019-0224-1](https://doi.org/10.1038/s41597-019-0224-1) ↗
  15. G. Kresse and J. Furthmüller. Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set. Physical Review B, 54(16):11169–11186, Oct. 1996. doi: 10.1103/PhysRevB.54.11169. URL https://link.aps.org/doi/10.1103/PhysRevB.54. 11169. Publisher: American Physical Soc… — [doi:10.1103/physrevb.54.11169](https://doi.org/10.1103/physrevb.54.11169) ↗
  16. P. Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/hash/ 6b493230205f780e1bc26945df7481e5-Abstract.html. ↗
  17. P. Lewis et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Apr. 2021. URL http://arxiv.org/abs/2005.11401. arXiv:2005.11401 [cs] version: 4. ↗
  18. J. Li et al. True Composition and Structure of Hexagonal “YAlO3”, Actually Y3Al3O8CO3. Inorganic Chemistry, 54(3):837–844, Feb. 2015. ISSN 0020-1669. doi: 10.1021/ic502027k. URL https://doi.org/10.1021/ic502027k. Publisher: American Chemical Society. — [doi:10.1021/ic502027k](https://doi.org/10.1021/ic502027k) ↗
  19. P. Liang et al. Holistic Evaluation of Language Models, Oct. 2023. URL http://arxiv.org/ abs/2211.09110. arXiv:2211.09110 [cs]. ↗
  20. Y. Luo, C. Liu, and S. Ji. Towards Symmetry-Aware Generation of Periodic Materials, July 2023. URL https://arxiv.org/abs/2307.02707v2. — [arXiv:2307.02707](https://arxiv.org/abs/2307.02707) ↗
  21. A. M. Bran et al. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6(5):525–535, May 2024. ISSN 2522-5839. doi: 10.1038/s42256-024-00832-8. URL https://www.nature.com/articles/s42256-024-00832-8. Publisher: Nature Publishing Group. — [doi:10.1038/s42256-024-00832-8](https://doi.org/10.1038/s42256-024-00832-8) ↗
  22. A. Mallen et al. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories, July 2023. URL http://arxiv.org/abs/2212.10511. arXiv:2212.10511 [cs]. ↗
  23. A. L. Maximenko and E. A. Olevsky. Effective diffusion coefficients in solid-state sintering. Acta Materialia, 52(10):2953–2963, June 2004. ISSN 1359-6454. doi: 10.1016/ j.actamat.2004.02.042. URL https://www.sciencedirect.com/science/article/pii/ S1359645404001326. ↗
  24. A. Merchant et al. Scaling deep learning for materials discovery. Nature, pages 1–6, Nov. 2023. ISSN 1476-4687. doi: 10.1038/s41586-023-06735-9. URL https://www.nature.com/articles/s41586-023-06735-9. Publisher: Nature Publishing Group. ↗
  25. S. Miret and N. M. A. Krishnan. Are LLMs Ready for Real-World Materials Discovery?, Sept. 2024. URL http://arxiv.org/abs/2402.05200. arXiv:2402.05200 [cond-mat]. ↗
  26. S. Ouyang et al. Structured Chemistry Reasoning with Large Language Models, Feb. 2024. URL http://arxiv.org/abs/2311.09656. arXiv:2311.09656 [cs]. — [arXiv:2311.09656](https://arxiv.org/abs/2311.09656) ↗
  27. G. Pilania, J. E. Gubernatis, and T. Lookman. Multi-fidelity machine learning models for accurate bandgap predictions of solids. Computational Materials Science, 129:156– 163, Mar. 2017. ISSN 0927-0256. doi: 10.1016/j.commatsci.2016.12.004. URL https://www.sciencedirect.com/science/article/pii/S092… — [doi:10.1016/j.commatsci.2016.12.004](https://doi.org/10.1016/j.commatsci.2016.12.004) ↗
  28. M. P. Polak and D. Morgan. Extracting Accurate Materials Data from Research Papers with Conversational Language Models and Prompt Engineering, June 2023. URL http://arxiv.org/abs/2303.05352. arXiv:2303.05352 [cond-mat]. — [arXiv:2303.05352](https://arxiv.org/abs/2303.05352) ↗
  29. A. Radford et al. Learning Transferable Visual Models From Natural Language Supervision, Feb. 2021. URL http://arxiv.org/abs/2103.00020. arXiv:2103.00020 [cs]. ↗
  30. Y. Shao et al. Advances in molecular quantum chemistry contained in the Q-Chem 4 program package. Molecular Physics, 113(2):184–215, Jan. 2015. ISSN 0026-8976. doi: 10. 1080/00268976.2014.952696. URL https://www.tandfonline.com/doi/full/10.1080/ 00268976.2014.952696. Publisher: Taylor & Francis. ↗
  31. E. W. C. Spotte-Smith et al. A database of molecular properties integrated in the Materials Project, Oct. 2023. URL https://chemrxiv.org/engage/chemrxiv/article-details/ 651a3302ade1178b247ee6d1. ↗
  32. N. J. Szymanski et al. An autonomous laboratory for the accelerated synthesis of novel materials. Nature, 624(7990):86–91, Dec. 2023. ISSN 1476-4687. doi: 10.1038/ s41586-023-06734-w. URL https://www.nature.com/articles/s41586-023-06734-w. Number: 7990 Publisher: Nature Publishing Group. ↗
  33. P. K. Todd et al. Selectivity in Yttrium Manganese Oxide Synthesis via Local Chemical Potentials in Hyperdimensional Phase Space. Journal of the American Chemical Society, 143(37):15185–15194, Sept. 2021. ISSN 0002-7863. doi: 10.1021/jacs.1c06229. URL https://doi.org/10.1021/jacs.1c06229. Publisher:… — [doi:10.1021/jacs.1c06229](https://doi.org/10.1021/jacs.1c06229) ↗
  34. V. Tshitoyan et al. Unsupervised word embeddings capture latent knowledge from materials science literature. Nature, 571(7763):95–98, July 2019. ISSN 1476-4687. doi: 10.1038/ s41586-019-1335-8. URL https://www.nature.com/articles/s41586-019-1335-8. Number: 7763 Publisher: Nature Publishing Group. ↗
  35. W. Wang et al. Augmenting Language Models with Long-Term Memory, June 2023. URL http://arxiv.org/abs/2306.07174. arXiv:2306.07174 [cs]. ↗
  36. M. Wen et al. Chemical reaction networks and opportunities for machine learning. Nature Computational Science, 3(1):12–24, Jan. 2023. ISSN 2662-8457. doi: 10.1038/ s43588-022-00369-z. URL https://www.nature.com/articles/s43588-022-00369-z. Publisher: Nature Publishing Group. ↗
  37. M. Wen et al. An equivariant graph neural network for the elasticity tensors of all seven crystal systems. Digital Discovery, 3(5):869–882, May 2024. ISSN 2635-098X. doi: 10. 1039/D3DD00233K. URL https://pubs.rsc.org/en/content/articlelanding/2024/ dd/d3dd00233k. Publisher: RSC. ↗
  38. T. Xie et al. DARWIN Series: Domain Specific Large Language Models for Natural Science, Aug. 2023. URL http://arxiv.org/abs/2308.13565. arXiv:2308.13565 [cond-mat, physics:physics]. ↗
  39. Z. Xu, S. Jain, and M. Kankanhalli. Hallucination is Inevitable: An Innate Limitation of Large Language Models, Jan. 2024. URL http://arxiv.org/abs/2401.11817. arXiv:2401.11817 [cs]. ↗
  40. F. Yan et al. Berkeley Function Calling Leaderboard. 2024. URL https://gorilla.cs.berkeley.edu/blogs/8 berkeley function calling leaderboard.html. ↗
  41. S. Yang et al. Accurate Prediction of Experimental Band Gaps from Large Language Model-Based Data Extraction. Nov. 2023. URL https://openreview.net/forum?id=oRKWhmtUG6. ↗
  42. S. Yao et al. ReAct: Synergizing Reasoning and Acting in Language Models, Mar. 2023. URL http://arxiv.org/abs/2210.03629. arXiv:2210.03629 [cs]. ↗
  43. D. Zagorac et al. Recent developments in the Inorganic Crystal Structure Database: theoretical crystal structure data and related features. Journal of Applied Crystallography, 52 (5):918–925, Oct. 2019. ISSN 1600-5767. doi: 10.1107/S160057671900997X. URL https://journals.iucr.org/j/issues/2019/05/00… — [doi:10.1107/s160057671900997x](https://doi.org/10.1107/s160057671900997x) ↗
  44. C. Zeni et al. MatterGen: a generative model for inorganic materials design, Jan. 2024. URL http://arxiv.org/abs/2312.03687. arXiv:2312.03687 [cond-mat]. — [arXiv:2312.03687](https://arxiv.org/abs/2312.03687) ↗
  45. Z. Zheng et al. ChatGPT Chemistry Assistant for Text Mining and Prediction of MOF Synthesis. Journal of the American Chemical Society, 145(32):18048–18062, Aug. 2023. ISSN 0002-7863, 1520-5126. doi: 10.1021/jacs.3c05819. URL http://arxiv.org/abs/2306.11296. arXiv:2306.11296 [cond-mat, physics:physic… ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chiang-2024-llamp-large-language-model-made/