Elamur Research⌕
☰
home / papers / 2024 / Are LLMs Ready for Real-World Materials Discovery?

Are LLMs Ready for Real-World Materials Discovery?

Paper · 2024 · Santiago Miret and N M Anoop Krishnan

Summary

This perspective paper asks why large language models are still not dependable tools for materials science, even as they gain traction in chemistry and biology. The authors define two requirements for a materials-focused model (MatSci-LLM). It needs domain knowledge with reasoning tied to core principles, and it must support researchers reliably and interpretably. They then review failure cases drawn from an undergraduate-level question set by Zaki et al. (2024). In these cases, GPT-4 with chain-of-thought often states a correct rule but applies it wrongly, especially for crystal symmetry and numerical steps. The central argument is that progress depends mainly on large, high-quality, multi-modal datasets mined from peer-reviewed literature. Several obstacles stand in the way. Key information is scattered across tables, figures and cited papers. Materials also lack a systematic naming scheme comparable to IUPAC names. The paper proposes a six-step roadmap that runs from user query to lab execution, and it discusses effects on sustainability, inclusivity and policy. Its value lies in setting a shared agenda for domain-grounded scientific LLMs.

Key points

  • The evidence base is the benchmark of Zaki et al. (2024), which has 650 undergraduate-level materials questions. Commercial models such as GPT-4 and LLaMA-2 show incomplete materials knowledge and inconsistent reasoning on it, in both question-answering and code generation.
  • In one example, GPT-4 (CoT) chose crystal class 2/m instead of 4mm as piezoelectric and ferroelectric, wrongly calling 2/m polar. In another, it selected (100), (110), (200) as the first Bragg peaks for BCC iron, which contradicts its own even h+k+l rule; the correct answer is (110), (200), (220). Appendix C lists ten such cases.
  • The authors name three directions for progress:
    • hypothesis generation grounded in materials knowledge;
    • multi-modal datasets that cover tables, figures and inline formulas;
    • connections to actual simulation and laboratory tools, extending the chemistry agents described by Boiko et al. (2023) as well as those of Bran et al. (2023).
  • The roadmap has six steps: Materials Query → Data Retrieval → Materials Design → In-silico Evaluation → Experiment Planning → Experiment Execution.
  • The paper builds on earlier work, including domain-pretrained BERT models (analysed by Song et al., 2023a), ChemDataExtractor, and in-context LLM extraction (Dagdelen et al., 2024). It notes that labelled materials data remains scarce.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IntelSanta Clara · United States22 works · 43,146 citations🎓Indian Institute of Technology DelhiNew Delhi · India26 works · 6,995 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksMaterials Science · 58 worksMaterials Chemistry · 58 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Santiago Miret and N M Anoop Krishnan. 2024. Are LLMs Ready for Real-World Materials Discovery?. arXiv:2402.05200.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2402.05200 · arXiv 2402.05200

@misc{miret2024are,
  title = {{Are LLMs Ready for Real-World Materials Discovery?}},
  author = {Santiago Miret and N M Anoop Krishnan},
  year = {2024},
  doi = {10.48550/arxiv.2402.05200},
  eprint = {2402.05200},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2402.05200}
}

References

77 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. Antunes, L. M., Butler, K. T., and Grau-Crespo, R. Crystal structure generation with autoregressive large language modeling. arXiv preprint arXiv:2307.04340, 2023. — [arXiv:2307.04340](https://arxiv.org/abs/2307.04340) find
  3. Aroyo, M. I., Perez-Mato, J. M., Capillas, C., Kroumova, E., Ivantchev, S., Madariaga, G., Kirov, A., and Wondratschek, H. Bilbao crystallographic server: I. databases and crystallographic computing programs. Zeitschrift für Kristallographie-Crystalline Materials, 221(1):15–27, 2006. find
  4. BehnamGhader, P., Miret, S., and Reddy, S. Can retriever-augmented language models reason? the blame game between the retriever and the language model. In Bouamor, H., Pino, J., and Bali, K. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15492–15509, Singapore, De… find
  5. Boiko, D. A., MacKnight, R., Kline, B., and Gomes, G. Autonomous chemical research with large language models. Nature, 624(7992):570–578, 2023.
  6. Bran, A. M., Cox, S., White, A. D., and Schwaller, P. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376, 2023.
  7. Buehler, M. J. Generative retrieval-augmented ontologic graph and multiagent strategies for interpretive large language model-based materials design. ACS Engineering Au, 2023. find
  8. Buehler, M. J. Mechgpt, a language-based strategy for mechanics and materials modeling that connects knowledge across scales, disciplines, and modalities. Applied Mechanics Reviews, 76(2):021001, 2024. find
  9. Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  10. Computer, T. Redpajama: an open dataset for training large language models, 2023. URL https://github.com/ togethercomputer/RedPajama-Data. ↗
  11. Cox, S., Hammerling, M., Lála, J., Laurent, J., Rodriques, S., Rubashkin, M., and White, A. Wikicrow: Automating synthesis of human scientific knowledge. Cui, H., Wang, C., Maan, H., Pang, K., Luo, F., and Wang, B. scgpt: Towards building a foundation model for single-cell multi-omics using generat… find
  12. Dagdelen, J., Dunn, A., Lee, S., Walker, N., Rosen, A. S., Ceder, G., Persson, K. A., and Jain, A. Structured information extraction from scientific text with large language models. Nature Communications, 15(1):1418, 2024.
  13. Dahl, M., Magesh, V., Suzgun, M., and Ho, D. E. Large legal fictions: Profiling legal hallucinations in large language models. arXiv preprint arXiv:2401.01301, 2024.
  14. Dalla-Torre, H., Gonzalez, L., Mendoza Revilla, J., Lopez Carranza, N., Henryk Grywaczewski, A., Oteri, F., Dallago, C., Trop, E., Sirelkhatim, H., Richard, G., et al. The nucleotide transformer: Building and evaluating robust foundation models for human genomics. bioRxiv, pp. 2023–01, 2023. find
  15. Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. BERT: Pre-training of deep bidirectional transformers for language understanding. In Burstein, J., Doran, C., and Solorio, T. (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics…
  16. Fan, A., Gokkaya, B., Harman, M., Lyubarskiy, M., Sengupta, S., Yoo, S., and Zhang, J. M. Large language models for software engineering: Survey and open problems. arXiv preprint arXiv:2310.03533, 2023. — [arXiv:2310.03533](https://arxiv.org/abs/2310.03533) find
  17. Flam-Shepherd, D. and Aspuru-Guzik, A. Language models can generate molecules, materials, and protein binding sites directly in three dimensions as xyz, cif, and pdb files. arXiv preprint arXiv:2305.05708, 2023. — [arXiv:2305.05708](https://arxiv.org/abs/2305.05708) find
  18. Ganose, A. M. and Jain, A. Robocrystallographer: automated crystal structure text descriptions and analysis. MRS Communications, 9(3):874–881, 2019. find
  19. Gouldstone, A., Chollacoop, N., Dao, M., Li, J., Minor, A. M., and Shen, Y.-L. Indentation across size scales and disciplines: Recent developments in experimentation and modeling. Acta Materialia, 55(12):4015–4039, 2007. find
  20. Govindarajan, P., Miret, S., Rector-Brooks, J., Phielipp, M., Rajendran, J., and Chandar, S. Learning conditional policies for crystal design using offline reinforcement learning. In AI for Accelerated Materials Design - NeurIPS 2023 Workshop, 2023. URL https://openreview.net/forum?id=VbjD8w2ctG. ↗
  21. Groom, C. R., Bruno, I. J., Lightfoot, M. P., and Ward, S. C. The cambridge structural database. Acta Crystallographica Section B: Structural Science, Crystal Engineering and Materials, 72(2):171–179, 2016. find
  22. Gruver, N., Sriram, A., Madotto, A., Wilson, A. G., Zitnick, C. L., and Ulissi, Z. W. Fine-tuned language models generate stable inorganic materials as text. In AI for Accelerated Materials Design - NeurIPS 2023 Workshop, 2023. URL https://openreview.net/forum?id=0r5DE2ZSwJ. ↗
  23. Gupta, T., Zaki, M., Krishnan, N. A., and Mausam. Matscibert: A materials domain language model for text mining and information extraction. npj Computational Materials, 8(1):102, 2022. find
  24. Gupta, T., Zaki, M., Khatsuriya, D., Hira, K., Krishnan, N. M. A., and Mausam. DiSCoMaT: Distantly supervised composition extraction from tables in materials science articles. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Comput… find
  25. He, K., Mao, R., Lin, Q., Ruan, Y., Lan, X., Feng, M., and Cambria, E. A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics. arXiv preprint arXiv:2310.05694, 2023.
  26. Hellwich, K.-H., Hartshorn, R. M., Yerin, A., Damhus, T., and Hutton, A. T. Brief guide to the nomenclature of organic chemistry (iupac technical report). Pure and Applied Chemistry, 92(3):527–539, 2020. find
  27. Hira, K., Zaki, M., Sheth, D. B., ., M., and Krishnan, N. M. A. Reconstructing materials tetrahedron: Challenges in materials information extraction. In AI for Accelerated Materials Design - NeurIPS 2023 Workshop, 2023. URL https://openreview.net/forum?id=u6ndfkFRJC. ↗
  28. Hsu, C., Verkuil, R., Liu, J., Lin, Z., Hie, B., Sercu, T., Lerer, A., and Rives, A. Learning inverse folding from millions of predicted structures. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Le… find
  29. Huang, S. and Cole, J. M. Batterybert: A pretrained language model for battery database enhancement. Journal of Chemical Information and Modeling, 62(24):6365–6377, 2022. find
  30. Jablonka, K. M., Ai, Q., Al-Feghali, A., Badhwar, S., Bocarsly, J. D., Bran, A. M., Bringuier, S., Brinson, L. C., Choudhary, K., Circi, D., et al. 14 examples of how llms can transform materials science and chemistry: a reflection on a large language model hackathon. Digital Discovery, 2(5):1233–12… find
  31. Jensen, Z., Kwon, S., Schwalbe-Koda, D., Paris, C., Gómez-Bombarelli, R., Román-Leshkov, Y., Corma, A., Moliner, M., and Olivetti, E. A. Discovering relationships between osdas and zeolites through data mining and generative neural networks. ACS central science, 7(5):858–867, 2021. find
  32. Jiao, R., Huang, W., Lin, P., Han, J., Chen, P., Lu, Y., and Liu, Y. Crystal structure prediction by joint equivariant diffusion. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/ forum?id=DNdN26m2Jk. ↗
  33. Kanarik, K. J., Osowiecki, W. T., Lu, Y., Talukder, D., Roschewsky, N., Park, S. N., Kamon, M., Fried, D. M., and Gottscho, R. A. Human–machine collaboration for improving semiconductor process development. Nature, 616(7958):707–711, 2023. find
  34. Kasimuthumaniyan, S., Gosvami, N. N., and Krishnan, N. A. Towards understanding the scratchability in functional glasses. Ceramics International, 47(15):20821–20843, 2021. find
  35. Kearnes, S. M., Maser, M. R., Wleklinski, M., Kast, A., Doyle, A. G., Dreher, S. D., Hawkins, J. M., Jensen, K. F., and Coley, C. W. The open reaction database. Journal of the American Chemical Society, 143(45):18820–18826, 2021. find
  36. Kim, E., Huang, K., Saunders, A., McCallum, A., Ceder, G., and Olivetti, E. Materials synthesis insights from scientific literature via text extraction and machine learning. Chemistry of Materials, 29(21):9436–9444, 2017. find
  37. Kim, E., Jensen, Z., van Grootel, A., Huang, K., Staib, M., Mysore, S., Chang, H.-S., Strubell, E., McCallum, A., Jegelka, S., et al. Inorganic materials synthesis planning with literature-trained neural networks. Journal of chemical information and modeling, 60(3):1194–1201, 2020. find
  38. Kononova, O., He, T., Huo, H., Trewartha, A., Olivetti, E. A., and Ceder, G. Opportunities and challenges of text mining in materials research. Iscience, 24(3), 2021. find
  39. Lai, Y., Li, C., Wang, Y., Zhang, T., Zhong, R., Zettlemoyer, L., Yih, W.-t., Fried, D., Wang, S., and Yu, T. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pp. 18319–18345. PMLR, 2023. find
  40. Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smetanin, N., Verkuil, R., Kabeli, O., Shmueli, Y., et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123–1130, 2023. find
  41. Lo, K., Wang, L. L., Neumann, M., Kinney, R., and Weld, D. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4969–4983, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.…
  42. Mercado, R., Kearnes, S. M., and Coley, C. W. Data sharing in chemistry: lessons learned and a case for mandating structured reaction data. Journal of Chemical Information and Modeling, 63(14):4253–4265, 2023. find
  43. Merchant, A., Batzner, S., Schoenholz, S. S., Aykol, M., Cheon, G., and Cubuk, E. D. Scaling deep learning for materials discovery. Nature, pp. 1–6, 2023. find
  44. Mirza, A., Alampara, N., Kunchapu, S., Emoekabu, B., Krishnan, A., Wilhelmi, M., Okereke, M., Eberhardt, J., Elahi, A. M., Greiner, M., et al. Are large language models superhuman chemists? arXiv preprint arXiv:2404.01475, 2024. — [arXiv:2404.01475](https://arxiv.org/abs/2404.01475) find
  45. Mukaddem, K. T., Beard, E. J., Yildirim, B., and Cole, J. M. Imagedataextractor: a tool to extract and quantify data from microscopy images. Journal of chemical information and modeling, 60(5):2492–2509, 2019. find
  46. Mysore, S., Jensen, Z., Kim, E., Huang, K., Chang, H.-S., Strubell, E., Flanigan, J., McCallum, A., and Olivetti, E. The materials science procedural text corpus: Annotating materials synthesis procedures with shallow semantic structures. arXiv preprint arXiv:1905.06939, 2019. find
  47. Olivetti, E. A., Cole, J. M., Kim, E., Kononova, O., Ceder, G., Han, T. Y.-J., and Hiszpanski, A. M. Data-driven materials research enabled by natural language processing and information extraction. Applied Physics Reviews, 7(4), 2020. find
  48. Ravinder, Venugopal, V., Bishnoi, S., Singh, S., Zaki, M., Grover, H. S., Bauchy, M., Agarwal, M., and Krishnan, N. A. Artificial intelligence and machine learning in glass science and technology: 21 challenges for the 21st century. International journal of applied glass science, 12(3):277–292, 2021… find
  49. Rubungo, A. N., Arnold, C., Rand, B. P., and Dieng, A. B. Llm-prop: Predicting physical and electronic properties of crystalline solids from their text descriptions. arXiv preprint arXiv:2310.14029, 2023. — [arXiv:2310.14029](https://arxiv.org/abs/2310.14029) find
  50. Sim, M., Vakili, M. G., Strieth-Kalthoff, F., Hao, H., Hickman, R., Miret, S., Pablo-Garcı́a, S., and Aspuru-Guzik, A. Chemos 2.0: an orchestration architecture for chemical self-driving laboratories. 2023. find
  51. Song, Y., Miret, S., and Liu, B. MatSci-NLP: Evaluating scientific language models on materials science language tasks using text-to-schema modeling. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volum… find
  52. Song, Y., Miret, S., Zhang, H., and Liu, B. Honeybee: Progressive instruction finetuning of large language models for materials science. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5724–5739, 2023b. find
  53. Swain, M. C. and Cole, J. M. Chemdataextractor: a toolkit for automated extraction of chemical information from the scientific literature. Journal of chemical information and modeling, 56(10):1894–1904, 2016. find
  54. Szymanski, N. J., Rendy, B., Fei, Y., Kumar, R. E., He, T., Milsted, D., McDermott, M. J., Gallant, M., Cubuk, E. D., Merchant, A., et al. An autonomous laboratory for the accelerated synthesis of novel materials. Nature, 624(7990):86–91, 2023. find
  55. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  56. Vasudevan, R. K., Orozco, E., and Kalinin, S. V. Discovering mechanisms for materials microstructure optimization via reinforcement learning of a generative model. Machine Learning: Science and Technology, 3(4):04LT03, 2022. find
  57. Venugopal, V. and Olivetti, E. Matkg: An autonomously generated knowledge graph in material science. Scientific Data, 11 (1):217, 2024. find
  58. Venugopal, V., Sahoo, S., Zaki, M., Agarwal, M., Gosvami, N. N., and Krishnan, N. A. Looking through glass: Knowledge discovery from materials science literature using natural language processing. Patterns, 2(7), 2021. find
  59. Völker, C., Rug, T., Jablonka, K. M., and Kruschwitz, S. Llms can design sustainable concrete–a systematic benchmark. 2024. find
  60. Walker, N., Trewartha, A., Huo, H., Lee, S., Cruse, K., Dagdelen, J., Dunn, A., Persson, K., Ceder, G., and Jain, A. The impact of domain-specific pre-training on named entity recognition tasks in materials science. Available at SSRN 3950755, 2021. find
  61. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  62. White, A. D., Hocky, G. M., Gandhi, H. A., Ansari, M., Cox, S., Wellawatte, G. P., Sasmal, S., Yang, Z., Liu, K., Singh, Y., et al. Assessment of chemistry knowledge in large language models that generate code. Digital Discovery, 2(2):368–376, 2023. find
  63. Wu, S., Irsoy, O., Lu, S., Dabravolski, V., Dredze, M., Gehrmann, S., Kambadur, P., Rosenberg, D., and Mann, G. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303.17564, 2023a.
  64. Wu, Y., Jia, F., Zhang, S., Wu, Q., Li, H., Zhu, E., Wang, Y., Lee, Y. T., Peng, R., and Wang, C. An empirical study on challenging math problem solving with gpt-4. arXiv preprint arXiv:2306.01337, 2023b.
  65. Xie, T., Fu, X., Ganea, O.-E., Barzilay, R., and Jaakkola, T. Crystal diffusion variational autoencoder for periodic material generation. arXiv preprint arXiv:2110.06197, 2021. find
  66. Xie, T., Wan, Y., Huang, W., Yin, Z., Liu, Y., Wang, S., Linghu, Q., Kit, C., Grazian, C., Zhang, W., et al. Darwin series: Domain specific large language models for natural science. arXiv preprint arXiv:2308.13565, 2023.
  67. Xu, M., Yuan, X., Miret, S., and Tang, J. Protst: Multi-modality learning of protein sequences and biomedical texts. arXiv preprint arXiv:2301.12040, 2023. — [arXiv:2301.12040](https://arxiv.org/abs/2301.12040) find
  68. Yoshikawa, N., Skreta, M., Darvish, K., Arellano-Rubach, S., Ji, Z., Bjørn Kristensen, L., Li, A. Z., Zhao, Y., Xu, H., Kuramshin, A., et al. Large language models for chemistry robotics. Autonomous Robots, 47(8):1057–1086, 2023. find
  69. Zaki, M., Krishnan, N. A., et al. Extracting processing and testing parameters from materials science literature for improved property prediction of glasses. Chemical Engineering and Processing-Process Intensification, 180:108607, 2022a. find
  70. Zaki, M., Namireddy, S. R., Pittie, T., Bihani, V., Keshri, S. R., Venugopal, V., Gosvami, N. N., Krishnan, N. A., et al. Natural language processing-guided meta-analysis and structure factor database extraction from glass literature. Journal of Non-Crystalline Solids: X, 15:100103, 2022b. find
  71. Zaki, M., Sharma, S., Gurjar, S. K., Goyal, R., Krishnan, N. A., et al. Cementron: Machine learning the alite and belite phases in cement clinker from optical images. Construction and Building Materials, 397:132425, 2023. find
  72. Zaki, M., Jayadeva, J., Mausam, M., and Krishnan, N. A. Mascqa: Investigating materials science knowledge of large language models. Digital Discovery, 2024. find
  73. Zan, D., Chen, B., Zhang, F., Lu, D., Wu, B., Guan, B., Yongji, W., and Lou, J.-G. Large language models meet nl2code: A survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 7443–7464, 2023. find
  74. Zeni, C., Pinsler, R., Zügner, D., Fowler, A., Horton, M., Fu, X., Shysheya, S., Crabbé, J., Sun, L., Smith, J., et al. Mattergen: a generative model for inorganic materials design. arXiv preprint arXiv:2312.03687, 2023. — [arXiv:2312.03687](https://arxiv.org/abs/2312.03687) find
  75. Zhang, C., Zhang, C., Zhang, M., and Kweon, I. S. Text-to-image diffusion model in generative ai: A survey. arXiv preprint arXiv:2303.07909, 2023. — [arXiv:2303.07909](https://arxiv.org/abs/2303.07909) find
  76. Zhao, J. and Cole, J. M. A database of refractive indices and dielectric constants auto-generated using chemdataextractor. Scientific data, 9(1):192, 2022. find
  77. Zhao, J., Huang, S., and Cole, J. M. Opticalbert and opticaltable-sqa: Text-and table-based language models for the optical-materials domain. Journal of Chemical Information and Modeling, 63(7):1961–1981, 2023. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/miret-2024-llms-ready-real-world-materials/