Elamur Research⌕
☰
home / papers / 2025 / Contrastive Learning of English Language and Crystal Graphs for Multimodal Representation of Materials Knowledge

Contrastive Learning of English Language and Crystal Graphs for Multimodal Representation of Materials Knowledge

Paper · 2025 · Yang Jeong Park, Mayank Kumaran, Chia-Wei Hsu et al.

Summary

Crystal design has lacked the multimodal text–structure models that molecular chemistry already uses. There are few crystal datasets, research attention clusters on a small set of popular materials, and published papers rarely give clean descriptions linked to specific structures. The authors present CLaC, a contrastive model that places 3D crystal graphs and English descriptions in one shared embedding space. Crystal structures come from Materials Project and are converted to graphs with Jarvis-tools. Each structure is paired with a GPT-generated narrative from earlier work, giving 126k pairs. For comparison, they also train the same model on sentences mined from a large private collection of papers. Both modalities are augmented during training: graphs lose edges, nodes or subgraphs, and text has tokens masked.

CLaC achieves state-of-the-art zero-shot crystal understanding, ahead of recent LLMs. Its pre-trained encoders also improve NER and paper abstract classification over unimodal baselines. The main lesson is that synthetic pairs can offset both the scarcity and the skew of real literature.

Key points

  • Training data. 126k structure–text pairs. Structures come from Materials Project and are turned into graphs with Jarvis-tools. The texts are GPT-3.5 narratives from Park et al. (2023).
  • Literature baseline. Chemical formulas were used as weak-supervision queries over the full text of 4M papers. This produced about 400k sentence pairs, but they cover only about 400 inorganic compounds, which shows how narrow the real literature is.
  • Augmentations.
    • Graphs: edge removal, node dropping and subgraph sampling, all intended to keep structures physically plausible.
    • Text: random replacement of tokens with [MASK].
  • Evaluation.
    • Zero-shot cross-modal retrieval: best results, surpassing recent LLMs.
    • NER and paper abstract classification (PAC): transfer from the multimodal encoders beats unimodal models.
  • Embedding analysis.
    • UMAP plots separate crystal systems, oxide types and application terms such as battery or solar cell. Pristine MatSciBERT separates them less clearly.
    • Davies-Bouldin and Calinski-Harabasz indices both indicate better clustering.
    • The Silhouette score stays near 0, which the authors attribute to noisy labels and high dimensionality.

Interesting ideas

  • I1. Train the same model once on synthetic pairs and once on literature-mined pairs, so the benefit of synthetic data can be measured directly instead of assumed.
  • I2. A structured graph and its text description are pulled together in a shared embedding space, so each can retrieve the other without labels.
  • I3. Each modality gets two randomly perturbed views (graph pieces dropped, tokens masked), and the model learns to treat both views as the same thing.
  • I4. Cluster quality is judged with several metrics at once (DB, CH, Silhouette), and their disagreement is read as a sign of label noise rather than ignored.
  • I5. The authors note that a bare formula like SiO2 does not reveal whether it means a crystal, an amorphous solid or nanoparticles. Surface names alone underdetermine entity type.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Silver data from LLM teacher tiers could be tested against naturally occurring text using the same parser and the same test set, showing whether generated examples truly fill gaps for rare constructions.
  • I2 (back): An encoder that aligns world graphs with sentences could score how faithfully generated text, or the text → graph → text compressor output, preserves the meaning graph.
  • I3 (learn): Self-play could mask words or drop clauses and check that the extracted frames stay stable, flagging fragile parts of the pipeline.
  • I4 (learn): Evaluations of verb-class or frame embeddings could report several metrics side by side, so noisy gold labels show up before they mislead training.
  • I5 (world): Entity typing should rely on surrounding context and ontology cues instead of a fixed name-to-type lookup, especially for domain terms used in school-course texts.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksMaterials Science · 26 worksMaterials Chemistry · 26 worksElectronic, Optical and Magnetic Materials · 1 work

🧭 Topics 2Research subjects and languages the work is about.English · 2,263 worksVision & language · 657 works

Cite

Cite as Yang Jeong Park, Mayank Kumaran, Chia-Wei Hsu, Elsa Olivetti and Ju Li. 2025. Contrastive Learning of English Language and Crystal Graphs for Multimodal Representation of Materials Knowledge. arXiv:2502.16451.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2502.16451 · arXiv 2502.16451

@misc{park2025contrastive,
  title = {{Contrastive Learning of English Language and Crystal Graphs for Multimodal Representation of Materials Knowledge}},
  author = {Yang Jeong Park, Mayank Kumaran, Chia-Wei Hsu, Elsa Olivetti and Ju Li},
  year = {2025},
  doi = {10.48550/arxiv.2502.16451},
  eprint = {2502.16451},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2502.16451}
}

References

53 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Edward Kim, Kevin Huang, Stefanie Jegelka, and Elsa Olivetti. Virtual screening of inorganic materials synthesis parameters with deep learning. npj Computational Materials, 3(1):53, 2017. find
  2. Murat Cihan Sorkun, Séverin Astruc, JM Vianney A Koelman, and Süleyman Er. An artificial intelligence-aided virtual screening recipe for two-dimensional materials discovery. npj Computational Materials, 6(1):106, 2020. find
  3. Tian Xie and Jeffrey C Grossman. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. Physical review letters, 120(14):145301, 2018. ↗
  4. Kamal Choudhary and Brian DeCost. Atomistic line graph neural network for improved materials property predictions. npj Computational Materials, 7(1):185, 2021. find
  5. Prathik R Kaundinya, Kamal Choudhary, and Surya R Kalidindi. Prediction of the electron density of states for crystalline compounds with atomistic line graph neural networks (alignn). arXiv preprint arXiv:2201.08348, 2022. ↗
  6. Jordan Hoffmann, Louis Maestrati, Yoshihide Sawada, Jian Tang, Jean Michel Sellier, and Yoshua Bengio. Data-driven approach to encoding and decoding 3-d crystal structures. arXiv preprint arXiv:1909.00949, 2019. ↗
  7. Juhwan Noh, Jaehoon Kim, Helge S Stein, Benjamin Sanchez-Lengeling, John M Gregoire, Alan Aspuru-Guzik, and Yousung Jung. Inverse design of solid-state materials via a continuous representation. Matter, 1(5):1370–1384, 2019. ↗
  8. Callum J Court, Batuhan Yildirim, Apoorv Jain, and Jacqueline M Cole. 3-d inorganic crystal structure generation and property prediction via representation learning. Journal of Chemical Information and Modeling, 60(10):4518–4535, 2020. ↗
  9. Tian Xie, Xiang Fu, Octavian-Eugen Ganea, Regina Barzilay, and Tommi Jaakkola. Crystal diffusion variational autoencoder for periodic material generation. arXiv preprint arXiv:2110.06197, 2021. ↗
  10. Peder Lyngby and Kristian Sommer Thygesen. Data-driven discovery of 2d materials by deep generative models. npj Computational Materials, 8(1):232, 2022. find
  11. Sheng Gong, Keqiang Yan, Tian Xie, Yang Shao-Horn, Rafael Gomez-Bombarelli, Shuiwang Ji, and Jeffrey C Grossman. Examining graph neural networks for crystal structures: limitations and opportunities for capturing periodicity. Science Advances, 9(45):eadi3245, 2023. ↗
  12. Teerachote Pakornchote, Natthaphon Choomphon-Anomakhun, Sorrjit Arrerut, Chayanon Atthapak, Sakarn Khamkaeo, Thiparat Chotibut, and Thiti Bovornratanaraks. Diffusion probabilistic models enhance variational autoencoder for crystal structure generative modeling. Scientific Reports, 14(1):1275, 2024. ↗
  13. Turab Lookman, Prasanna V Balachandran, Dezhen Xue, and Ruihao Yuan. Active learning in materials science with emphasis on adaptive sampling using uncertainties for targeted design. npj Computational Materials, 5(1):21, 2019. find
  14. Chun-Teh Chen and Grace X Gu. Generative deep neural networks for inverse materials design using backpropagation and active learning. Advanced Science, 7(5):1902607, 2020. ↗
  15. Keith T Butler, Daniel W Davies, Hugh Cartwright, Olexandr Isayev, and Aron Walsh. Machine learning for molecular and materials science. Nature, 559(7715):547–555, 2018. ↗
  16. Ankit Agrawal and Alok Choudhary. Perspective: Materials informatics and big data: Realization of the “fourth paradigm” of science in materials science. Apl Materials, 4(5), 2016. find
  17. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. ↗
  18. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  19. Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation learning. In Larry Davis, Philip Torr, and Song-Chun Zhu, editors, Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1920–1929, 2019. ↗
  20. Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4(3):279–287, 2022. ↗
  21. Xiaomin Fang, Lihang Liu, Jieqiong Lei, Donglong He, Shanzhuo Zhang, Jingbo Zhou, Fan Wang, Hua Wu, and Haifeng Wang. Geometry-enhanced molecular representation learning for property prediction. Nature Machine Intelligence, 4(2):127–134, 2022. ↗
  22. Teddy Koker, Keegan Quigley, Will Spaeth, Nathan C Frey, and Lin Li. Graph contrastive learning for materials. arXiv preprint arXiv:2211.13408, 2022. ↗
  23. Rishikesh Magar, Yuyang Wang, and Amir Barati Farimani. Crystal twins: self-supervised learning for crystalline material property prediction. npj Computational Materials, 8(1):231, 2022. find
  24. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8748–8763. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/radford21a.html. ↗
  25. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Rama Chellappa, Jiri Matas, Long Quan, and Mubarak Shah, editors, Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022. ↗
  26. Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8821–8831. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/ramesh21a.html. ↗
  27. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.08485, 2023a. ↗
  28. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. arXiv preprint arXiv:2306.00890, 2023. ↗
  29. John J Irwin and Brian K Shoichet. Zinc- a free database of commercially available compounds for virtual screening. Journal of chemical information and modeling, 45(1):177–182, 2005. ↗
  30. Alec Belsky, Mariette Hellenbrandt, Vicky Lynn Karen, and Peter Luksch. New developments in the inorganic crystal structure database (icsd): accessibility in support of materials research and design. Acta Crystallographica Section B: Structural Science, 58(3):364–369, 2002. ↗
  31. Anubhav Jain, Shyue Ping Ong, Geoffroy Hautier, Wei Chen, William Davidson Richards, Stephen Dacek, Shreyas Cholia, Dan Gunter, David Skinner, Gerbrand Ceder, et al. The materials project: A materials genome approach to accelerating materials innovation, apl mater. APL Materials, 2013. find
  32. Yang Jeong Park, Sung Eun Jerng, Jin-Sung Park, Choah Kwon, Chia-Wei Hsu, Zhichu Ren, Sungroh Yoon, and Ju Li. 1.5 million materials narratives generated by chatbots. arXiv preprint arXiv:2308.13687, 2023. ↗
  33. Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1263–1272. PMLR, 06–11 Aug 2017. URL https://proceedings.mlr.press/v70/gilmer17a.html. ↗
  34. Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stephane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 12310–12320. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/zbontar21a.html. ↗
  35. Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nature communications, 13(1):862, 2022a. ↗
  36. David Weininger. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31–36, 1988. find
  37. David Weininger, Arthur Weininger, and Joseph L Weininger. Smiles. 2. algorithm for generation of unique smiles notation. Journal of chemical information and computer sciences, 29(2):97–101, 1989. find
  38. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Dan S Weld. S2orc: The semantic scholar open research corpus. arXiv preprint arXiv:1911.02782, 2019. ↗
  39. Bing Su, Dazhao Du, Zhao Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Hao Sun, Zhiwu Lu, and Ji-Rong Wen. A molecular multimodal foundation model associating molecule graphs with natural language. arXiv preprint arXiv:2209.05481, 2022. ↗
  40. Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. Advances in neural information processing systems, 33:5812–5823, 2020. find
  41. Iz Beltagy, Kyle Lo, and Arman Cohan. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676, 2019. ↗
  42. Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, and Animashree Anandkumar. Multi-modal molecule structure–text model for text-based retrieval and editing. Nature Machine Intelligence, 5(12):1447–1457, 2023b. ↗
  43. Kishalay Das, Pawan Goyal, Seung-Cheol Lee, Satadeep Bhattacharjee, and Niloy Ganguly. CrysMMNet: Multimodal representation for crystal property prediction. In Robin J. Evans and Ilya Shpitser, editors, Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pages 507–517. PMLR, 31 Jul–04 Aug 2023. URL https://proceedings.mlr.press/v216/das23a.html. ↗
  44. Alex M Ganose and Anubhav Jain. Robocrystallographer: automated crystal structure text descriptions and analysis. MRS Communications, 9(3):874–881, 2019. ↗
  45. Kamal Choudhary, Kevin F Garrity, Andrew CE Reid, Brian DeCost, Adam J Biacchi, Angela R Hight Walker, Zachary Trautt, Jason Hattrick-Simpers, A Gilad Kusne, Andrea Centrone, et al. The joint automated repository for various integrated simulations (jarvis) for data-driven materials design. npj computational materials, 6(1):173, 2020. find
  46. Kristof Schütt, Oliver Unke, and Michael Gastegger. Equivariant message passing for the prediction of tensorial properties and molecular spectra. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 9377–9388. PMLR, 18–24 Jul 2021. URL https://proceedings.mlr.press/v139/schutt21a.html. ↗
  47. Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. arXiv preprint arXiv:2110.05208, 2021. ↗
  48. Aman Shrivastava, Ramprasaath R. Selvaraju, Nikhil Naik, and Vicente Ordonez. Clip-lite: Information efficient visual representation learning with language supervision. In Francisco Ruiz, Jennifer Dy, and Jan-Willem van de Meent, editors, Proceedings of The 26th International Conference on Artificial Intelligence and Statistics, volume 206 of Proceedings of Machine Learning Research, pages 8433–8447. PMLR, 25–27 Apr 2023. URL https://proceedings.mlr.press/v206/shrivastava23a.html. ↗
  49. Tanishq Gupta, Mohd Zaki, NM Anoop Krishnan, and Mausam. Matscibert: A materials domain language model for text mining and information extraction. npj Computational Materials, 8(1):102, 2022. find
  50. Xiangxiang Zeng, Hongxin Xiang, Linhui Yu, Jianmin Wang, Kenli Li, Ruth Nussinov, and Feixiong Cheng. Accurate prediction of molecular properties and drug targets using a self-supervised image representation learning framework. Nature Machine Intelligence, 4(11):1004–1016, 2022b. ↗
  51. Nicholas Walker, Amalie Trewartha, Haoyan Huo, Sanghoon Lee, Kevin Cruse, John Dagdelen, Alexander Dunn, Kristin Persson, Gerbrand Ceder, and Anubhav Jain. The impact of domain-specific pre-training on named entity recognition tasks in materials science. Available at SSRN 3950755, 2021. ↗
  52. Vineeth Venugopal, Sourav Sahoo, Mohd Zaki, Manish Agarwal, Nitya Nand Gosvami, and NM Anoop Krishnan. Looking through glass: Knowledge discovery from materials science literature using natural language processing. Patterns, 2(7), 2021. ↗
  53. Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426, 2018. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/park-2025-contrastive-learning-english-language-crystal/