Elamur Research⌕
☰
home / papers / 2023 / GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text

GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text

Paper · 2023 · Pengfei Liu, Yiming Ren, Jun Tao et al.

Summary

Most language models for chemistry handle molecules only as text, such as SMILES strings or captions. They miss information carried by molecular graphs and by depictions of molecules in images. The authors address this with GIT-Mol, a multi-modal large language model of about 700M parameters that combines graph, image, and text inputs.

Its central component is GIT-Former, a fusion module inspired by the Q-Former from BLIP2. It uses learnable queries with self-attention and cross-attention to project inputs of different lengths and modalities into one fixed-size shared space, aligned at the level of whole molecules. GIT-Former is pre-trained with two self-supervised objectives (XTM and XTC) while the graph and image encoders stay frozen. A MolT5 decoder then turns the fused embeddings into SMILES or captions, and an MLP head handles property prediction.

The model covers molecule captioning, text-based molecule generation, image-to-SMILES recognition, and property prediction. It reports clear gains over baselines, which suggests that combining several modalities can benefit AI-aided drug discovery.

Key points

  • Results: GIT-Mol improves property prediction accuracy by roughly 5–10% and the validity of generated molecules by 20.2% over baselines.
  • Ablation: fusing several modalities outperforms single-modality variants by about 10–15%.
  • Pre-training: the two objectives are XTM, which learns whether a cross-modal input and a text belong together, and XTC, which contrasts cross-modal representations against text representations. Each run pairs one to three modalities with a target text modality, uses learning rates between 1e-4 and 5e-5, and trains for fewer than 20 epochs.
  • Design and limitations: an "any-to-language" translation strategy is meant to extend the model to compound name recognition and reaction prediction. Training speed remains a limitation, and the authors plan to address it with LoRA, Adapters, or PEFT.
  • Availability: code and data are at github.com/AI-HPC-Research-Team/GIT-Mol for non-commercial use only.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sun Yat-sen UniversityGuangzhou · China80 works · 9,799 citations🔬Peng Cheng LaboratoryShenzhen · China48 works · 2,812 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksComputer Vision and Pattern Recognition · 3,621 worksComputational Theory and Mathematics · 496 worksMaterials Science · 58 worksMaterials Chemistry · 58 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksVision & language · 827 works

Cite

Cite as Pengfei Liu, Yiming Ren, Jun Tao and Zhixiang Ren. 2023. GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text. arXiv:2308.06911 · Computers in Biology and Medicine, 108,073, 2024, ISSN 0010-4825.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1016/j.compbiomed.2024.108073 · arXiv 2308.06911

@misc{liu2023git,
  title = {{GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text}},
  author = {Pengfei Liu, Yiming Ren, Jun Tao and Zhixiang Ren},
  year = {2023},
  doi = {10.1016/j.compbiomed.2024.108073},
  eprint = {2308.06911},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1016/j.compbiomed.2024.108073}
}

References

35 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alayrac, J.B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al., 2022. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35, 23716–23736.
  2. Bao, F., Nie, S., Xue, K., Li, C., Pu, S., Wang, Y., Yue, G., Cao, Y., Su, H., Zhu, J., 2023. One transformer fits all distributions in multi-modal diffusion at scale. arXiv preprint arXiv:2303.06555 . Bao, H., Wang, W., Dong, L., Liu, Q., Mohammed, O.K., Aggarwal, K., Som, S., Piao, S., Wei, F., 20… find
  3. Beltagy, I., Lo, K., Cohan, A., 2019. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676 . Bento, A.P., Hersey, A., Félix, E., Landrum, G., Gaulton, A., Atkinson, F., Bellis, L.J., De Veij, M., Leach, A.R., 2020. An open source chemical structure curation pipel…
  4. Bilodeau, C., Jin, W., Jaakkola, T., Barzilay, R., Jensen, K.F., 2022. Generative models for molecular discovery: Recent advances and challenges. Wiley Interdisciplinary Reviews: Computational Molecular Science 12, e1608. find
  5. Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 . Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G.,…
  6. Edwards, C., Zhai, C., Ji, H., 2021. Text2mol: Cross-modal molecule retrieval with natural language queries, in: Proceedings of the 2021
  7. Conference on Empirical Methods in Natural Language Processing, pp. 595–607. Filippov, I.V., Nicklaus, M.C., 2009. Optical structure recognition software to recover chemical information: Osra, an open source solution. find
  8. Floridi, L., Chiriatti, M., 2020. Gpt-3: Its nature, scope, limits, and consequences. Minds and Machines 30, 681–694. find
  9. Hastings, J., Owen, G., Dekker, A., Ennis, M., Kale, N., Muthukrishnan, V., Turner, S., Swainston, N., Mendes, P., Steinbeck, C., 2016. Chebi in 2016: Improved services and an expanding collection of metabolites. Nucleic Acids Research 44, D1214–D1219. find
  10. He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778.
  11. Hou, Z., Liu, X., Cen, Y., Dong, Y., Yang, H., Wang, C., Tang, J., 2022. Graphmae: Self-supervised masked graph autoencoders, in: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 594–604. find
  12. Huang, S., Dong, L., Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O.K., Liu, Q., et al., 2023. Language is not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045 . Jia, C., Yang, Y., Xia, Y., Chen, Y.T., Parekh, Z., Pham, H., Le, Q., Sung, Y…
  13. Kim, S., Chen, J., Cheng, T., Gindulyte, A., He, J., He, S., Li, Q., Shoemaker, B.A., Thiessen, P.A., Yu, B., et al., 2019. find
  14. Pubchem 2019 update: improved access to chemical data. Nucleic Acids Research 47, D1102–D1109. find
  15. Li, J., Li, D., Savarese, S., Hoi, S., 2023a. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 . Li, J., Liu, Y., Fan, W., Wei, X.Y., Liu, H., Tang, J., Li, Q., 2023b. Empowering molecule discovery for molecule-ca…
  16. Liu, S., Nie, W., Wang, C., Lu, J., Qiao, Z., Liu, L., Tang, J., Xiao, C., Anandkumar, A., 2022. Multi-modal molecule structure-text model for text-based retrieval and editing. arXiv preprint arXiv:2212.10789 . Liu, S., Wang, H., Liu, W., Lasenby, J., Guo, H., Tang, J., 2021a. Pre-training molecular… find
  17. Luo, R., Sun, L., Xia, Y., Qin, T., Zhang, S., Poon, H., Liu, T.Y., 2022. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics 23, bbac409. find
  18. OpenAI, 2023. GPT-4 technical report. CoRR abs/2303.08774.
  19. Peryea, T., Katzel, D., Zhao, T., Southall, N., Nguyen, D.T., 2019. Molvec: Open source library for chemical structure recognition, in: Abstracts of Papers of the American Chemical Society, Amer Chemical Soc 1155 16TH ST, NW, WASHINGTON, DC 20036 USA. find
  20. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al., 2021. Learning transferable visual models from natural language supervision, in: International Conference on Machine Learning, PMLR. pp. 8748–8763.
  21. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al., 2019. Language models are unsupervised multitask learners. OpenAI blog 1, 9. find
  22. Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J., 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21, 5485–5551.
  23. Rajan, K., Zielesny, A., Steinbeck, C., 2021. Decimer 1.0: deep learning for chemical image recognition using transformers. Journal of Cheminformatics 13, 1–16. find
  24. Ramesh, A., Dhariwal, P., Nichol, A., Chu, C., Chen, M., 2022. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 . Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S.G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J.T.…
  25. Stefanini, M., Cornia, M., Baraldi, L., Cascianelli, S., Fiameni, G., Cucchiara, R., 2022. From show to tell: A survey on deep learning-based image captioning. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 539–559. find
  26. Su, B., Du, D., Yang, Z., Zhou, Y., Li, J., Rao, A., Sun, H., Lu, Z., Wen, J.R., 2022. A molecular multimodal foundation model associating molecule graphs with natural language. arXiv preprint arXiv:2209.05481 . Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł.,… — [arXiv:2209.05481](https://arxiv.org/abs/2209.05481) find
  27. Wang, H., Ma, S., Huang, S., Dong, L., Wang, W., Peng, Z., Wu, Y., Bajaj, P., Singhal, S., Benhaim, A., et al., 2022a. Foundation transformers. arXiv preprint arXiv:2210.06423 . Wang, J., Shen, Z., Liao, Y., Yuan, Z., Li, S., He, G., Lan, M., Qian, X., Zhang, K., Li, H., 2022b. Multi-modal chemical… — [arXiv:2210.06423](https://arxiv.org/abs/2210.06423) find
  28. Wang, W., Bao, H., Dong, L., Bjorck, J., Peng, Z., Liu, Q., Aggarwal, K., Mohammed, O.K., Singhal, S., Som, S., et al., 2022c. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 . Wang, Y., Wang, J., Cao, Z., Barati Farimani, A., 2… find
  29. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q.V., Zhou, D., et al., 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems 35, 24824–24837.
  30. Weininger, D., 1988. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of Chemical Information and Computer Sciences 28, 31–36. find
  31. Wu, C., Yin, S., Qi, W., Wang, X., Tang, Z., Duan, N., 2023. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint arXiv:2303.04671 . Wu, Z., Ramsundar, B., Feinberg, E.N., Gomes, J., Geniesse, C., Pappu, A.S., Leswing, K., Pande, V., 2018. Moleculenet: a benchma…
  32. Xu, Z., Li, J., Yang, Z., Li, S., Li, H., 2022. Swinocsr: end-to-end optical chemical structure recognition using a swin transformer. Journal of Cheminformatics 14, 1–13. find
  33. Yang, L., Zhang, Z., Song, Y., Hong, S., Xu, R., Zhao, Y., Shao, Y., Zhang, W., Cui, B., Yang, M.H., 2022. Diffusion models: A comprehensive survey of methods and applications. arXiv preprint arXiv:2209.00796 . Zeng, Z., Yao, Y., Liu, Z., Sun, M., 2022. A deep-learning system bridging molecule struc… — [arXiv:2209.00796](https://arxiv.org/abs/2209.00796) find
  34. Zhou, G., Gao, Z., Ding, Q., Zheng, H., Xu, H., Wei, Z., Zhang, L., Ke, G., 2023. Uni-mol: A universal 3d molecular representation learning framework, in: The Eleventh International Conference on Learning Representations. URL: https://openreview.net/ forum?id=6K2RM6wVqKu. ↗
  35. Zhu, D., Chen, J., Shen, X., Li, X., Elhoseiny, M., 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592 .

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2023-git-mol-multi-modal-large/