Elamur Research⌕
☰
home / papers / 2025 / InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery

InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery

Paper · 2025 · He Cao, Zijing Liu, Xingyu Lu et al.

Summary

Large language models handle many tasks well, but in drug discovery they still trail specialist models trained for single tasks. The authors trace this to weak alignment between molecular data and text, little study of which structure encoder to use, and simplistic training pipelines. They present InstructMol, a multi-modal LLM that couples a pretrained molecule graph encoder with a general-purpose LLM through a lightweight projector. Optionally, a SELFIES-style sequence of the molecule is also supplied.

Training has two stages:

  • Alignment pretraining: molecule–description pairs teach the projector to map node-level graph features into the LLM's text space.
  • Task-specific instruction tuning: the graph encoder stays frozen while LoRA adapters on the LLM are trained for each task.

The model is evaluated on property prediction, molecule description generation and chemical reaction tasks. It beats generalist LLMs and earlier instruction-tuned molecular models, and comes much closer to dedicated expert systems. Because the adapters are small, task skills can be loaded as plugins while the base model keeps its general dialogue ability.

Key points

  • Two model variants are tested: InstructMol-G uses graph input only, and InstructMol-GS adds the sequence input. The GS variant is clearly stronger on reaction tasks.
  • On forward reaction prediction, InstructMol-GS reaches 0.536 exact match. This compares with 0.045 for Mol-Instruction and 0.454 for Text+ChemT5.
  • On retrosynthesis, InstructMol-GS scores 0.407 exact match against 0.009 for Mol-Instruction. The specialist Retroformer-untyped still leads at 0.536.
  • The molecule–text training data has about 330K pairs, far fewer than typical vision–language datasets.
  • In ablations, a two-layer MLP projector beats a linear one, and moving from a 7B to a 13B LLM brings only small gains. Code is available at github.com/IDEA-XL/InstructMol.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations🎓Tsinghua Shenzhen International Graduate SchoolShenzhen · China28 works · 3,035 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksComputational Theory and Mathematics · 252 worksMaterials Science · 26 worksMaterials Chemistry · 26 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as He Cao, Zijing Liu, Xingyu Lu, Yuan Yao and Yu Li. 2025. InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{cao2025instructmol,
  title = {{InstructMol: Multi-Modal Integration for Building a Versatile and Reliable Molecular Assistant in Drug Discovery}},
  author = {He Cao, Zijing Liu, Xingyu Lu, Yuan Yao and Yu Li},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.25/}
}

References

96 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. PubChem Structure Search. https://pubchem.ncbi.nlm.nih.gov/search/ help_search.html. ↗
  2. Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andy Brock,… — Flamingo: a Visual Language Model for Few-Shot Learning
  3. Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Marie Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniela Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Ke Xu, Yunhan Xu, Lin Wu Xue, Pengcheng Y… — PaLM 2 Technical Report
  4. Heba Askr, Enas Elgeldawi, Heba Aboul Ella, Yaseen A.M.M. Elshaier, Mamdouh M. Gomaa, and Aboul Ella Hassanien. 2022. Deep learning in drug discovery: an integrative review and future challenges. Artificial Intelligence Review, 56:5975 – 6037. find
  5. Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Johnson, Dmitry Lepikhin, Alexandre Tachard Sinan Tan, Peng Wang, Junyang Lin, Chang Passos, Siamak Shakeri, Emanuel Taropa, Zhou, and Jingren Zhou. 2023. Qwen-vl: A Paige Bailey, Z. Chen, Eric Chu, J. C…
  6. Paul Barham, Jan A. Botha, James Bradbury, Siddhartha Brahma, Kevin Michael Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scib-Brooks, Michele Catasta, Yongzhou Cheng, ert: A pretrained language model for scientific Colin Cherry, Christopher A. Choquette-Choo, text. In Conference on Empirical Methods…
  7. He Cao, Weidi Luo, Yu Wang, Zijing Liu, Bing Feng, Yuan Yao, and Yu Li. 2024a. Guide for defense (g4d): Dynamic guidance for robust and balanced defense in large language models. arXiv preprint arXiv:2410.17922. — [arXiv:2410.17922](https://arxiv.org/abs/2410.17922) find
  8. He Cao, Yanjun Shao, Zhiyuan Liu, Zijing Liu, Xiangru Tang, Yuan Yao, and Yu Li. 2024b. PRESTO: Progressive pretraining enhances synthetic chemistry outcomes. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10197–10224, Miami, Florida, USA. Association for Computation… find
  9. Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. 2023. X-llm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. ArXiv, abs/2305.04160. understanding. In North American Chapter of the Association for Computational Li… — [arXiv:2305.04160](https://arxiv.org/abs/2305.04160) find
  10. Joseph L. Durant, Burton A. Leland, Douglas R. Henry, and James G. Nourse. 2002. Reoptimization of mdl keys for use in drug discovery. Journal of chemical information and computer sciences, 42 6:1273–80. find
  11. Carl N. Edwards, T. Lai, Kevin Ros, Garrett Honke, and Heng Ji. 2022. Translation between molecules and natural language. ArXiv, abs/2204.11817.
  12. Carl N. Edwards, Chengxiang Zhai, and Heng Ji. 2021. Text2mol: Cross-modal molecule retrieval with natural language queries. In Conference on Empirical Methods in Natural Language Processing.
  13. Li et.al. 2023. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML.
  14. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  15. Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. 2023. Mol-instructions: A large-scale biomolecular instruction dataset for large language models. ArXiv, abs/2306.08018. find
  16. Seyone Chithrananda, Gabriel Grand, and Bharath Ramsundar. 2020. Chemberta: Large-scale self-supervised pretraining for molecular property prediction. ArXiv, abs/2010.09885. find
  17. Bin Feng, Zequn Liu, Nanlan Huang, Zhiping Xiao, Haomiao Zhang, Srbuhi Mirzoyan, Hanwen Xu, Jiaran Hao, Yinghui Xu, Ming Zhang, et al. 2024. A bioactivity foundation model using pairwise meta-learning. Nature Machine Intelligence, 6(8):962–974. find
  18. Dimitrios Christofidellis, Giorgio Giannone, Jannis Born, Ole Winther, Teodoro Laino, and Matteo Manica. 2023. Unifying molecular and textual representations via multi-task language modelling. In International Conference on Machine Learning. find
  19. Connor W. Coley. 2020. Defining and exploring chemical spaces. Trends in Chemistry. find
  20. Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Albert Li, Pascale Fung, and Steven C. H. Hoi. 2023. Instructblip: Towards general-purpose vision-language models with instruction tuning. ArXiv, abs/2305.06500.
  21. Yu Gu, Robert Tinn, Hao Cheng, Michael R. Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020. Domain-specific language model pretraining for biomedical natural language processing. ACM Transactions on Computing for Healthcare (HEALTH), 3:1 – 23.
  22. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Xiaodong Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. ArXiv, abs/2009.03300.
  23. J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Jacob Devlin, Ming-Wei Chang, Kenton Lee, and and Weizhu Chen. 2021. find
  24. Lora: Low-rank Kristina Toutanova. 2019. Bert: Pre-training adaptation of large language models. ArXiv, of deep bidirectional transformers for language abs/2106.09685. 364
  25. Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay S. Pande, and Jure Leskovec. 2019. Strategies for pre-training graph neural networks. arXiv: Learning. find
  26. Ross Irwin, Spyridon Dimitriadis, Jiazhen He, and Esben Jannik Bjerrum. 2021. Chemformer: a pre-trained transformer for computational chemistry. Machine Learning: Science and Technology, 3. find
  27. Zhu Jinhua, Xia Yingce, Wu Lijun, Xie Shufang, Zhou Wengang, Qin Tao, Li Houqiang, and Liu Tie-Yan. 2023. Dual-view molecular pre-training. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3615–3627. find
  28. Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. 2023. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, pages 15696–15707. PMLR.
  29. Jin Kim, Sera Park, Dongbo Min, and Wankyu Kim. 2021. Comprehensive survey of recent drug discovery using deep learning. International Journal of Molecular Sciences, 22. find
  30. Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A. Shoemaker, Paul A. Thiessen, Bo Yu, Leonid Y. Zaslavsky, Jian Zhang, and Evan E. Bolton. 2022. find
  31. Pubchem 2023 update. Nucleic acids research. find
  32. Sunghwan Kim, Paul A. Thiessen, Tiejun Cheng, Jian Zhang, Asta Gindulyte, and Evan E. Bolton. 2019. Pug-view: programmatic access to chemical annotations integrated in pubchem. Journal of Cheminformatics, 11. find
  33. Mario Krenn, Florian Hase, AkshatKumar Nigam, Pascal Friederich, and Alán Aspuru-Guzik. 2019. Self-referencing embedded strings (selfies): A 100% robust molecular string representation. Machine Learning: Science and Technology, 1. find
  34. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36:1234 – 1240.
  35. Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. 2023a. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. ArXiv, abs/2306.00890.
  36. Jiatong Li, Yunqing Liu, Wenqi Fan, Xiao Wei, Hui Liu, Jiliang Tang, and Qing Li. 2023b. Empowering molecule discovery for molecule-caption translation with large language models: A chatgpt perspective. ArXiv, abs/2306.06615. — [arXiv:2306.06615](https://arxiv.org/abs/2306.06615) find
  37. Youwei Liang, Ruiyi Zhang, Li Zhang, and Peng Xie. 2023. Drugchat: Towards enabling chatgpt-like capabilities on drug molecule graphs. ArXiv, abs/2309.03907. — [arXiv:2309.03907](https://arxiv.org/abs/2309.03907) find
  38. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Annual Meeting of the Association for Computational Linguistics.
  39. Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a. Improved baselines with visual instruction tuning. ArXiv, abs/2310.03744.
  40. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b. Visual instruction tuning. ArXiv, abs/2304.08485.
  41. Jiawei Liu, Cheng Yang, Zhiyuan Lu, Junze Chen, Yibo Li, Mengmei Zhang, Ting Bai, Yuan Fang, Lichao Sun, Philip S. Yu, and Chuan Shi. 2023c. Towards graph foundation models: A survey and beyond. ArXiv, abs/2310.11829. — [arXiv:2310.11829](https://arxiv.org/abs/2310.11829) find
  42. Peng Liu, Yiming Ren, and Zhixiang Ren. 2023d. Git-mol: A multi-modal large language model for molecular science with graph, image, and text. ArXiv, abs/2308.06911. find
  43. Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, and Anima Anandkumar. 2022. Multi-modal molecule structure-text model for text-based retrieval and editing. ArXiv, abs/2212.10789. find
  44. Shengchao Liu, Hanchen Wang, Weiyang Liu, Joan Lasenby, Hongyu Guo, and Jian Tang. 2021. Pre-training molecular graph representation with 3d geometry. ArXiv, abs/2110.07728. 365 — [arXiv:2110.07728](https://arxiv.org/abs/2110.07728) find
  45. Zequn Liu, W. Zhang, Yingce Xia, Lijun Wu, Shufang Xie, Tao Qin, Ming Yang Zhang, and Tie-Yan Liu. 2023e. Molxpt: Wrapping molecules with text for generative pre-training. ArXiv, abs/2305.10688. find
  46. Zhiyuan Liu, Sihang Li, Yancheng Luo, Hao Fei, Yixin Cao, Kenji Kawaguchi, Xiang Wang, and Tat-Seng Chua. 2023f. Molca: Molecular graph-language modeling with cross-modal projector and uni-modal adapter. In Conference on Empirical Methods in Natural Language Processing. find
  47. Jieyu Lu and Yingkai Zhang. 2022a. Unified deep learning model for multitask reaction predictions with explanation. Journal of chemical information and modeling. find
  48. Jieyu Lu and Yingkai Zhang. 2022b. Unified deep learning model for multitask reaction predictions with explanation. Journal of chemical information and modeling. find
  49. Xingyu Lu, He Cao, Zijing Liu, Shengyuan Bai, Leqing Chen, Yuan Yao, Hai-Tao Zheng, and Yu Li. 2024. MoleculeQA: A dataset to evaluate factual accuracy in molecular comprehension. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3769–3789, Miami, Florida, USA. Associat… find
  50. Gen Luo, Yiyi Zhou, Tianhe Ren, Shen Chen, Xiaoshuai Sun, and Rongrong Ji. 2023a. Cheap and quick: Efficient vision-language instruction tuning for large language models. ArXiv, abs/2305.15023. — [arXiv:2305.15023](https://arxiv.org/abs/2305.15023) find
  51. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… — Training language models to follow instructions with human feedback
  52. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2001. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics - ACL ’02.
  53. Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, and Lingpeng Kong Tong Zhang. 2023. Detgpt: Detect what you need via reasoning. ArXiv, abs/2305.14167. find
  54. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In International Conference on Machin…
  55. Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21:140:1–140:67.
  56. Yi Luo, Kai Yang, Massimo Hong, Xingyi Liu, and Zaiqing Nie. 2023b. Molfm: A multimodal molecular foundation model. ArXiv, abs/2307.09484. — [arXiv:2307.09484](https://arxiv.org/abs/2307.09484) find
  57. Raghunathan Ramakrishnan, Pavlo O. Dral, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld. 2014a. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1. find
  58. Yi Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. 2023c. Biomedgpt: Open multimodal generative pre-trained transformer for biomedicine. ArXiv, abs/2308.09442. — [arXiv:2308.09442](https://arxiv.org/abs/2308.09442) find
  59. Raghunathan Ramakrishnan, Pavlo O Dral, Matthias Rupp, and O Anatole Von Lilienfeld. 2014b. Quantum chemistry structures and properties of 134 kilo molecules. Scientific data, 1(1):1–7. find
  60. B. Ramsundar, P. Eastman, P. Walters, and V. Pande. 2019. Deep Learning for the Life Sciences: Applying Deep Learning to Ge-OpenAI. 2023b. Gpt-4 technical report. ArXiv, nomics, Microscopy, Drug Discovery, and More. abs/2303.08774. O’Reilly. 366
  61. OpenAI. 2023a. "chatgpt: A language model for conversational ai. find
  62. Ahmet Sureyya Rifaioglu, Heval Atas, Maria Jesus Martin, Rengul Cetin-Atalay, Volkan Atalay, and Tunca Dogan. 2018. Recent applications of deep learning and machine intelligence on in silico drug discovery: methods, tools and databases. Briefings in Bioinformatics, 20:1878 – 1912. find
  63. Nadine Schneider, Roger A. Sayle, and Gregory A. Landrum. 2015. Get your atoms in order - an open-source implementation of a novel and robust molecular canonicalization algorithm. Journal of chemical information and modeling, 55 10:2111–20. find
  64. Philippe Schwaller, Teodoro Laino, Théophile Gaudin, Peter Bolgar, Constantine Bekas, and Alpha Albert Lee. 2018. Molecular transformer: A model for uncertainty-calibrated chemical reaction prediction. ACS Central Science, 5:1572 – 1583. find
  65. Hayal Bulbul Sonmez, Figen Kuloğlu, Koksal Karadag, and Fred Wudl. 2012. Terephthalaldehyde- and isophthalaldehyde-based polyspiroacetals. Polymer Journal, 44:217–223. find
  66. Hannes Stärk, D. Beaini, Gabriele Corso, Prudencio Tossou, Christian Dallago, Stephan Gunnemann, and Pietro Lio’. 2021. 3d infomax improves gnns for molecular property prediction. In International Conference on Machine Learning. find
  67. Bing Su, Dazhao Du, Zhao-Qing Yang, Yujie Zhou, Jiangmeng Li, Anyi Rao, Haoran Sun, Zhiwu Lu, and Ji rong Wen. 2022. A molecular multimodal foundation model associating molecule graphs with natural language. ArXiv, abs/2209.05481. — [arXiv:2209.05481](https://arxiv.org/abs/2209.05481) find
  68. Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, and Chao Huang. 2023. Graphgpt: Graph instruction tuning for large language models. ArXiv, abs/2310.13023. find
  69. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/ stanford_alpaca. and Robert Stojnic. 2022. Galactica: A large language model for… ↗
  70. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. ArXiv,…
  71. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Bria… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  72. Ahmad Walid, Simon Elana, Chithrananda Seyone, Grand Gabriel, and Ramsundar Bharath. 2022. Chemberta-2: Towards chemical foundation models. arXiv preprint arXiv:2209.01712. — [arXiv:2209.01712](https://arxiv.org/abs/2209.01712) find
  73. Sheng Wang, Yuzhi Guo, Yuhong Wang, Hongmao Sun, and Junzhou Huang. 2019. Smiles-bert: Large scale unsupervised pre-training for molecular property prediction. Proceedings of the 10th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics. find
  74. Ross Taylor, Marcin Kardas, Guillem Cucu- Wen Wang, Zhe Chen, Xiaokang Chen, Jiannan rull, Thomas Scialom, Anthony S. Hartshorn, Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Elvis Saravia, Andrew Poulton, Viktor Kerkez, Lu, Jie Zhou, Y. Qiao, and Jifeng Dai. 2023. 367 find
  75. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. ArXiv, abs/2305.11175. — [arXiv:2305.11175](https://arxiv.org/abs/2305.11175) find
  76. Yu Wang, Xiaogeng Liu, Yu Li, Muhao Chen, and Chaowei Xiao. 2024. Adashield: Safeguarding multimodal large language models from structure-based attack via adaptive shield prompting. arXiv preprint arXiv:2403.09513. — [arXiv:2403.09513](https://arxiv.org/abs/2403.09513) find
  77. Yuyang Wang, Jianren Wang, Zhonglin Cao, and Amir Barati Farimani. 2021. Molecular contrastive learning of representations via graph neural networks. Nature Machine Intelligence, 4:279 – 287. find
  78. Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners. ArXiv, abs/2109.01652.
  79. Jinmao Wei, Xiao-Jie Yuan, Qinghua Hu, and Shuqin Wang. 2010. A novel measure for evaluating classifiers. Expert Syst. Appl., 37:3799– 3809. find
  80. David Weininger. 1988. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. J. Chem. Inf. Comput. Sci., 28:31–36. find
  81. Aidan Wong, He Cao, Zijing Liu, and Yu Li. 2024. Smiles-prompting: A novel approach to llm jailbreak attacks in chemical synthesis. arXiv preprint arXiv:2410.15641. — [arXiv:2410.15641](https://arxiv.org/abs/2410.15641) find
  82. Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023. Pmc-llama: Towards building open-source language models for medicine.
  83. Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay S. Pande. 2017. Moleculenet: A benchmark for molecular machine learning. arXiv: Learning. find
  84. Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023. Baize: An open-source chat model with parameter-efficient tuning on self-chat data. ArXiv, abs/2304.01196.
  85. Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023. Gpt4tools: Teaching large language model to use tools via self-instruction. ArXiv, abs/2305.18752. find
  86. Weiran Yao, Shelby Heinecke, Juan Carlos Niebles, Zhiwei Liu, Yihao Feng, Le Xue, Rithesh Murthy, Zeyuan Chen, Jianguo Zhang, Devansh Arpit, Ran Xu, Phi Thi Mi, Haiquan Wang, Caiming Xiong, and Silvio Savarese. 2022. Retroformer: Pushing the limits of interpretable end-to-end retrosynthesis transfor… find
  87. Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yi Zhou, Junyan Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, Chenliang Li, Yuanhong Xu, Hehong Chen, Junfeng Tian, Qiang Qi, Ji Zhang, and Feiyan Huang. 2023. mplug-owl: Modularization empowers large language models with multimodality. ArXiv, abs/23…
  88. Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. 2023. A survey on multimodal large language models. ArXiv, abs/2306.13549.
  89. Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020. Graph contrastive learning with augmentations. ArXiv, abs/2010.13902. find
  90. Rong Yu, Bian Yatao, Xu Tingyang, Xie Weiyang, Wei Ying, Huang Wenbing, and Huang Junzhou. 2020. Self-supervised graph transformer on large-scale molecular data. Advances in neural information processing systems, 33:12559– 12571. find
  91. Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, P. Zhang, Yuxiao Dong, and Jie Tang. 2022a. Glm-130b: An open bilingual pre-trained model. ArXiv, abs/2210.02414.
  92. Zheni Zeng, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2022b. A deep-learning system bridging molecule structure and biomedical text with comprehension comparable to human professionals. Nature Communications, 13. find
  93. Keyulu Xu, Weihua Hu, Jure Leskovec, and Ste- Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Weix-fanie Jegelka. 2018. How powerful are graph iong Lin, Ya Zhang, Yanfeng Wang, and Weidi neural networks? ArXiv, abs/1810.00826.
  94. Xie. 2023. Pmc-vqa: Visual instruction tuning 368 for medical visual question answering. ArXiv, abs/2305.10415. — [arXiv:2305.10415](https://arxiv.org/abs/2305.10415) find
  95. Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng, Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. 2023. Uni-mol: A universal 3d molecular representation learning framework. In International Conference on Learning Representations. find
  96. Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ArXiv, abs/2304.10592.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cao-2025-instructmol-multi-modal-integration-building/