Elamur Research⌕
☰
home / papers / 2021 / Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries

Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries

Paper · 2021 · Carl Edwards, ChengXiang Zhai and Heng Ji

Summary

The authors introduce Text2Mol, a new retrieval task: given a free-text description, find the matching molecule in a collection that has no accompanying text. This matters because databases such as PubChem hold over 100 million compounds, and most of them will never be described by experts. Existing tools search either text or chemical structure, not across both. The authors treat molecules as a language with an unusual grammar, which makes the problem an extreme case of cross-lingual retrieval. They built a parallel benchmark of descriptions paired with compounds. On it they trained two encoders that map into a shared embedding space, where cosine similarity ranks candidates. Text is encoded with SciBERT. Molecules are encoded either by an MLP over Mol2vec features or by a GCN over the molecular graph. A cross-modal attention model produces association rules between words and substructures. These rules support reranking and explain model behaviour. Combining the architectures in an ensemble raises MRR from 0.372 to 0.499. The work opens a direction for linking chemistry literature with molecular machine learning.

Key points

  • Dataset: 33,010 text–compound pairs, each description having exactly one correct molecule. Results are measured with Hits@1 and MRR.
  • Architecture and training:
    • The text side uses SciBERT with a linear projection and layer normalization.
    • The molecule side uses Mol2vec, which builds Morgan-fingerprint substructure "sentences." It is fed either to an MLP or to a GCN that adds explicit graph structure.
    • Training uses a CLIP-style contrastive loss, extended with negative samples.
  • Reranking and explainability:
    • A 3-layer transformer decoder applies cross-modal attention and reranks the top 10 MLP candidates using the 10 most confident association rules. This beats classic FPGrowth rule mining.
    • The rules reveal interpretable links, for example "pollutant" pointing to F–C substructures.
  • Results: Ensembling the MLP and GCN models lifts MRR from 0.372 to 0.499, since the two architectures rank many molecules very differently. Errors remain for near-identical molecules (e.g., Pro-Arg vs. Arg-Pro) and for vague descriptions such as Fura red. Integrating external knowledge is left for future work.
  • Availability: Code and data are released for research at github.com/cnedwards/text2mol.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States108 works · 67,432 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksComputer Vision and Pattern Recognition · 1,282 worksInformation Systems · 523 worksComputational Theory and Mathematics · 116 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 145 works

Cite

Cite as Carl Edwards, ChengXiang Zhai and Heng Ji. 2021. Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2021.emnlp-main.47

@inproceedings{edwards2021text2mol,
  title = {{Text2Mol: Cross-Modal Molecule Retrieval with Natural Language Queries}},
  author = {Carl Edwards, ChengXiang Zhai and Heng Ji},
  booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.47},
  url = {https://doi.org/10.18653/v1/2021.emnlp-main.47}
}

References

66 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Derek T. Ahneman, Jesús G. Estrada, Shishi Lin, Spencer D. Dreher, and Abigail G. Doyle. 2018. Predicting reaction performance in c–n cross-coupling using machine learning. Science, 360(6385):186– 190. find
  2. Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
  3. John M. Barnard. 1993. Substructure searching methods: Old and new. Journal of chemical information and computer sciences, 33(4):532–538. find
  4. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. SciB-ERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615– 3620,… find
  5. In this work, we present Text2Mol: a novel and challenging cross-modal information retrieval task to retrieve molecules using natural language descriptions. To tackle this problem, we apply contrastive representation learning to a BERT-based text encoder and both MLP and GCN-based Samarth Bharadwaj,… find
  6. Seyone Chithrananda, Gabe Grand, and Bharath Ramsundar. 2020. Chemberta: Large-scale self-supervised pretraining for molecular property prediction. arXiv preprint arXiv:2010.09885. find
  7. Jiawei Han and Jian Pei. 2000. Mining frequent patterns by pattern-growth: methodology and implications. ACM SIGKDD explorations newsletter, 2(2):14–20. find
  8. Connor W. Coley, Regina Barzilay, Tommi S. Jaakkola, William H. Green, and Klavs F. Jensen. 2017. Prediction of organic reaction outcomes using machine learning. ACS central science, 3(5):434–443. find
  9. Janna Hastings, Gareth Owen, Adriano Dekker, Marcus Ennis, Namrata Kale, Venkatesh Muthukrishnan, Steve Turner, Neil Swainston, Pedro Mendes, and Christoph Steinbeck. 2016. Chebi in 2016: Improved services and an expanding collection of metabolites. Nucleic acids research, 44(D1):D1214–D1219. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  11. Harris Drucker, Corinna Cortes, Lawrence D. Jackel, Yann LeCun, and Vladimir Vapnik. 1994. Boosting and other ensemble methods. Neural Computation, 6(6):1289–1301. find
  12. Benedek Fabian, Thomas Edlich, Héléna Gaspar, Marwin Segler, Joshua Meyers, Marco Fiscato, and Mohamed Ahmed. 2020. Molecular representation learning with language models and domain-relevant auxiliary tasks. arXiv preprint arXiv:2011.13230. — [arXiv:2011.13230](https://arxiv.org/abs/2011.13230) find
  13. Weiguo Fan, Michael Gordon, and Praveen Pathak. 2006. On linear mixture of expert approaches to information retrieval. Decision Support Systems, 42(2):975–987. find
  14. William Fedus, Barret Zoph, and Noam Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. arXiv preprint arXiv:2101.03961. find
  15. David Fooshee, Aaron Mood, Eugene Gutman, Mohammadamin Tavakoli, Gregor Urban, Frances Liu, Nancy Huynh, David Van Vranken, and Pierre Baldi. 2018. Deep learning for chemical reaction prediction. Molecular Systems Design & Engineering, 3(3):442–452. find
  16. Marta Glavatskikh, Jules Leguy, Gilles Hunault, Thomas Cauchy, and Benoit Da Mota. 2019. Dataset’s chemical diversity limits the generalizability of machine learning predictions. Journal of Cheminformatics, 11(1):1–15. find
  17. Kunal Goyal, Utkarsh Gupta, Abir De, and Soumen Chakrabarti. 2020. Deep neural matching models for graph retrieval. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 1701– 1704… find
  18. Thomas R. Hagadone. 1992. Molecular substructure similarity searching: efficient retrieval in two-dimensional structure databases. Journal of chemical information and computer sciences, 32(5):515– 521. find
  19. Rezarta Islamaj Doğan, Sun Kim, Andrew Chatr-Aryamontri, Chih-Hsuan Wei, Donald C. Comeau, Rui Antunes, Sérgio Matos, Qingyu Chen, Aparna Elangovan, Nagesh C. Panyam, et al. 2019. Overview of the biocreative vi precision medicine track: mining protein interactions and mutations for precision medici… find
  20. Sabrina Jaeger, Simone Fulle, and Samo Turk. 2018. Mol2vec: unsupervised machine learning approach with chemical intuition. Journal of chemical information and modeling, 58(1):27–35. find
  21. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017. Billion-scale similarity search with gpus. arXiv preprint arXiv:1702.08734.
  22. Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A. Shoemaker, Paul A. Thiessen, Bo Yu, et al. 2019. find
  23. Pubchem 2019 update: improved access to chemical data. Nucleic acids research, 47(D1):D1102– D1109. find
  24. Sunghwan Kim, Paul A. Thiessen, Evan E. Bolton, Jie Chen, Gang Fu, Asta Gindulyte, Lianyi Han, Jane He, Siqian He, Benjamin A. Shoemaker, et al. 2016. Pubchem substance and compound databases. Nucleic acids research, 44(D1):D1202–D1213. find
  25. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings. find
  26. Thomas N. Kipf and Max Welling. 2017. Semi-supervised classification with graph convolutional networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
  27. Martin Krallinger, Obdulia Rabal, Analia Lourenco, Julen Oyarzabal, and Alfonso Valencia. 2017. Information retrieval and text mining technologies for chemistry. Chemical reviews, 117(12):7673–7761. find
  28. Miroslav Kratochvíl, Jiří Vondrášek, and Jakub Galgo-nek. 2018. Sachem: a chemical cartridge for high-performance substructure search. Journal of cheminformatics, 10(1):1–11. find
  29. Altman. Chemical reaction vector embeddings: towards predicting drug metabolism in the human gut microbiome. In Biocomputing 2018, pages 56–67. find
  30. Miroslav Kratochvíl. 2019. Accelerating structure search in small-molecule databases. Master’s thesis. find
  31. Tuan Lai, Heng Ji, ChengXiang Zhai, and Quan H. Tran. 2021. Joint biomedical entity and relation extraction with knowledge-enhanced collective inference. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural… find
  32. Greg Landrum. 2021. Rdkit: Open-source cheminformatics software. find
  33. Diya Li, Lifu Huang, Heng Ji, and Jiawei Han. 2019. Biomedical event extraction based on knowledge-driven tree-LSTM. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pa… find
  34. Diya Li and Heng Ji. 2019. Syntax-aware multi-task graph convolutional networks for biomedical relation extraction. In Proceedings of the Tenth International Workshop on Health Text Mining and Information Analysis (LOUHI 2019), pages 28–33, Hong Kong. Association for Computational Linguistics. find
  35. Hongzhi Li, Joseph G. Ellis, Heng Ji, and Shih-Fu Chang. 2016. Event specific multimodal pattern mining for knowledge base construction. In Proceedings of the 2016 ACM Conference on Multimedia Conference, MM 2016, Amsterdam, The Netherlands, October 15-19, 2016, pages 821–830. find
  36. Manling Li, Alireza Zareian, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, and Shih-Fu Chang. 2020. Cross-media structured common space for multimedia event extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2557–2568, Online. Association for… find
  37. Daniel M. Lowe, Peter T. Corbett, Peter Murray-Rust, and Robert C. Glen. 2011. Chemical name to structure: Opsin, an open source solution. Journal of Chemical Information and Modeling, 51(3):739– 753. PMID: 21384929. find
  38. Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, Decembe…
  39. Emily K. Mallory, Ambika Acharya, Stefano E. Rensi, Peter J. Turnbaugh, Roselie A. Bright, and Russ B. find
  40. Saeed Masoudnia and Reza Ebrahimpour. 2014. Mixture of experts: a literature survey. Artificial Intelligence Review, 42(2):275–293. find
  41. Christian Meilicke, Melisachew W. Chekol, Manuel Fink, and Heiner Stuckenschmidt. 2020. Reinforced anytime bottom up rule learning for knowledge graph completion. arXiv preprint arXiv:2004.04412. find
  42. Christian Meilicke, Melisachew W. Chekol, Daniel Ruffinelli, and Heiner Stuckenschmidt. 2019. Anytime bottom-up rule learning for knowledge graph completion. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, p… find
  43. Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. In 1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings.
  44. Tomás Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceeding…
  45. Aditya Mogadala, Marimuthu Kalimuthu, and Dietrich Klakow. 2020. Trends in integration of vision and language research: A survey of tasks, datasets, and methods. arXiv preprint arXiv:1907.09358. — [arXiv:1907.09358](https://arxiv.org/abs/1907.09358) find
  46. Simon Ott, Christian Meilicke, and Matthias Samwald. 2021. SAFRAN: An interpretable, rule-based link prediction method outperforming embedding models. In 3rd Conference on Automated Knowledge Base Construction. find
  47. Robi Polikar. 2012. Ensemble learning. In Ensemble machine learning, pages 1–34. Springer. find
  48. Jingwei Qu, Penghui Sun, Xin Li, Bei Wang, Xiaoqing Lu, Zhi Tang, and Chengcui Zhang. 2019. A retrieval system of medicine molecules based on graph similarity. IEEE MultiMedia, 26(4):17–27. find
  49. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th Internatio…
  50. Stefano Rensi and Russ B. Altman. 2017. Flexible analog search with kernel pca embedded molecule vectors. Computational and structural biotechnology journal, 15:320–327. find
  51. Tim Rocktäschel, Torsten Huber, Michael Weidlich, and Ulf Leser. 2013. WBI-NER: The impact of domain-specific features on the performance of identifying and classifying mentions of drugs. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh I… find
  52. David Rogers and Mathew Hahn. 2010. Extended-connectivity fingerprints. Journal of chemical information and modeling, 50(5):742–754. find
  53. Frederik Sandfort, Felix Strieth-Kalthoff, Marius Küh-nemund, Christian Beecks, and Frank Glorius. 2020. A structure-based platform for predicting chemical reactivity. Chem, 6(6):1379–1390. find
  54. Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages 1849–1857. find
  55. Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020. VL-BERT: pre-training of generic visual-linguistic representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
  56. Zhenqin Wu, Bharath Ramsundar, Evan N. Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S. Pappu, Karl Leswing, and Vijay Pande. 2018. Moleculenet: a benchmark for molecular machine learning. Chemical science, 9(2):513–530. find
  57. Xifeng Yan, Philip S. Yu, and Jiawei Han. 2005. Substructure similarity search in graph databases. In Proceedings of the 2005 ACM SIGMOD international conference on Management of data, pages 766–777. find
  58. Liang Zhang and Xiaobing Zhao. 2020. An overview of cross-language information retrieval. In International Conference on Artificial Intelligence and Security, pages 26–37. Springer. find
  59. Zixuan Zhang, Nikolaus Parulian, Heng Ji, Ahmed Elsayed, Skatje Myers, and Martha Palmer. 2021. Fine-grained information extraction from biomedical literature based on knowledge-enriched Abstract Meaning Representation. In Proceedings of the 59th Annual Meeting of the Association for Computational L… find
  60. Jin Guang Zheng, Daniel Howsmon, Boliang Zhang, Juergen Hahn, Deborah McGuinness, James Hendler, and Heng Ji. 2014. Entity linking for biomedical literature. In BMC Medical Informatics and Decision Making. find
  61. Yingyao Zhou, Bin Zhou, Shumei Jiang, and Frederick J. King. 2010. Chemical- text hybrid search engines. Journal of chemical information and modeling, 50(1):47–54. find
  62. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017,…
  63. Ivan Vulic and Marie-Francine Moens. 2015. Monolingual and cross-lingual information retrieval models based on (bilingual) word embeddings. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, Santiago, Chile, August 9-13, 2015, pages 36… find
  64. David Weininger. 1988. Smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. Journal of chemical information and computer sciences, 28(1):31–36. find
  65. David Weininger, Arthur Weininger, and Joseph L. Weininger. 1989. Smiles. 2. algorithm for generation of unique smiles notation. Journal of chemical information and computer sciences, 29(2):97–101. find
  66. Model MLP1 MLP2 MLP3 GCN1 GCN2 GCN3 MLP-Ensemble GCN-Ensemble All-Ensemble Mean Rank 43.66 47.42 41.15 41.78 41.23 42.19 30.60 28.89 24.95 Validation MRR Hits@1 0.374 22.5% 0.360 22.1% 0.376 21.2% 0.367 22.2% 0.367 22.1% 0.360 21.2% 0.442 28.7% 0.435 27.7% 0.479 31.7% Hits@10 68.8% 68.9% 68.2% 68.4%… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/edwards-2021-text2mol-cross-modal-molecule-retrieval/