Elamur Research⌕
☰
home / papers / 2024 / Tx-LLM: A Large Language Model for Therapeutics

Tx-LLM: A Large Language Model for Therapeutics

Paper · 2024 · Juan Manuel Zambrano Chaves, Eric Wang, Tao Tu et al.

Summary

Bringing a drug to approval takes many years and large sums, and most candidates fail because they must meet many requirements at once: binding the target, low toxicity, good pharmacokinetics, and manufacturability. Existing AI tools usually handle one narrow task each. The authors built Tx-LLM, a single generalist model created by fine-tuning PaLM-2. It is trained on TxT, a new instruction-tuning collection derived from the Therapeutics Data Commons (TDC). TxT turns tabular data into prompts that mix free-text instructions and context with string encodings of entities, such as SMILES for small molecules and amino acid sequences for proteins. With one set of weights, the model handles classification, regression and generation across the drug-development pipeline, from target discovery to clinical trials. It matches or beats specialist state-of-the-art systems on most tasks and does especially well where molecules are paired with textual entities. This points toward one LLM that could support several screening stages.

Key points

  • Training data: TxT contains 709 TDC datasets covering 66 tasks, roughly 10 million instruction–answer pairs, with a median dataset size of about 11,000.
  • Overall results: Tx-LLM is competitive with SOTA on 43 of 66 tasks and surpasses it on 22. It reaches or exceeds SOTA on 19 of 29 regression datasets.
  • Strongest area: On average it beats the best specialist models on tasks combining SMILES with text, such as cell line or disease names. The authors attribute this to knowledge gained during pretraining.
  • Transfer and ablations:
    • Training on biological-sequence datasets (e.g., proteins) improves results on small-molecule tasks.
    • Model scale, domain fine-tuning and prompting strategy all significantly affect performance.
  • Limitations:
    • The model is not tuned to follow natural language, so it cannot explain its predictions.
    • A contamination check found little overlap with PaLM-2 training data, though it did not account for differences in data format.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations🏢Google DeepMindLondon · United Kingdom56 works · 78,880 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputational Theory and Mathematics · 252 worksMaterials Science · 26 worksMaterials Chemistry · 26 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Juan Manuel Zambrano Chaves, Eric Wang, Tao Tu, Eeshit Dhaval Vaishnav, Byron Lee, S. Sara Mahdavi, Christopher Semturs, David Fleet, Vivek Natarajan and Shekoofeh Azizi. 2024. Tx-LLM: A Large Language Model for Therapeutics. arXiv:2406.06316.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.06316 · arXiv 2406.06316

@misc{chaves2024tx,
  title = {{Tx-LLM: A Large Language Model for Therapeutics}},
  author = {Juan Manuel Zambrano Chaves, Eric Wang, Tao Tu, Eeshit Dhaval Vaishnav, Byron Lee, S. Sara Mahdavi, Christopher Semturs, David Fleet, Vivek Natarajan and Shekoofeh Azizi},
  year = {2024},
  doi = {10.48550/arxiv.2406.06316},
  eprint = {2406.06316},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.06316}
}

References

12 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.11.12.13.14.15.16.17.18.19.20.21.22. Sun, D., Gao, W., Hu, H. & Zhou, S. Why 90% of clinical drug development fails and how to improve it? en. Acta Pharmaceutica Sinica B 12, 3049–3062. i s s n: 22113835. https://linkinghub.elsevier.com/retrieve/pii/S2211383522000521 (2… ↗
  2. Gruver, N., Finzi, M., Qiu, S. & Wilson, A. G. Large Language Models Are Zero-Shot Time Series Forecasters 2023. arXiv: 2310.07820 [cs.LG]. Song, X., Li, O., Lee, C., Yang, B., Peng, D., Perel, S. & Chen, Y. OmniPred: Language Models as Universal Regressors 2024. arXiv: 2402.14547 [cs.LG]. Singhal,… find
  3. Tu, T., Azizi, S., Driess, D., Schaekermann, M., Amin, M., Chang, P.-C., Carroll, A., Lau, C., Tanno, R., Ktena, I., Mustafa, B., Chowdhery, A., Liu, Y., Kornblith, S., Fleet, D., Mansfield, P., Prakash, S., Wong, R., Virmani, S., Semturs, C., Mahdavi, S. S., Green, B., Dominowska, E., y Arcas, B. A… find
  4. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. & Polosukhin, I. Attention Is All You Need 2023. arXiv: 1706.03762 [cs.CL]. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal,…
  5. Morrone, J. A., Weber, J. K., Huynh, T., Luo, H. & Cornell, W. D. Combining Docking Pose Rank and Structure with Deep Learning Improves Protein–Ligand Binding Mode Prediction over a Baseline Docking Approach. Journal of Chemical Information and Modeling 60. PMID: 32077698, 4170–4179. eprint: https:/… — [doi:10.1021/acs.jcim.9b00927](https://doi.org/10.1021/acs.jcim.9b00927) find
  6. Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S. A. A., Ballard, A. J., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clanc… — Highly accurate protein structure prediction with AlphaFold
  7. Tunyasuvunakool, K., Adler, J., Wu, Z., Green, T., Zielinski, M., Žídek, A., Bridgland, A., Cowie, A., Meyer, C., Laydon, A., Velankar, S., Kleywegt, G. J., Bateman, A., Evans, R., Pritzel, A., Figurnov, M., Ronneberger, O., Bates, R., Kohl, S. A. A., Potapenko, A., Ballard, A. J., Romera-Paredes, B… find
  8. Senior, A. W., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., Qin, C., Žídek, A., Nelson, A. W. R., Bridgland, A., Penedones, H., Petersen, S., Simonyan, K., Crossan, S., Kohli, P., Jones, D. T., Silver, D., Kavukcuoglu, K. & Hassabis, D. Improved protein structure prediction using po… find
  9. Yu, B., Baker, F. N., Chen, Z., Ning, X. & Sun, H. LlaSMol: Advancing Large Language Models for Chemistry with a Large-Scale, Comprehensive, High-Quality Instruction Tuning Dataset 2024. arXiv: 2402.09391 [cs.AI]. M. Bran, A., Cox, S., Schilter, O., Baldassari, C., White, A. D. & Schwaller, P. Augme… find
  10. Kim, S., Chen, J., Cheng, T., Gindulyte, A., He, J., He, S., Li, Q., Shoemaker, B. A., Thiessen, P. A., Yu, B., et al. PubChem 2023 update. Nucleic acids research 51, D1373–D1380 (2023). ToxCast and Tox21 Summary Files from invitrodb_v3. — PaLM 2 Technical Report https://www.epa.gov/chemical-research/toxicity- forecaster-to… ↗
  11. Dalke, A. The chemfp project. en. Journal of Cheminformatics 11, 76. i s s n: 1758-2946. https://jcheminf.biomedcentral.com/articles/10.1186/s13321-019-0398-8 (2024) (Dec. 2019). Sievers, F., Wilm, A., Dineen, D., Gibson, T. J., Karplus, K., Li, W., Lopez, R., McWilliam, H., Remmert, M., Söding, J.… — [doi:10.1186/s13321-019-0398-8](https://doi.org/10.1186/s13321-019-0398-8) ↗
  12. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., Schulman, J., Hilton, J., Kelton, F., Miller, L., Simens, M., Askell, A., Welinder, P., Christiano, P., Leike, J. & Lowe, R. Training language models to follow instructions with hu… — Training language models to follow instructions with human feedback

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chaves-2024-tx-llm-large-language-model/