Elamur Research⌕
☰
home / papers / 2026 / STELLA: A Multimodal LLM for Protein Functional Annotation via Unified Sequence-Structure Encoding

STELLA: A Multimodal LLM for Protein Functional Annotation via Unified Sequence-Structure Encoding

Paper · 2026 · Hongwang Xiao, Wenjun Lin, Xi Chen et al.

Summary

STELLA addresses protein functional annotation. Protein language models capture biophysical signals well but seldom connect them to the textual knowledge that describes what a protein does. Earlier protein–text multimodal LLMs used separate sequence and structure encoders, which needed fusion layers. STELLA instead uses one encoder, ESM3 (esm3_sm_open_v1, 1.4B), which embeds sequence and structure jointly. A single linear projection maps these embeddings into Llama-3.1-8B-Instruct, in a LLaVA-style design.

Training is two-stage multimodal instruction tuning on the new OPI-Struc dataset. Because good protein–text pairs are scarce, both stages reuse the same data:

  • Stage 1: only the connector is trained.
  • Stage 2: the connector and the LLM are trained together with different learning rates; the encoder stays frozen.

The authors report state-of-the-art results on two tasks: generating a protein's functional description (FP) and predicting the reactions an enzyme catalyzes (EP). They argue that coupling a unified structure-aware encoder with a generative LLM moves beyond what pure pLMs can do for biology and biomedicine.

Key points

  • Architecture: ESM3 encoder (frozen), a one-layer linear adapter, and a Llama-3.1-8B-Instruct backbone. Code is at github.com/ocx-lab/STELLA.
  • Resource: OPI-Struc, a released instruction-tuning dataset that pairs protein structures with natural-language annotations.
  • Training: the two-stage schedule outperformed single-stage joint training on all reported FP metrics (FP_ft_eval).
  • Encoder comparison: ESM3, Prot2Text and SaProt encoders were tested with several LLMs (Llama-3/3.1, Phi-3, Mistral, BioMedGPT-LM, BioMistral). SaProt-based variants scored lowest.
  • Reasoning LLMs: DeepSeek-R1-distilled backbones gave no clear gain over Llama-3.1-8B-Instruct. Scores fell sharply on the out-of-distribution set FP_ft_eval_v2401; for example, BLEU-4 dropped from about 0.43 to about 0.05.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China244 works · 39,359 citations🎓Renmin University of ChinaBeijing · China114 works · 16,118 citations🔬Beijing Academy of Artificial IntelligenceBeijing · China32 works · 7,511 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Biochemistry, Genetics and Molecular Biology · 234 worksMolecular Biology · 231 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksVision & language · 657 works

Cite

Cite as Hongwang Xiao, Wenjun Lin, Xi Chen, Hui Wang, Kai Chen, Jiashan Li, Yuancheng Sun, Sicheng Dai, Boya Wu and Qiwei Ye. 2026. STELLA: A Multimodal LLM for Protein Functional Annotation via Unified Sequence-Structure Encoding. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1254

@inproceedings{xiao2026stella,
  title = {{STELLA: A Multimodal LLM for Protein Functional Annotation via Unified Sequence-Structure Encoding}},
  author = {Hongwang Xiao, Wenjun Lin, Xi Chen, Hui Wang, Kai Chen, Jiashan Li, Yuancheng Sun, Sicheng Dai, Boya Wu and Qiwei Ye},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1254},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1254}
}

References

63 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Jyoti Aneja, and et al Hany Awadalla. 2024. Phi-3 technical report: A highly capable language model locally on your phone. Preprint, arXiv:2404.14219.
  2. Hadi Abdine, Michail Chatzianastasis, Costas Bouyioukos, and Michalis Vazirgiannis. 2023. Prot2Text: Multimodal protein’s function generation with gnns and transformers. arXiv preprint arXiv:2307.14367. — [arXiv:2307.14367](https://arxiv.org/abs/2307.14367) find
  3. Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J Ballard, Joshua Bambrick, and 1 others. 2024. Accurate structure prediction of biomolecular interactions with AlphaFold 3. Nature, 630(8016):493–500. find
  4. AI@Meta. 2024. Llama 3 model card. find
  5. Ethan C Alley, Grigory Khimulya, Surojit Biswas, Mohammed AlQuraishi, and George M Church. 2019. find
  6. Unified rational protein engineering with sequence-based deep representation learning. Nature methods, 16(12):1315–1322. find
  7. Helen M. Berman, John Westbrook, Zukang Feng, Gary Gilliland, T. N. Bhat, Helge Weissig, Ilya N. Shindyalov, and Philip E. Bourne. 2000. The Protein Data Bank. Nucleic Acids Research, 28(1):235–242. find
  8. Chai-Discovery-Team, Jacques Boitreaud, Jack Dent, Matthew McPartlon, Joshua Meier, Vinicius Reis, Alex Rogozhonikov, and Kevin Wu. 2024. Chai-1: Decoding the molecular interactions of life. BioRxiv, pages 2024–10. find
  9. Bo Chen, Xingyi Cheng, Pan Li, Yangli-ao Geng, Jing Gong, Shen Li, Zhilei Bei, Xu Tan, Boyan Wang, Xin Zeng, and 1 others. 2024. xTrimoPGLM: unified 100b-scale pre-trained transformer for deciphering the language of protein. arXiv preprint arXiv:2401.06199. — [arXiv:2401.06199](https://arxiv.org/abs/2401.06199) find
  10. Mark Chen and Jerry Tworek et al. 2021. Evaluating large language models trained on code. Preprint, arXiv:2107.03374.
  11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168.
  12. Jose M Dana, Aleksandras Gutmanas, Nidhi Tyagi, Guoying Qi, Claire O’Donovan, Maria Martin, and Sameer Velankar. 2018. SIFTS: updated Structure Integration with Function, Taxonomy and Sequences resource allows 40-fold increase in coverage of structure-based annotations for proteins. Nucleic Acids Re… find
  13. Georgy Derevyanko, Sergei Grudinin, Yoshua Bengio, and Guillaume Lamoureux. 2018. Deep convolutional networks for quality assessment of protein folds. Bioinformatics, 34(23):4046–4053. find
  14. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  15. Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang, Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. 2022. ProtTrans: Toward understanding the language of life through self-supervised learning. IEEE Transactio… find
  16. Hehe Fan, Zhangyang Wang, Yi Yang, and Mohan Kankanhalli. 2022. Continuous-discrete convolution for geometry-sequence modeling in proteins. In The Eleventh International Conference on Learning Representations. find
  17. Wenqi Fan, Yi Zhou, Shijie Wang, Yuyao Yan, Hui Liu, Qian Zhao, Le Song, and Qing Li. 2025. Computational protein science in the era of large language models (LLMs). arXiv preprint arXiv:2501.10282. — [arXiv:2501.10282](https://arxiv.org/abs/2501.10282) find
  18. Yin Fang, Xiaozhuan Liang, Ningyu Zhang, Kangwei Liu, Rui Huang, Zhuo Chen, Xiaohui Fan, and Huajun Chen. Mol-Instructions: A large-scale biomolecular instruction dataset for large language models. In The Twelfth International Conference on Learning Representations. find
  19. Noelia Ferruz, Steffen Schmidt, and Birte Höcker. 2022. ProtGPT2 is a deep unsupervised language model for protein design. Nature Communications, 13. find
  20. Vladimir Gligorijević, P Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Berenberg, Tommi Vatanen, Chris Chandler, Bryn C Taylor, Ian M Fisk, Hera Vlamakis, and 1 others. 2021. Structure-based protein function prediction using graph convolutional networks. Nature communications, 12(1… find
  21. Tomas Hayes, Roshan Rao, Halil Akin, Nicholas J Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q Tran, Jonathan Deaton, Marius Wiggert, and 1 others. 2024. Simulating 500 million years of evolution with a language model. bioRxiv, pages 2024–07. find
  22. Muyang He, Yexin Liu, Boya Wu, Jianhao Yuan, Yueze Wang, Tiejun Huang, and Bo Zhao. 2024. Efficient multimodal learning from data-centric perspective. Preprint, arXiv:2402.11530. — [arXiv:2402.11530](https://arxiv.org/abs/2402.11530) find
  23. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300.
  24. Pedro Hermosilla and Timo Ropinski. 2022a. Contrastive representation learning for 3D protein structures. Preprint, arXiv:2205.15675. — [arXiv:2205.15675](https://arxiv.org/abs/2205.15675) find
  25. Pedro Hermosilla and Timo Ropinski. 2022b. Contrastive representation learning for 3D protein structures. arXiv preprint arXiv:2205.15675. — [arXiv:2205.15675](https://arxiv.org/abs/2205.15675) find
  26. Pedro Hermosilla, Marco Schäfer, Matej Lang, Gloria Fackelmann, Pere-Pau Vázquez, Barbora Kozlikova, Michael Krone, Tobias Ritschel, and Timo Ropinski. 2021. Intrinsic-extrinsic convolution and pooling for learning on 3D protein structures. In International Conference on Learning Representations. find
  27. Shulin Huang, Linyi Yang, Yan Song, Shuang Chen, Leyang Cui, Ziyu Wan, Qingcheng Zeng, Ying Wen, Kun Shao, Weinan Zhang, and 1 others. 2025. ThinkBench: Dynamic out-of-distribution evaluation for robust llm reasoning. arXiv preprint arXiv:2502.16268. — [arXiv:2502.16268](https://arxiv.org/abs/2502.16268) find
  28. Mingjia Huo, Han Guo, Xingyi Cheng, Digvijay Singh, Hamidreza Rahmani, Shen Li, Philipp Gerlof, Trey Ideker, Danielle A. Grotjahn, Elizabeth Villa, Le Song, and Pengtao Xie. 2024. Multi-modal large language model enables protein function prediction. bioRxiv. find
  29. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  30. John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zídek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A A Kohl, Andy Ballard, Andrew Cowie, Bernardino Romera-Paredes, Stanislav Nikolov, Rishub Jain,… — Highly accurate protein structure prediction with AlphaFold
  31. Yanis Labrak, Adrien Bazoge, Emmanuel Morin, Pierre-Antoine Gourraud, Mickael Rouvier, and Richard Dufour. 2024. BioMistral: A collection of open-source pretrained large language models for medical domains. Preprint, arXiv:2402.10373.
  32. Jiashan Li, Xi Chen, He Huang, Mingliang Zeng, Jingcheng Yu, Xinqi Gong, and Qiwei Ye. 2025. Sable: bridging the gap in protein structure understanding with an empowering and versatile pre-training paradigm. Briefings in Bioinformatics, 26(2):bbaf120. find
  33. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  34. Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, and 1 others. 2023. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123–1130. find
  35. Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023a. Visual instruction tuning. arXiv preprint arXiv:2304.08485.
  36. Shengchao Liu, Yutao Zhu, Jiarui Lu, Zhao Xu, Weili Nie, Anthony Gitter, Chaowei Xiao, Jian Tang, Hongyu Guo, and Anima Anandkumar. 2023b. A text-guided protein design framework. arXiv preprint arXiv:2302.04611. — [arXiv:2302.04611](https://arxiv.org/abs/2302.04611) find
  37. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, Online. Association for Computational Linguistics.
  38. Yizhen Luo, Jiahuan Zhang, Siqi Fan, Kai Yang, Yushuai Wu, Mu Qiao, and Zaiqing Nie. 2023. BioMedGPT: Open multimodal generative pre-trained transformer for biomedicine. Preprint, arXiv:2308.09442. — [arXiv:2308.09442](https://arxiv.org/abs/2308.09442) find
  39. OC-Team and Qiwei Ye. 2025. Towards unraveling biomolecular conformational landscapes with a generative foundation model. bioRxiv. find
  40. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
  41. Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Peter Chen, John Canny, Pieter Abbeel, and Yun Song. 2019. Evaluating protein transfer learning with TAPE. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc. find
  42. Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. 2021. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy… find
  43. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022. Language models are multilingual chain-of-thought reasoners. Preprint, arXiv:2210.03057.
  44. Venkat Krishna Srinivasan, Zhen Dong, Banghua Zhu, Brian Yu, Hanzi Mao, Damon Mosk-Aoyama, Kurt Keutzer, Jiantao Jiao, and Jian Zhang. 2023. Nexusraven: a commercially-permissive language model for function calling. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. find
  45. Martin Steinegger, Markus Meier, Milot Mirdita, Harald Vöhringer, Stephan J Haunsberger, and Johannes Söding. 2019. HH-suite3 for fast remote homology detection and deep protein annotation. BMC bioinformatics, 20:1–15. find
  46. Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. 2023. SaProt: Protein language modeling with structure-aware vocabulary. bioRxiv 2023.10.01.560349. find
  47. Michel Van Kempen, Stephanie S Kim, Charlotte Tumescheit, Milot Mirdita, Jeongjae Lee, Cameron LM Gilchrist, Johannes Söding, and Martin Steinegger. 2024. Fast and accurate protein structure search with Foldseek. Nature biotechnology, 42(2):243–246. find
  48. Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, Augustin Žídek, Tim Green, Kathryn Tunyasuvunakool, Stig Petersen, John Jumper, Ellen Clancy, Richard Green, Ankur Vora, Mira Lutfi, and 8 others. 20… find
  49. Chao Wang, Hehe Fan, Ruijie Quan, and Yi Yang. 2024. ProtChatGPT: Towards understanding proteins with large language models. arXiv preprint arXiv:2402.09649. — [arXiv:2402.09649](https://arxiv.org/abs/2402.09649) find
  50. Ru Wang, Wei Huang, Selena Song, Haoyu Zhang, Yusuke Iwasawa, Yutaka Matsuo, and Jiaxian Guo. 2025a. Beyond in-distribution success: Scaling curves of cot granularity for language model generalization. arXiv preprint arXiv:2502.18273. — [arXiv:2502.18273](https://arxiv.org/abs/2502.18273) find
  51. Zeyuan Wang, Qiang Zhang, Keyan Ding, Ming Qin, Xiang Zhuang, Xiaotong Li, and Huajun Chen. 2023. InstructProtein: Aligning human and protein language via knowledge instruction. arXiv preprint arXiv:2310.03269. find
  52. Zhicong Wang, Zicheng Ma, Ziqiang Cao, Changlong Zhou, Jun Zhang, and Yi Qin Gao. 2025b. Prot2Chat: protein large language model with early fusion of text, sequence, and structure. Bioinformatics, 41. find
  53. Jeremy Wohlwend, Gabriele Corso, Saro Passaro, Noah Getz, Mateo Reveiz, Ken Leidal, Wojtek Swiderski, Liam Atkinson, Tally Portnoi, Itamar Chinn, and 1 others. 2025. Boltz-1 democratizing biomolecular interaction modeling. BioRxiv, pages 2024–11. find
  54. Yijia Xiao, Edward Sun, Yiqiao Jin, Qifan Wang, and Wei Wang. 2024. ProteinGPT: Multimodal LLM for protein property prediction and structure understanding. arXiv preprint arXiv:2408.11363. — [arXiv:2408.11363](https://arxiv.org/abs/2408.11363) find
  55. Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. 2023. ProtST: Multi-modality learning of protein sequences and biomedical texts. arXiv preprint arXiv:2301.12040. — [arXiv:2301.12040](https://arxiv.org/abs/2301.12040) find
  56. Minghao Xu, Zuobai Zhang, Jiarui Lu, Zhaocheng Zhu, Yangtian Zhang, Ma Chang, Runcheng Liu, and Jian Tang. 2022. PEER: a comprehensive and multi-task benchmark for protein sequence understanding. Advances in Neural Information Processing Systems, 35:35156–35173. find
  57. Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/bl ogs/8_berkeley_function_calling_leaderbo ard.html. ↗
  58. Xinhao Yao, Ruifeng Ren, Yun Liao, and Yong Liu. 2025. Unveiling the mechanisms of explicit CoT training: How cot enhances reasoning generalization. arXiv preprint arXiv:2502.04667. — [arXiv:2502.04667](https://arxiv.org/abs/2502.04667) find
  59. Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. Language models are super mario: Absorbing abilities from homologous models as a free lunch. Preprint, arXiv:2311.03099.
  60. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. BERTscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675. find
  61. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Khai Hao, Xu Han, Zhen Leng Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024. ∞bench: Extending long context evaluation beyond 100k tokens. Preprint, arXiv:2402.13718. — [arXiv:2402.13718](https://arxiv.org/abs/2402.13718) find
  62. Zuobai Zhang, Minghao Xu, Arian Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. 2022. Protein representation learning by geometric structure pretraining. arXiv preprint arXiv:2203.06125. — [arXiv:2203.06125](https://arxiv.org/abs/2203.06125) find
  63. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-judge with mt-bench and chatbot arena. Preprint, arXiv:2306.05685.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2026-stella-multimodal-llm-protein-functional/