Elamur Research⌕
☰
home / papers / 2026 / GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding

GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding

Paper · 2026 · Weicai Long, Yusen Hou, Junning Feng et al.

Summary

General-purpose LLMs are increasingly used as assistants in genomics, but it is unclear how they behave when given raw DNA sequences rather than annotations or text-only questions. Existing benchmarks either test trainable DNA-specific models, through fine-tuning or probing, or test biological knowledge without any sequence input. The authors fill this gap with GenomeQA, a question-answering benchmark of 5,200 samples. The data come from ENCODE, EPDnew, SCREEN, NCBI RefSeq, JASPAR and the Nucleotide Transformer downstream tasks. All samples went through the same processing:

  • window selection and length standardization with Bedtools,
  • removal of overlapping samples,
  • filtering of ambiguous bases.

Each sequence is then placed into a natural-language template, posed as a binary-choice (BCQ) or four-option (MCQ) question, and answered under one fixed system prompt. Six frontier LLMs usually beat random baselines when local cues such as GC content or short motifs are informative. They falter when the answer requires indirect, multi-step inference. The benchmark therefore serves as a diagnostic tool for sequence-aware use of LLMs.

Key points

  • Coverage: GenomeQA spans six task families with sequences of 6–1,000 bp:
    • Recognizing enhancers and promoters
    • Detecting splice sites
    • Classifying the taxonomic origin of sequences
    • Histone Mark Prediction (10 marks)
    • TFBS Prediction (20 TFs)
    • TF Motif Prediction (20 motifs)
  • Models evaluated: Claude-Sonnet-4.5, GPT-5.1, Gemini-3-Pro, Grok-4.1, Llama-4 and Qwen3-Max.
  • Effect of naming the target: On CTCF TFBS prediction, accuracy drops when the target is implied rather than named. For Gemini-3-Pro, MCQ accuracy fell from 67.47% to 44.58%, and BCQ accuracy from 74.70% to 60.24%.
  • Failure modes: An analysis of 200 Gemini-3-Pro errors found four types:
    • Sequence Motif Over-reliance (47%)
    • Base Composition Over-reliance (32%)
    • Character Fidelity Loss (13%), i.e., hallucinated sub-sequences
    • Noise Distinction Failure (8%) on shuffled negative samples
  • Limitations and availability: The dataset is moderately sized and not intended for fine-tuning. Results come from single runs with Wilson intervals. Data and code are at github.com/ai4nucleome/GenomeQA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China315 works · 43,268 citations🎓University of Hong KongPok Fu Lam · China118 works · 13,496 citations🎓The Hong Kong University of Science and Technology (Guangzhou)Guangzhou · China67 works · 1,765 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksBiochemistry, Genetics and Molecular Biology · 263 worksMolecular Biology · 260 works

Cite

Cite as Weicai Long, Yusen Hou, Junning Feng, Houcheng su, Shuo Yang, Donglin Xie and Yanlin Zhang. 2026. GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1655

@inproceedings{long2026genomeqa,
  title = {{GenomeQA: Benchmarking General Large Language Models for Genome Sequence Understanding}},
  author = {Weicai Long, Yusen Hou, Junning Feng, Houcheng su, Shuo Yang, Donglin Xie and Yanlin Zhang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1655},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1655}
}

References

46 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Alibaba. 2025. Qwen3-max: Just scale it. Accessed: 2025-09-24. find
  2. Anthropic. 2025. Introducing claude sonnet 4.5. Accessed: 2025-09-29. find
  3. Daman Arora, Himanshu Singh, and 1 others. 2023. Have llms advanced enough? a challenging problem solving benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7527–7543.
  4. Seongsu Bae, Daeun Kyung, Jaehee Ryu, Eunbyeol Cho, Gyubok Lee, Sunjun Kweon, Jungwoo Oh, Lei Ji, Eric Chang, Tackeun Kim, and 1 others. 2023. Ehrxqa: A multi-modal question answering dataset for electronic health records with chest x-ray images. Advances in Neural Information Processing Systems, 36… find
  5. Gonzalo Benegas, Carlos Albors, Alan J Aw, Chengzhong Ye, and Yun S Song. 2025. A dna language model based on multispecies alignment predicts the effects of genome-wide variants. Nature Biotechnology, pages 1–6. find
  6. Garyk Brixi, Matthew G Durrant, Jerome Ku, Michael Poli, Greg Brockman, Daniel Chang, Gabriel A Gonzalez, Samuel H King, David B Li, Aditi T Merchant, and 1 others. 2025. Genome modeling and design across all domains of life with evo 2. BioRxiv, pages 2025–02. find
  7. Qingyu Chen, Yan Hu, Xueqing Peng, Qianqian Xie, Qiao Jin, Aidan Gilson, Maxwell B Singer, Xuguang Ai, Po-Ting Lai, Zhizheng Wang, and 1 others. 2025. Benchmarking large language models for biomedical natural language processing applications and recommendations. Nature communications, 16(1):3280. find
  8. Wenduo Cheng, Junhong Shen, Mikhail Khodak, Jian Ma, and Ameet Talwalkar. 2025a. L2g: Repurposing language models for genomics tasks. Transactions on Machine Learning Research. find
  9. Wenduo Cheng, Zhenqiao Song, Yang Zhang, Shike Wang, Danqing Wang, Muyu Yang, Lei Li, and Jian Ma. 2025b. Dnalongbench: a benchmark suite for long-range dna prediction tasks. Nature Communications, 16(1):10108. find
  10. The ENCODE Project Consortium. 2012. An integrated encyclopedia of dna elements in the human genome. Nature, 489(7414):57–74. find
  11. Hugo Dalla-Torre, Liam Gonzalez, Javier Mendoza-Revilla, Nicolas Lopez Carranza, Adam Henryk Grzywaczewski, Francesco Oteri, Christian Dallago, Evan Trop, Bernardo P de Almeida, Hassan Sirelkha-tim, and 1 others. 2025. Nucleotide transformer: building and evaluating robust foundation models for huma… find
  12. Bernardo P de Almeida, Guillaume Richard, Hugo Dalla-Torre, Christopher Blum, Lorenz Hexemer, Priyanka Pandey, Stefan Laurent, Chandana Rajesh, Marie Lopez, Alexandre Laterre, and 1 others. 2025. A multimodal conversational agent for dna, rna and protein tasks. Nature Machine Intelligence, pages 1–1… find
  13. René Dreos, Giovanna Ambrosini, Rouayda Cavin Périer, and Philipp Bucher. 2013. Epd and epdnew, high-quality promoter resources in the next-generation sequencing era. Nucleic acids research, 41(D1):D157–D164. find
  14. Adibvafa Fallahpour, Andrew Magnuson, Purav Gupta, Shihao Ma, Jack Naimer, Arnav Shah, Haonan Duan, Omar Ibrahim, Hani Goodarzi, Chris J. Maddison, and BO WANG. 2025. Bioreason: Incentivizing multimodal biological reasoning within a DNA-LLM model. In The Thirty-ninth Annual Conference on Neural Info… find
  15. Veniamin Fishman, Yuri Kuratov, Aleksei Shmelev, Maxim Petrov, Dmitry Penzar, Denis Shepelin, Nikolay Chekanov, Olga Kardymon, and Mikhail Burtsev. 2025. Gena-lm: a family of open-source foundational dna language models for long sequences. Nucleic Acids Research, 53(2):gkae1310. find
  16. Google. 2025. Gemini 3 pro: Best for complex tasks and bringing creative concepts to life. Accessed: 2025-11-18. find
  17. Charles E. Grant, Timothy L. Bailey, and William Stafford Noble. 2011. Fimo: scanning for occurrences of a given motif. Bioinformatics, 27(7):1017–1018. find
  18. Katarína Grešová, Vlastimil Martinek, David Čechák, Petr Šimeček, and Panagiotis Alexiou. 2023. Genomic benchmarks: a collection of datasets for genomic sequence classification. BMC Genomic Data, 24(1):25. find
  19. Li Hao, He CAO, Bin Feng, Daniel Shao, Xiangru Tang, Zhiyuan Yan, Yonghong Tian, Li Yuan, and Yu Li. 2025. Beyond chemical QA: Evaluating LLM’s chemical reasoning with modular chemical operations. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks… find
  20. Jiyue Jiang, Pengan Chen, Jiuming Wang, Dongchen He, Ziqin Wei, Liang Hong, Licheng Zong, Sheng Wang, Qinze Yu, Zixian Ma, and 1 others. 2025. Benchmarking large language models on multiple tasks in bioinformatics nlp with prompting. arXiv preprint arXiv:2503.04013. — [arXiv:2503.04013](https://arxiv.org/abs/2503.04013) find
  21. Minghui Jiang, James Anderson, Joel Gillespie, and Martin Mayne. 2008. ushuffle: A useful tool for shuffling biological sequences while preserving the k-let counts. BMC Bioinformatics, 9(1):192. find
  22. Chia Hsiang Kao, Evan Trop, McKinley Polen, Yair Schiff, Bernardo P de Almeida, Aaron Gokaslan, Thomas PIERROT, and Volodymyr Kuleshov. 2024. Advancing dna language models: The genomics long-range benchmark. In ICLR 2024 Workshop on Machine Learning for Genomics Explorations. find
  23. Jon M Laurent, Joseph D Janizek, Michael Ruzo, Michaela M Hinks, Michael J Hammerling, Siddharth Narayanan, Manvitha Ponnapati, Andrew D White, and Samuel G Rodriques. 2024. Lab-bench: Measuring capabilities of language models for biology research. arXiv preprint arXiv:2407.10362.
  24. Zehui Li, Vallijah Subasri, Yifei Shen, Dongsheng Li, Wentao Gu, Guy-Bart Stan, Yiren Zhao, and Caihua Shan. 2025. Omni-DNA: A genomic model supporting sequence understanding, long-context, and textual annotation. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  25. Adi Lin, Bin Xie, Cheng Ye, Cheng Wang, Duoyuan Chen, Ercheng Wang, Fanfeng Lu, Guirong Xue, Haiqiang Zhang, Jiajie Zhan, Jianfeng Zhang, Jiang-shuan Pang, Jianqiang Liang, Jiawei Lin, Jiaxin Ma, Jie Hu, Jing Ma, Jinni Dong, Jiongzhen Li, and 43 others. 2025. Genos: A human-centric genomic foundatio… find
  26. Junling Liu, Peilin Zhou, Yining Hua, Dading Chong, Zhongyu Tian, Andrew Liu, Helin Wang, Chenyu You, Zhenhua Guo, LEI ZHU, and Michael Lingzhi Li. 2023. Benchmarking large language models on cmexam - a comprehensive chinese medical exam dataset. In Advances in Neural Information Processing Systems,… find
  27. Owen Queen, Harrison G Zhang, and James Zou. 2025. CGBench: Benchmarking language model scientific reasoning for clinical genetics research. In The Thirty-ninth Annual Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  28. Andres M. Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. 2024. Augmenting large language models with chemistry tools. Nature Machine Intelligence, 6(5):525– 535. find
  29. Aaron R. Quinlan and Ira M. Hall. 2010. Bedtools: a flexible suite of utilities for comparing genomic features. Bioinformatics, 26(6):841–842. find
  30. Frederikke Isa Marin, Felix Teufel, Marc Horlacher, Dennis Madsen, Dennis Pultz, Ole Winther, and Wouter Boomsma. 2024. BEND: Benchmarking DNA language models on biologically meaningful tasks. In The Twelfth International Conference on Learning Representations. find
  31. Meta. 2025. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. Accessed: 2025-04-05. find
  32. National Center for Biotechnology Information. 1988– 2025. National Center for Biotechnology Information (NCBI). https://www.ncbi.nlm.nih.gov/. ↗
  33. Eric Nguyen, Michael Poli, Matthew G. Durrant, Brian Kang, Dhruva Katrekar, David B. Li, Liam J. Bartie, Armin W. Thomas, Samuel H. King, Garyk Brixi, Jeremy Sullivan, Madelena Y. Ng, Ashley Lewis, Aaron Lou, Stefano Ermon, Stephen A. Baccus, Tina Hernandez-Boussard, Christopher Ré, Patrick D. Hsu,… find
  34. Eric Nguyen, Michael Poli, Marjan Faizi, Armin W Thomas, Michael Wornow, Callum Birch-Sykes, Stefano Massaroli, Aman Patel, Clayton M. Rabideau, Yoshua Bengio, Stefano Ermon, Christopher Re, and Stephen Baccus. 2023. HyenaDNA: Long-range genomic sequence modeling at single nucleotide resolution. In… find
  35. Nuala A. O’Leary, Mathew W. Wright, J. Rodney Bris-ter, Stacy Ciufo, Diana Haddad, Rich McVeigh, Bhanu Rajput, Barbara Robbertse, Brian Smith-White, Danso Ako-Adjei, Alexander Astashyn, Azat Badretdin, Yiming Bao, Olga Blinkova, Vyacheslav Brover, Vyacheslav Chetvernin, Jinna Choi, Eric Cox, Olga Er… find
  36. OpenAI. 2025. Gpt-5.1: A smarter, more conversational chatgpt. Accessed: 2025-11-12. find
  37. Aman Patel, Arpita Singhal, Austin Wang, Anusri Pampari, Maya Kasowski, and Anshul Kundaje. 2024. DART-eval: A comprehensive DNA language model evaluation benchmark on regulatory DNA. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track. find
  38. Ieva Rauluseviciute, Rafael Riudavets-Puig, Romain Blanc-Mathieu, Jaime A Castro-Mondragon, Katalin Ferenc, Vipin Kumar, Roza Berhanu Lemma, Jérémy Lucas, Jeanne Chèneby, Damir Baranasic, Aziz Khan, Oriol Fornes, Sveinung Gundersen, Morten Johansen, Eivind Hovig, Boris Lenhard, Albin Sandelin, Wyeth… find
  39. Jaspar 2024: 20th anniversary of the open-access database of transcription factor binding profiles. Nucleic Acids Research, 52(D1):D174–D182. find
  40. Yair Schiff, Chia Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu, and Volodymyr Kuleshov. 2024. Caduceus: Bi-directional equivariant long-range DNA sequence modeling. In Forty-first International Conference on Machine Learning. find
  41. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, and 1 others. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172–180.
  42. Ming Su, Dali Han, Jerome Boyd-Kirkup, Xiaoming Yu, and Jing-Dong J Han. 2014. Evolution of alu elements toward enhancers. Cell reports, 7(2):376– 385. xAI. 2025. Grok 4.1 fast and agent tools api. Accessed: 2025-11-19. find
  43. Yinggan Xu, Hana Kimlee, Yijia Xiao, and Di Luo. 2025. Advancing ai-scientist understanding: Making llm think like a physicist with interpretable reasoning. arXiv preprint arXiv:2504.01911. — [arXiv:2504.01911](https://arxiv.org/abs/2504.01911) find
  44. Ming Yin, Yuanhao Qu, Dyllan Liu, Ling Yang, Le Cong, and Mengdi Wang. 2025. Genome-bench: A scientific reasoning benchmark from real-world expert discussions. bioRxiv, pages 2025–06. find
  45. Yizhen Zheng, Huan Yee Koh, Jiaxin Ju, Anh T. N. Nguyen, Lauren T. May, Geoffrey I. Webb, and Shirui Pan. 2025. Large language models for scientific discovery in molecular property prediction. Nature Machine Intelligence, 7(3):437–447. find
  46. Zhihan Zhou, Yanrong Ji, Weijian Li, Pratik Dutta, Ramana V Davuluri, and Han Liu. 2024. DNABERT-2: Efficient foundation model and benchmark for multi-species genomes. In The Twelfth International Conference on Learning Representations. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/long-2026-genomeqa-benchmarking-general-large-language/