📄 BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models

Paper · 2023 · Xiangru Tang, Bill Qian, Rick Gao et al.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

🏛️ Institutions

🏢GoogleMountain View · United States533 works · 443,885 citations🎓Yale UniversityNew Haven · United States112 works · 14,009 citations

🧭 Fields, topics and concepts

Cite

Cite as Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen and Mark Gerstein. 2023. BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models. arXiv:2308.16458.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2308.16458 · arXiv 2308.16458

@misc{tang2023biocoder,
  title = {{BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models}},
  author = {Xiangru Tang, Bill Qian, Rick Gao, Jiakang Chen, Xinyun Chen and Mark Gerstein},
  year = {2023},
  doi = {10.48550/arxiv.2308.16458},
  eprint = {2308.16458},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2308.16458}
}

References

64 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

1. Allal, L. B., Li, R., Kocetkov, D. et al. (2023). Santacoder: don’t reach for the stars! arXiv preprint arXiv:2301.03988.
2. Askell, A., Bai, Y., Chen, A. et al. (2021). A general language assistant as a laboratory for alignment. arXiv preprint arXiv:2112.00861. find
3. Athiwaratkun, B., Gouda, S. K., Wang, Z. et al. (2023). Multi-lingual evaluation of code generation models.
4. Austin, J., Odena, A., Nye, M. et al. (2021). Program synthesis with large language models.
5. Bai, Y., Jones, A., Ndousse, K. et al. (2022). Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find
6. Barke, S., James, M. B. and Polikarpova, N. (2023). Grounded copilot: How programmers interact with code-generating models. Proceedings of the ACM on Programming Languages, 7(OOPSLA1), 85–111. find
7. Biderman, S., Schoelkopf, H., Anthony, Q. et al. (2023). Pythia: A suite for analyzing large language models across training and scaling. find
8. Bommasani, R., Hudson, D. A., Adeli, E. et al. (2022). On the opportunities and risks of foundation models.
9. Brown, T. B., Mann, B., Ryder, N. et al. (2020). Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
10. Bui, N. D., Le, H., Wang, Y. et al. (2023). Codetf: One-stop transformer library for state-of-the-art code llm. arXiv preprint arXiv:2306.00029. — [arXiv:2306.00029](https://arxiv.org/abs/2306.00029) find
11. Cassano, F., Gouwar, J., Nguyen, D. et al. (2023). MultiPL-E: a scalable and polyglot approach to benchmarking neural code generation. IEEE Transactions on Software Engineering, pages 1–17. find
12. BioCoder Chen, B., Zhang, F., Nguyen, A. et al. (2022). Codet: Code generation with generated tests. arXiv preprint arXiv:2207.10397. find
13. Chen, M., Tworek, J., Jun, H. et al. (2021). Evaluating large language models trained on code.
14. Chen, T. Y., Ho, J. W., Liu, H. et al. (2009). An innovative approach for testing bioinformatics programs using metamorphic testing. BMC Bioinformatics, 10(1), 24. find
15. Chen, X., Lin, M., SchÃd’rli, N. et al. (2023). Teaching large language models to self-debug. find
16. Chowdhery, A., Narang, S., Devlin, J. et al. (2022). Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
17. Christopoulou, F., Lampouras, G., Gritta, M. et al. (2022). PanGu-Coder: program synthesis with function-level language modeling. arXiv preprint arXiv:2207.11280.
18. Du, X., Liu, M., Wang, K. et al. (2023). Classeval: A manually-crafted benchmark for evaluating llms on class-level code generation. arXiv preprint arXiv:2308.01861. — [arXiv:2308.01861](https://arxiv.org/abs/2308.01861) find
19. Fried, D., Aghajanyan, A., Lin, J. et al. (2023). Incoder: A generative model for code infilling and synthesis. find
20. Gao, L., Tow, J., Biderman, S. et al. (2022). Eleutherai/lm-evaluation-harness: v0.3.0. find
21. Hendrycks, D., Basart, S., Kadavath, S. et al. (2021). Measuring coding challenge competence with apps.
22. NeurIPS. Hoffmann, J., Borgeaud, S., Mensch, A. et al. (2022). Training compute-optimal large language models.
23. Khlaaf, H., Mishkin, P., Achiam, J. et al. (2022). A hazard analysis framework for code synthesis large language models. arXiv preprint arXiv:2207.14157. — [arXiv:2207.14157](https://arxiv.org/abs/2207.14157) find
24. Lai, Y., Li, C., Wang, Y. et al. (2022). DS-1000: a natural and reliable benchmark for data science code generation. ArXiv, abs/2211.11501. find
25. Lee, T., Hong, S., Ahn, J. et al. (2023). Who wrote this code? watermarking for code generation. arXiv preprint arXiv:2305.15060.
26. Li, R., Allal, L. B., Zi, Y. et al. (2023). Starcoder: may the source be with you! Li, Y., Choi, D., Chung, J. et al. (2022). Competition-level code generation with alphacode. arXiv preprint arXiv:2203.07814.
27. Liu, T., Xu, C. and McAuley, J. (2023). Repobench: Benchmarking repository-level code auto-completion systems.
28. Luo, Z., Xu, C., Zhao, P. et al. (2023). Wizardcoder: Empowering code large language models with evol-instruct. arXiv preprint arXiv:2306.08568. find
29. Mikolov, T., Chen, K., Corrado, G. et al. (2013). Efficient estimation of word representations in vector space.
30. Mitchell, M., Luccioni, A. S., Lambert, N. et al. (2023). Measuring data. find
31. MOI, A., Patry, N., Cistac, P. et al. (2022). huggingface/tokenizers: Rust 0.13.2. find
32. Muennighoff, N., Wang, T., Sutawika, L. et al. (2023a). Crosslingual generalization through multitask finetuning.
33. Muennighoff, N., Liu, Q., Zebaze, A. et al. (2023b). Octopack: Instruction tuning code large language models. arXiv preprint arXiv:2308.07124. find
34. Nijkamp, E., Pang, B., Hayashi, H. et al. (2023a). CodeGen: an open large language model for code with multi-turn program synthesis. In The Eleventh International Conference on Learning Representations. find
35. Nijkamp, E., Hayashi, H., Xiong, C. et al. (2023b). Codegen2: Lessons for training llms on programming and natural languages. arXiv preprint.
36. Olausson, T. X., Inala, J. P., Wang, C. et al. (2023). Demystifying gpt self-repair for code generation. arXiv preprint arXiv:2306.09896.
37. OpenAI (2023). Gpt-4 technical report.
38. Orlanski, G., Xiao, K., Garcia, X. et al. (2023). Measuring the impact of programming language distribution. In International Conference on Machine Learning, pages 26619–26645.
39. PMLR. Pan, R., Ibrahimzada, A. R., Krishna, R. et al. (2023). Understanding the effectiveness of large language models in code translation. arXiv preprint arXiv:2308.03109.
40. Patil, S. G., Zhang, T., Wang, X. et al. (2023). Gorilla: Large language model connected with massive apis. arXiv preprint arXiv:2305.15334.
41. Qin, Y., Liang, S., Ye, Y. et al. (2023). Toolllm: Facilitating large language models to master 16000+ real-world apis. arXiv preprint arXiv:2307.16789.
42. Radford, A., Wu, J., Child, R. et al. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8), 9. find
43. Russell, P. H., Johnson, R. L., Ananthan, S. et al. (2018). A large-scale analysis of bioinformatics code on github. PloS one, 13(10), e0205898. find
44. Shen, B., Zhang, J., Chen, T. et al. (2023). Pangu-coder2: Boosting large language models for code with ranking feedback. arXiv preprint arXiv:2307.14936. — [arXiv:2307.14936](https://arxiv.org/abs/2307.14936) find
45. Shrivastava, D., Kocetkov, D., de Vries, H. et al. (2023). Repofusion: Training code models to understand your repository. find
46. Thoppilan, R., Freitas, D. D., Hall, J. et al. (2022). Lamda: Language models for dialog applications.
47. Touvron, H., Lavril, T., Izacard, G. et al. (2023). LLaMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
48. Wang, S., Li, Z., Qian, H. et al. (2022a). Recode: Robustness evaluation of code generation models. arXiv preprint arXiv:2212.10264. find
49. Wang, Y., Wang, W., Joty, S. et al. (2021). Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. find
50. Wang, Y., Le, H., Gotmare, A. D. et al. (2023a). Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922. find
51. Wang, Z., Zhou, S., Fried, D. et al. (2022b). Execution-based evaluation for open-domain code generation. arXiv preprint arXiv:2212.10481.
52. Wang, Z., Cuenca, G., Zhou, S. et al. (2023b). Mconala: A benchmark for code generation from multiple natural languages. find
53. Wong, M.-F., Guo, S., Hang, C.-N. et al. (2023). Natural language generation and understanding of big code for ai-assisted programming: A review. Entropy, 25(6), 888. find
54. Xu, Q., Hong, F., Li, B. et al. (2023). On the tool manipulation capability of open-source large language models. arXiv preprint arXiv:2305.16504.
55. Yu, H., Shen, B., Ran, D. et al. (2023). Codereval: A benchmark of pragmatic code generation with generative pre-trained models. arXiv preprint. find
56. Yuan, Z., Liu, J., Zi, Q. et al. (2023). Evaluating instruction-tuned large language models on code comprehension and generation. arXiv preprint arXiv:2308.01240. — [arXiv:2308.01240](https://arxiv.org/abs/2308.01240) find
57. Zan, D., Chen, B., Yang, D. et al. (2022a). Cert: Continual pre-training on sketches for library-oriented code generation. find
58. Zan, D., Chen, B., Lin, Z. et al. (2022b). When language model meets private library. In Conference on Empirical Methods in Natural Language Processing. find
59. Zan, D., Chen, B., Zhang, F. et al. (2023). Large language models meet nl2code: A survey. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7443–7464. find
60. Zeng, A., Liu, X., Du, Z. et al. (2022). GLM-130B: an open bilingual pre-trained model. arXiv preprint arXiv:2210.02414.
61. Zhang, F., Chen, B., Zhang, Y. et al. (2023a). find
62. Zhang, S. D., Tigges, C., Biderman, S. et al. (2023b). Can transformers learn to solve problems recursively? arXiv preprint arXiv:2305.14699.
63. Zheng, Q., Xia, X., Zou, X. et al. (2023). Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. find
64. Zhuo, T. Y. (2023). Large language models are state-of-the-art evaluators of code generation. arXiv preprint arXiv:2304.14317.

Updated 3 November 2026

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2023-biocoder-benchmark-bioinformatics-code-generation/