Elamur Research⌕
☰
home / papers / 2024 / SciCode: A Research Coding Benchmark Curated by Scientists

SciCode: A Research Coding Benchmark Curated by Scientists

Paper · 2024 · Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang et al.

Summary

Many established LM benchmarks are now saturated, and synthetic hard benchmarks may not reflect real use. SciCode tests whether models can write code for genuine research tasks in the natural sciences. Scientists from 16 subfields of mathematics, physics, chemistry, biology and materials science built the problems, mostly from scripts they use in their own research. These scripts are rarely public, which limits training-data exposure. Each main problem is split into subproblems that the model solves as separate Python functions and then combines. The benchmark supplies docstrings, optional background text written by scientists, gold solutions and test cases, so scoring is automatic and based on execution. Evaluation can vary by including background or by conditioning on gold versus model-generated earlier steps. Ten current models were tested, and all performed poorly. The strongest, Claude3.5-Sonnet, fully solves only 4.6% of main problems in the most realistic setting. This shows a large gap between today's LMs and dependable scientific coding assistants.

Key points

  • The dataset has 80 main problems split into 338 subproblems, with a median of 3 and a maximum of 15 per problem. The development split holds 15 main problems (50 subproblems), and the test split holds 65 (288 subproblems).
  • Every problem, gold solution and test case was annotated and checked by at least two senior researchers (PhD level or higher). Problems were chosen to avoid any overlap with public datasets.
  • Without background, main-problem results are low:
    • Claude3.5-Sonnet: 4.6%
    • Deepseek-Coder-v2 (best open model): 3.1%
    • GPT-4o and Claude3-Opus: 1.5%
    • Llama-3-70B-Instruct and Mixtral-8x22B-Instruct: no main problems solved, though they pass some subproblems
  • Scientist-written background helps every model, but the best score rises only to 12.3% (Claude3.5-Sonnet), with GPT-4o and GPT-4-Turbo at 9.2%.
  • Supplying gold solutions for earlier subproblems usually raises subproblem accuracy. Gains reverse beyond about 9 prior steps, likely because of long-context difficulty. Data and leaderboard are at scicode-bench.github.io.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksInformation Systems · 1,730 worksDecision Sciences · 249 worksInformation Systems and Management · 52 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao, Ofir Press, Jamie Callan, Eliu Huerta and Hao Peng. 2024. SciCode: A Research Coding Benchmark Curated by Scientists. arXiv:2407.13168.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2407.13168 · arXiv 2407.13168

@misc{tian2024scicode,
  title = {{SciCode: A Research Coding Benchmark Curated by Scientists}},
  author = {Minyang Tian, Luyu Gao, Shizhuo Dylan Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, Shengyan Liu, Di Luo, Yutao Ma, Hao Tong, Kha Trinh, Chenyu Tian, Zihan Wang, Bohao Wu, Yanyu Xiong, Shengzhu Yin, Minhui Zhu, Kilian Lieret, Yanxin Lu, Genglin Liu, Yufeng Du, Tianhua Tao, Ofir Press, Jamie Callan, Eliu Huerta and Hao Peng},
  year = {2024},
  doi = {10.48550/arxiv.2407.13168},
  eprint = {2407.13168},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2407.13168}
}

References

102 references, 32 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. M. R. AI4Science and M. A. Quantum. The impact of large language models on scientific discovery: a preliminary study using gpt-4, 2023. find
  2. AI@Meta. Llama 3 model card. 2024. Introducing the next find
  3. Anthropic. https://www.anthropic.com/news/claude-3-family. generation ↗
  4. Anthropic. Introducing the next generation https://www.anthropic.com/news/claude-3-5-sonnet. of of claude, claude 3.5, 2024. 2024. ↗
  5. G. Ashiotis, A. Deschildre, Z. Nawaz, J. P. Wright, D. Karkoulis, F. E. Picca, and J. Kieffer. The fast azimuthal integration python library: pyfai. Journal of applied crystallography, 48(2):510–519, 2015. find
  6. A. Ashkin. Acceleration and trapping of particles by radiation pressure. Physical review letters, 24(4):156, 1970. find
  7. J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le, and C. Sutton. Program synthesis with large language models, 2021.
  8. J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y. Yao, B.… — Qwen Technical Report
  9. Y. Bazilevs and T. J. R. Hughes. Weak imposition of dirichlet boundary conditions in fluid mechanics. Computers & Fluids, 36:12–26, 2007. find
  10. J. Chen, X. Guo, C. Boyd, S. Bettler, C. Kengle, D. Chaudhuri, F. Hoveyda, A. Husain, J. Schneeloch, G. Gu, et al. Consistency between reflection momentum-resolved electron energy loss spectroscopy and optical spectroscopy measurements of the long-wavelength density response of bi 2 sr 2 cacu 2 o 8+… find
  11. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such,… — Evaluating Large Language Models Trained on Code
  12. M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, A. Ray, R. Puri, G. Krueger, M. Petrov, H. Khlaaf, G. Sastry, P. Mishkin, B. Chan, S. Gray, N. Ryder, M. Pavlov, A. Power, L. Kaiser, M. Bavarian, C. Winter, P. Tillet, F. P. Such,… — Evaluating Large Language Models Trained on Code
  13. Z. Chen, A. H. Cano, A. Romanou, A. Bonnet, K. Matoba, F. Salvi, M. Pagliardini, S. Fan, A. Köpf, A. Mohtashami, A. Sallinen, A. Sakhaeirad, V. Swamy, I. Krawczuk, D. Bayazit, A. Marmet, S. Montariol, M.-A. Hartley, M. Jaggi, and A. Bosselut. Meditron-70b: Scaling medical pretraining for large langu… — MEDITRON-70B: Scaling Medical Pretraining for Large Language Models
  14. K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems. volume abs/2110.14168, 2021.
  15. M. Cross and H. Greenside. Pattern Formation and Dynamics in Nonequilibrium Systems. Cambridge University Press, 2009. find
  16. M. Cross and P. Hohenberg. Pattern formation outside of equilibrium. Reviews of Modern Physics, 65:851–1112, 1993. find
  17. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018.
  18. Y. Ding, Z. Wang, W. U. Ahmad, H. Ding, M. Tan, N. Jain, M. K. Ramanathan, R. Nallapati, P. Bhatia, D. Roth, and B. Xiang. Crosscodeeval: A diverse and multilingual benchmark for cross-file code completion. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks… find
  19. Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou. Codebert: A pre-trained model for programming and natural languages, 2020.
  20. W. M. Foulkes, L. Mitas, R. Needs, and G. Rajagopal. Quantum monte carlo simulations of solids. Reviews of Modern Physics, 73(1):33, 2001. find
  21. L. Gao, A. Madaan, S. Zhou, U. Alon, P. Liu, Y. Yang, J. Callan, and G. Neubig. Pal: program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, 2023. find
  22. D. T. Gillespie. A general method for numerically simulating the stochastic time evolution of coupled chemical reactions. Journal of Computational Physics, 22(4):403–434, 1976. find
  23. D. T. Gillespie. Exact stochastic simulation of coupled chemical reactions. The Journal of Physical Chemistry, 81(25), 1977. find
  24. N. A. Gumerov and R. Duraiswami. Recursions for the computation of multipole translation and rotation coefficients for the 3-d helmholtz equation. SIAM Journal on Scientific Computing, 25(4):1344–1381, 2004. find
  25. D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. K. Li, F. Luo, Y. Xiong, and W. Liang. Deepseek-coder: When the large language model meets programming – the rise of code intelligence, 2024. find
  26. F. D. M. Haldane. Model for a quantum hall effect without landau levels: Condensed-matter realization of the" parity anomaly". Physical review letters, 61(18):2015, 1988. find
  27. D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, et al. Measuring coding challenge competence with apps. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  28. D. Hendrycks, S. Basart, S. Kadavath, M. Mazeika, A. Arora, E. Guo, C. Burns, S. Puranik, H. He, D. Song, and J. Steinhardt. Measuring coding challenge competence with apps. NeurIPS, 2021.
  29. D. Hendrycks, C. Burns, S. Kadavath, A. Arora, S. Basart, E. Tang, D. Song, and J. Steinhardt. Measuring mathematical problem solving with the math dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  30. A. A. Husain, E. W. Huang, M. Mitrano, M. S. Rak, S. I. Rubeck, X. Guo, H. Yang, C. Sow, Y. Maeno, B. Uchoa, et al. Pines’ demon observed as a 3d acoustic plasmon in sr2ruo4. Nature, 621(7977):66–70, 2023. find
  31. A. Hyvärinen and E. Oja. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):411–430, 2000. find
  32. J. K. Jain and P. B. Allen. Dielectric response of a semi-infinite layered electron gas and raman scattering from its bulk and surface plasmons. Physical Review B, 32(2):997, 1985. find
  33. A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. de las Casas, E. B. Hanna, F. Bressand, G. Lengyel, G. Bour, G. Lample, L. R. Lavaud, L. Saulnier, M.-A. Lachaux, P. Stock, S. Subramanian, S. Yang, S. Antoniak, T. L. Scao, T. Gervet, T. Lavril, T. Wang, T. L… find
  34. C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan. Swe-bench: Can language models resolve real-world github issues?, 2024.
  35. S. Khatri, K. Sharma, and M. M. Wilde. Information-theoretic aspects of the generalized amplitude-damping channel. Physical Review A, 102(1):012401, 2020. find
  36. D. Kiela, M. Bartolo, Y. Nie, D. Kaushik, A. Geiger, Z. Wu, B. Vidgen, G. Prasad, A. Singh, P. Ringshia, Z. Ma, T. Thrush, S. Riedel, Z. Waseem, P. Stenetorp, R. Jia, M. Bansal, C. Potts, and A. Williams. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North A… find
  37. W. Kohn and L. J. Sham. Self-consistent equations including exchange and correlation effects. Physical review, 140(4A):A1133, 1965. find
  38. A. N. Kolmogorov and V. H. Crespi. Registry-dependent interlayer potential for graphitic systems. Physical Review B, 71(23):235415, 2005. find
  39. M. J. Krogstad. Diffuse Scattering and Local Order in Lead-Based Relaxor Ferroelectrics. Northern Illinois University, 2018. find
  40. M. J. Krogstad, S. Rosenkranz, J. M. Wozniak, G. Jennings, J. P. Ruff, J. T. Vaughey, and R. Osborn. Reciprocal space imaging of ionic correlations in intercalation compounds. Nature materials, 19(1):63–68, 2020. find
  41. K. Krongchon, T. Rakib, S. Pathak, E. Ertekin, H. T. Johnson, and L. K. Wagner. Registry-dependent potential energy and lattice corrugation of twisted bilayer graphene from quantum monte carlo. Physical Review B, 108(23):235403, 2023. find
  42. Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, S. W. tau Yih, D. Fried, S. Wang, and T. Yu. Ds-1000: A natural and reliable benchmark for data science code generation, 2022. find
  43. A. Lewkowycz, A. Andreassen, D. Dohan, E. Dyer, H. Michalewski, V. Ramasesh, A. Slone, C. Anil, I. Schlag, T. Gutman-Solo, Y. Wu, B. Neyshabur, G. Gur-Ari, and V. Misra. Solving quantitative reasoning problems with language models, 2022.
  44. Q. Li, L. Cui, X. Zhao, L. Kong, and W. Bi. Gsm-plus: A comprehensive benchmark for evaluating the robustness of llms as mathematical problem solvers. ArXiv preprint, abs/2402.19255, 2024.
  45. R. Li, L. B. allal, Y. Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. LI, J. Chim, Q. Liu, E. Zheltonozhskii, T. Y. Zhuo, T. Wang, O. Dehaene, J. Lamy-Poirier, J. Monteiro, N. Gontier, M.-H. Yee, L. K. Umapathi, J. Zhu, B. Lipkin, M. Oblokulov, Z. Wang, R. Murthy, J. T. Stillerman,… find
  46. T. Li, S. Kheifets, D. Medellin, and M. G. Raizen. Measurement of the instantaneous velocity of a brownian particle. Science, 328(5986):1673–1675, 2010. find
  47. T. Li and M. G. Raizen. Brownian motion at short time scales. Annalen der Physik, 525(4):281– 295, 2013. find
  48. Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, W. Lu, N. Smetanin, A. dos Santos Costa, M. Fazel-Zarandi, T. Sercu, S. Candido, et al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. bioRxiv, 2022. find
  49. J. Liu, C. S. Xia, Y. Wang, and L. Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023.
  50. S. Liu, Z.-q. Yin, and T. Li. Prethermalization and nonreciprocal phonon transport in a levitated optomechanical array. Advanced Quantum Technologies, 3(3):1900099, 2020. find
  51. A. J. Lotka. Elements of physical biology. Williams & Wilkins, 1925. find
  52. A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y. Wei, T. Liu, M. Tian, D. Kocetkov, A. Zucker, Y. Belkada, Z. Wang, Q. Liu, D. Abulkhanov, I. Paul, Z. Li, W.-D. Li, M. Risdal, J. Li, J. Zhu, T. Y. Zhuo, E. Zheltonozhskii, N. O. O. Dade, W. Yu, L.… find
  53. R. Luo, L. Sun, Y. Xia, T. Qin, S. Zhang, H. Poon, and T.-Y. Liu. Biogpt: generative pre-trained transformer for biomedical text generation and mining. Briefings in Bioinformatics, 23, 09 2022. find
  54. Z. Maki, M. Nakagawa, and S. Sakata. Remarks on the unified model of elementary particles. Progress of Theoretical Physics, 28(5):870–880, 1962. find
  55. E. Marinari, G. Parisi, and J. Ruiz-Lorenzo. Numerical simulations of spin glass systems. In Spin Glasses and Random Fields, pages 59–98. World Scientific, 1998. find
  56. N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller. Equation of state calculations by fast computing machines. The journal of chemical physics, 21(6):1087– 1092, 1953. find
  57. M. Mézard, G. Parisi, and M. A. Virasoro. Spin glass theory and beyond: An Introduction to the Replica Method and Its Applications, volume 9. World Scientific Publishing Company, 1987. find
  58. S.-y. Miao, C.-C. Liang, and K.-Y. Su. A diverse corpus for evaluating and developing English math word problem solvers. In Proc. of ACL, 2020. find
  59. P. Moon and M. Koshino. Energy spectrum and quantum hall effect in twisted bilayer graphene. Physical Review B, 85(19):195458, 2012. find
  60. E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y. Zhou, S. Savarese, and C. Xiong. Codegen: An open large language model for code with multi-turn program synthesis, 2023. find
  61. T. Ohlsson and H. Snellman. Three flavor neutrino oscillations in matter. Journal of Mathematical Physics, 41(5):2768–2788, 2000. find
  62. L. Onsager. Crystal statistics. i. a two-dimensional model with an order-disorder transition. Physical Review, 65(3-4):117, 1944. find
  63. L. Onsager. Crystal statistics. i. a two-dimensional model with an order-disorder transition. Phys. Rev., 65:117–149, Feb 1944. find
  64. OpenAI. Gpt-4 technical report, 2023.
  65. OpenAI. Hello gpt-4o, 2024. https://openai.com/index/hello-gpt-4o/. ↗
  66. OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M… — GPT-4 Technical Report
  67. R. Osborn. Mapping structural correlations in real space. Acta Crystallographica Section B, 79(2):99–100, Apr 2023. find
  68. W. Ouyang, D. Mandelli, M. Urbakh, and O. Hod. Nanoserpents: Graphene nanoribbon motion on two-dimensional hexagonal materials. Nano letters, 18(9):6009–6016, 2018. find
  69. G. Parisi. A sequence of approximated solutions to the sk model for spin glasses. Journal of Physics A Mathematical General, 13(4):L115–L121, 1980. find
  70. A. Patel, S. Bhattamishra, and N. Goyal. Are NLP models really able to solve simple math word problems? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2021.
  71. S. Pathak, T. Rakib, R. Hou, A. Nevidomskyy, E. Ertekin, H. T. Johnson, and L. K. Wagner. Accurate tight-binding model for twisted bilayer graphene describes topological flat bands without geometric relaxation. Physical Review B, 105(11):115141, 2022. find
  72. D. Rein, B. L. Hou, A. C. Stickland, J. Petty, R. Y. Pang, J. Dirani, J. Michael, and S. R. Bowman. Gpqa: A graduate-level google-proof q&a benchmark, 2023.
  73. P. J. Reynolds, D. M. Ceperley, B. J. Alder, and W. A. Lester Jr. Fixed-node quantum monte carlo for molecules. The Journal of Chemical Physics, 77(11):5593–5603, 1982. find
  74. M. Rezaeian and A. Grant. Computation of total capacity for discrete memoryless multiple-access channels. IEEE transactions on information theory, 50(11):2779–2784, 2004. find
  75. B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y. Adi, J. Liu, R. Sauvestre, T. Remez, J. Rapin, A. Kozhevnikov, I. Evtimov, J. Bitton, M. Bhatt, C. C. Ferrer, A. Grattafiori, W. Xiong, A. Defossez, J. Copet, F. Azhar, H. Touvron, L. Martin, N. Usunier, T. Scialom, and G. Synnaev… — Code Llama: Open Foundation Models for Code
  76. I. Ruff, A. Baranyai, G. Pálinkás, and K. Heinzinger. Grand canonical Monte Carlo simulation of liquid argon. The Journal of Chemical Physics, 85(4):2169–2177, 08 1986. find
  77. L. Schatzki, L. Ma, E. Solomonik, and E. Chitambar. Tensor rank and other multipartite entanglement measures of graph states. arXiv preprint arXiv:2209.06320, 2022. — [arXiv:2209.06320](https://arxiv.org/abs/2209.06320) find
  78. Z. Shao, P. Wang, Q. Zhu, R. Xu, J. Song, M. Zhang, Y. Li, Y. Wu, and D. Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
  79. D. Sherrington and S. Kirkpatrick. Solvable model of a spin-glass. Physical Review Letters, 35(26):1792–1796, 1975. find
  80. K. Singhal, S. Azizi, T. Tu, S. S. Mahdavi, J. Wei, H. W. Chung, N. Scales, A. Tanwani, H. Cole-Lewis, S. Pfohl, P. Payne, M. Seneviratne, P. Gamble, C. Kelly, A. Babiker, N. Schärli, A. Chowdhery, P. Mansfield, D. Demner-Fushman, B. Agüera y Arcas, D. Webster, G. S. Corrado, Y. Matias, K. Chou, J.… — Large Language Models Encode Clinical Knowledge
  81. A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, A. Kluska, A. Lewkowycz, A. Agarwal, A. Power, A. Ray, A. Warstadt, A. W. Kocurek, A. Safaya, A. Tazarv, A. Xiang, A. Parrish, A. Nie, A. Hussain, A. Askell, A. Dsouza, A. Slon… — Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
  82. M. Suzgun, N. Scales, N. Schärli, S. Gehrmann, Y. Tay, H. W. Chung, A. Chowdhery, Q. V. Le, E. H. Chi, D. Zhou, , and J. Wei. Challenging big-bench tasks and whether chain-of-thought can solve them. ArXiv preprint, abs/2210.09261, 2022.
  83. J. Swift and P. Hohenberg. Hydrodynamic fluctuations at the convective instability. Physical Review A, 15:319–328, 1977. find
  84. R. Taylor, M. Kardas, G. Cucurull, T. Scialom, A. Hartshorn, E. Saravia, A. Poulton, V. Kerkez, and R. Stojnic. Galactica: A large language model for science, 2022.
  85. G. Team, M. Reid, N. Savinov, D. Teplyashin, Dmitry, Lepikhin, T. Lillicrap, J. baptiste Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittwieser, I. Antonoglou, R. Anil, S. Borgeaud, A. Dai, K. Millican, E. Dyer, M. Glaese, T. Sottiaux, B. Lee, F. Viola, M. Reynolds, Y. Xu, J. Molloy, J. Chen,… — Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
  86. Q. Team. Introducing qwen2, June 2024. find
  87. A. V. Tkachenko and S. Maslov. Onset of natural selection in populations of autocatalytic heteropolymers. The Journal of chemical physics, 149(13), 2018. find
  88. S. Vig, A. Kogar, M. Mitrano, A. Husain, L. Venema, M. Rak, V. Mishra, P. Johnson, G. Gu, E. Fradkin, et al. Measurement of the dynamic charge response of materials using low-energy, momentum-resolved electron energy-loss spectroscopy (m-eels). SciPost Physics, 3(4):026, 2017. find
  89. V. Volterra. Variations and fluctuations of the number of individuals in animal species living together. ICES Journal of Marine Science, 3(1):3–51, 1928. find
  90. L. K. Wagner. Correlations and effective interactions from first principles using quantum monte carlo. Handbook of Materials Modeling: Methods: Theory and Modeling, pages 417–433, 2020. find
  91. A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018.
  92. X. Wang, Z. Hu, P. Lu, Y. Zhu, J. Zhang, S. Subramaniam, A. R. Loomba, S. Zhang, Y. Sun, and W. Wang. Scibench: Evaluating college-level scientific problem-solving abilities of large language models, 2024.
  93. X. Wang, Z. Wang, J. Liu, Y. Chen, L. Yuan, H. Peng, and H. Ji. Mint: Evaluating llms in multi-turn interaction with tools and language feedback. ArXiv preprint, abs/2309.10691, 2023.
  94. Y. Wang, W. Wang, S. Joty, and S. C. H. Hoi. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation, 2021. find
  95. Z. Wang, Y. Fu, A. Goyal, and S. Maslov. Emergent ecological advantage of sequential metabolic strategies in complex microbial communities. Manuscript unsubmitted, jun 2024. find
  96. Z. Wang, A. Goyal, V. Dubinkina, A. B. George, T. Wang, Y. Fridman, and S. Maslov. Complementary resource preferences spontaneously emerge in diauxic microbial communities. Nature communications, 12(1):6661, 2021. find
  97. Z. Wang, A. Goyal, and S. Maslov. The ecological consequences of microbial metabolic strategies in fluctuating environments. bioRxiv, pages 2023–07, 2023. find
  98. W. A. Wheeler, S. Pathak, K. G. Kleiner, S. Yuan, J. N. Rodrigues, C. Lorsung, K. Krongchon, Y. Chang, Y. Zhou, B. Busemeyer, et al. Pyqmc: An all-python real-space quantum monte carlo module in pyscf. The Journal of Chemical Physics, 158(11), 2023. find
  99. B. Widom. Some Topics in the Theory of Fluids. The Journal of Chemical Physics, 39(11):2808–2812, 12 1963. find
  100. P. Yin, B. Deng, E. Chen, B. Vasilescu, and G. Neubig. Learning to mine aligned code and natural language pairs from stack overflow. In International Conference on Mining Software Repositories, MSR, pages 476–486. ACM, 2018.
  101. P. Yin, W.-D. Li, K. Xiao, A. Rao, Y. Wen, K. Shi, J. Howland, P. Bailey, M. Catasta, H. Michalewski, A. Polozov, and C. Sutton. Natural language to code generation in interactive data science notebooks, 2022.
  102. Q. Zhu, D. Guo, Z. Shao, D. Yang, P. Wang, R. Xu, Y. Wu, Y. Li, H. Gao, S. Ma, et al. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931, 2024.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tian-2024-scicode-research-coding-benchmark/