Elamur Research⌕
☰
home / papers / 2026 / MMSciCode: Real-world Evaluation of Multilingual Multi-Discipline Scientific Research Coding

MMSciCode: Real-world Evaluation of Multilingual Multi-Discipline Scientific Research Coding

Paper · 2026 · Xue Xia, Zheyuan Yang, Arman Cohan et al.

Summary

Most code-generation benchmarks test one programming language, one domain, or isolated functions. They miss what makes research coding hard: turning formulas and algorithm descriptions from a paper into working code that is both computationally and scientifically correct. MMSciCode fills this gap. Its 624 function-level problems come from 285 papers and their public repositories, covering 38 subjects in six disciplines and three languages (Python, C/C++, R). Domain experts matched paper content to the code that implements it, masked the target function, and wrote unit tests that run in containers. Model output is inserted back into the original project and checked automatically.

The authors tested 23 foundation models and 2 coding agents. The best non-agentic models scored well below human experts. Agents did much better, mainly because they could check and fix their own code. The benchmark therefore exposes a real weakness in scientific coding and points to executable feedback, rather than longer reasoning, as the key lever.

Key points

  • Data sources and annotation: Papers come from ICLR 2025 orals and spotlights plus Nature Communications (2024 onward). Annotation was done by 21 experts (18 PhD students and 3 authors) over roughly 1,400 hours.
  • Top non-agentic results vs. humans: GPT-5.4 (with CoT) reached 59.9% function-level accuracy and Qwen3.5-397B-A17B reached 50.8%. Humans scored 86.7% open-book and 93.3% in the oracle setting.
  • Coding agents: Claude Code (Sonnet 4.6) reached 88.8%, while Claude-Sonnet-4.6 alone rose only from 53.2% to 55.6% with CoT. The authors credit the agent's self-verification and repository exploration for the gap, not model scale.
  • Error analysis: Failures break down into algorithmic reasoning (19%), implementation (18%), data structure/schema (14%), domain knowledge (9%), and other errors (7%).
  • Limitations: The benchmark leaves out MATLAB, Julia, Fortran, and similar languages, and the paper proposes no training methods.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Yale UniversityNew Haven · United States101 works · 13,302 citations🎓Tongji UniversityShanghai · China33 works · 2,993 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 worksComputer Science Applications · 131 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 works

Cite

Cite as Xue Xia, Zheyuan Yang, Arman Cohan and Yilun Zhao. 2026. MMSciCode: Real-world Evaluation of Multilingual Multi-Discipline Scientific Research Coding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1566

@inproceedings{xia2026mmscicode,
  title = {{MMSciCode: Real-world Evaluation of Multilingual Multi-Discipline Scientific Research Coding}},
  author = {Xue Xia, Zheyuan Yang, Arman Cohan and Yilun Zhao},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1566},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1566}
}

References

33 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. OpenAI Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Hai-Biao Bao, Boaz Barak, Ally Bennett, Tyler Bertao, N. Archer Brett, Eugene Brevdo, Greg Brockman, Sébastien Bubeck, Cheng Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Cl… — gpt-oss-120b & gpt-oss-20b Model Card
  2. Anthropic. 2025. Claude code. https://docs.anthropic.com/en/docs/ claude-code/overview. ↗
  3. Anthropic. 2026. Claude sonnet 4.6 system card. find
  4. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  5. ByteDance Seed, Yuyu Zhang, Jing Su, Yifan Sun, Chenguang Xi, Xia Xiao, Shen Zheng, Anxiang Zhang, Kaibo Liu, Daoguang Zan, Tao Sun, Jinhua Zhu, Shulin Xin, Dong Huang, Yetao Bai, Lixin Dong, Chao Li, Jianchong Chen, Hanzhi Zhou, Yifan Huang, Guanghan Ning, Xierui Song, Jiaze Chen, Siyao Liu, Kai Sh… — Seed-Coder: Let the Code Model Curate Data for Itself
  6. Jun Shern Chan, Neil Chowdhury, Oliver Jaffe, James Aung, Dane Sherburn, Evan Mays, Giulio Starace, Kevin Liu, Leon Maksin, Tejal Patwardhan, et al. 2025. Mle-bench: Evaluating machine learning agents on machine learning engineering. In The Thirteenth International Conference on Learning Representat…
  7. Hui Chen, Miao Xiong, Yujie Lu, Wei Han, Ailin Deng, Yufei He, Jiaying Wu, Yibo Li, Yue Liu, and Bryan Hooi. 2025a. Mlr-bench: Evaluating AI agents on open-ended machine learning research. CoRR, abs/2505.19955. find
  8. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  9. Q. Chen, M. Yang, L. Qin, et al. 2025b. Ai4research: A survey of artificial intelligence for scientific research. arXiv preprint arXiv:2507.01903.
  10. Ziru Chen, Shijie Chen, Yuting Ning, Qianheng Zhang, Boshi Wang, Botao Yu, Yifei Li, Zeyi Liao, Chen Wei, Zitong Lu, Vishal Dey, Mingyi Xue, Frazier N. Baker, Benjamin Burns, Daniel Adu-Ampratwum, Xuhui Huang, Xia Ning, Song Gao, Yu Su, and Huan Sun. 2025c. Scienceagentbench: Toward rigorous assessm…
  11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony S. Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur’elien Rodrig… — The Llama 3 Herd of Models
  12. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319–18345. PMLR. find
  13. Siyu Liu, Jiamin Xu, Beilin Ye, Bo Hu, David J Srolovitz, and Tongqi Wen. 2025. Mattools: Benchmarking large language models for materials science tools. arXiv preprint arXiv:2505.10852. — [arXiv:2505.10852](https://arxiv.org/abs/2505.10852) find
  14. Chunyu Miao, Henry Peng Zou, Yangning Li, Yankai Chen, Yibo Wang, Fangxin Wang, Yifan Li, Wooseong Yang, Bowei He, Xinni Zhang, Dianzhi Yu, Hanchen Yang, Hoang H. Nguyen, Yue Zhou, Jie Yang, Jizhou Guo, Wenzhe Fan, Chin-Yuan Yeh, Panpan Meng, Liancheng Fang, Jinhu Qi, Wei-Chieh Huang, Zhengyao Gu, Y… — [arXiv:2510.06186](https://arxiv.org/abs/2510.06186) find
  15. Minimax. 2026. Minimax m2.7: Early echoes of self-evolution. find
  16. Mistral AI. 2024. Codestral. find
  17. Google DeepMind. 2025. Gemini 3 flash - model card. find
  18. Google DeepMind. 2026. Gemma 4. find
  19. Cheng Peng, Xi Yang, Aokun Chen, Kaleb E Smith, Nima PourNejatian, Anthony B Costa, Cheryl Martin, Mona G Flores, Ying Zhang, Tanja Magoc, et al. 2023. A study of generative large language model for medical research and healthcare. NPJ digital medicine, 6(1):210. find
  20. Ken Gu, Ruoxi Shang, Ruien Jiang, Keying Kuang, Richard-John Lin, Donghe Lyu, Yue Mao, Youran Pan, Teng Wu, Jiaqian Yu, Yikun Zhang, Tianmai M. Zhang, Lanyi Zhu, Mike A Merrill, Jeffrey Heer, and Tim Althoff. 2024. BLADE: Benchmarking language model agents for data-driven science. In Findings of the… find
  21. Tianyu Hua, Harper Hua, Violet Xiang, Benjamin Klieger, Sang T. Truong, Weixin Liang, Fan-Yun Sun, and Nick Haber. 2025. Researchcodebench: Benchmarking llms on implementing novel machine learning research code. CoRR, abs/2506.02314. find
  22. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Kai Dang, An Yang, Rui Men, Fei Huang, Shanghaoran Quan, Xingzhang Ren, Xuancheng Ren, Jingren Zhou, and Junyang Lin. 2024. Qwen2.5-coder technical report. ArXiv, abs/2409.12186.
  23. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2025. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representatio…
  24. Qwen. 2025. Qwen3-coder: Agentic coding in the world. find
  25. Qwen. 2026. Qwen3.5: Towards native multimodal agents. find
  26. Minju Seo, Jinheon Baek, Seongyun Lee, and Sung Ju Hwang. 2025. Paper2code: Automating code generation from scientific papers in machine learning. CoRR, abs/2504.17192. find
  27. Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. 2025. Paperbench: Evaluating AI’s ability to replicate AI research. In Forty-second International C…
  28. Minyang Tian, Luyu Gao, Shizhuo Zhang, Xinan Chen, Cunwei Fan, Xuefei Guo, Roland Haas, Pan Ji, Kittithat Krongchon, Yao Li, et al. 2024. Scicode: A research coding benchmark curated by scientists. Advances in Neural Information Processing Systems, 37:30624–30650.
  29. Yanzheng Xiang, Hanqi Yan, Shuyin Ouyang, Lin Gui, and Yulan He. 2025. Scireplicate-bench: Benchmarking llms in agent-driven algorithmic reproduction from research papers. CoRR, abs/2504.00255. find
  30. Shuo Yan, Ruochen Li, Ziming Luo, Zimu Wang, Daoyang Li, Liqiang Jing, Kaiyu He, Peilin Wu, George Michalopoulos, Yue Zhang, et al. 2025. Lmr-bench: Evaluating llm agent’s ability on reproducing language modeling research. arXiv preprint arXiv:2506.17335. find
  31. Zhaojian Yu, Yilun Zhao, Arman Cohan, and Xiao-Ping Zhang. 2024. Humaneval pro and mbpp pro: Evaluating large language models on self-invoking code generation. CoRR. find
  32. Haoxuan Zhang, Ruochi Li, Yang Zhang, Ting Xiao, Jiangping Chen, Junhua Ding, and Haihua Chen. 2025. The evolving role of large language models in scientific innovation: Evaluator, collaborator, and scientist. arXiv preprint arXiv:2507.11810. — [arXiv:2507.11810](https://arxiv.org/abs/2507.11810) find
  33. Terry Yue Zhuo, Vu Minh Chien, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. 2025. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. In The Thirteenth International Conference on L…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xia-2026-mmscicode-real-world-evaluation-multilingual/