Elamur Research⌕
☰
home / papers / 2026 / AlphaQT-Bench: Diagnosing the Gap between Financial Code Generation and Quantitative Reasoning in LLMs

AlphaQT-Bench: Diagnosing the Gap between Financial Code Generation and Quantitative Reasoning in LLMs

Paper · 2026 · Sichun Luo, Yi Huang, Shichang Meng et al.

Summary

LLMs increasingly write executable Python code for alpha mining rather than symbolic factor formulas. This adds expressiveness but also hidden risks: look-ahead bias (using future data) and bugs in stateful logic. Existing alpha benchmarks mostly score backtest profitability, which can reward leaked information or punish sound code because of market noise. The authors propose AlphaQT-Bench, a diagnostic benchmark of instruction-grounded tasks. In each task, a model turns a natural-language specification into a function that maps OHLCV data to a time-aligned signal series. Tasks span three difficulty levels. Evaluation has four layers:

  • whether the code runs;
  • causality safety, checked by a dynamic truncation test;
  • numerical agreement with expert-written Golden Code;
  • structural compliance, meaning loop-free vectorized implementation.

Across 12 LLMs, many models produce runnable code yet fail causal, functional or structural checks, and the gap widens as tasks get harder. The authors conclude that models act more like approximate retrievers of indicator templates than true reasoners. The benchmark matters because it audits process reliability rather than profit, which is safer for LLM-based quant pipelines.

Key points

  • Four evaluation layers: Executability, Causality Safety (dynamic truncation), Functional Accuracy versus Golden Code, and Structural Compliance (vectorization analysis); 12 LLMs were tested.
  • Semantic misalignment in stateful logic (e.g., RSI, Money Flow Index, Wilder's Smoothing) causes almost 60% of Level 3 errors.
  • Observed failure modes include:
    • Safety over-alignment: Llama-3.1-70B adds a needless shift "to avoid look-ahead bias".
    • Complexity trap: GPT-5-Mini builds a manual cumulative-sum OLS that yields NaNs.
    • ewm misconfiguration: wrong adjust settings in ewm.
  • One failure pattern appeared in about 15% of failure cases for Llama-3.1-70B and DeepSeek-V3, mostly in the Risk and Reversion categories.
  • Limitations: Python only, single-turn generation, and possible rejection of valid alternative implementations when financial definitions are ambiguous.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Hong KongPok Fu Lam · China89 works · 11,774 citations🎓City University of Hong KongHong Kong · China45 works · 3,014 citations🏢China MobileBeijing · China9 works · 21 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksDecision Sciences · 209 worksManagement Science and Operations Research · 108 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksCode & programming languages · 129 works

Cite

Cite as Sichun Luo, Yi Huang, Shichang Meng, Fengyuan Liu, Mukai Li, Qinghua Yao, Zefa Hu, Junlan Feng and Qi Liu. 2026. AlphaQT-Bench: Diagnosing the Gap between Financial Code Generation and Quantitative Reasoning in LLMs. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.138

@inproceedings{luo2026alphaqt,
  title = {{AlphaQT-Bench: Diagnosing the Gap between Financial Code Generation and Quantitative Reasoning in LLMs}},
  author = {Sichun Luo, Yi Huang, Shichang Meng, Fengyuan Liu, Mukai Li, Qinghua Yao, Zefa Hu, Junlan Feng and Qi Liu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.138},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.138}
}

References

31 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Claude haiku 4.5 system card. find
  2. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program synthesis with large language models.
  3. David H Bailey and Marcos López de Prado. 2014. The deflated sharpe ratio: Correcting for selection bias, backtest overfitting and non-normality. Journal of Portfolio Management, 40(5):94–107. find
  4. Lang Cao, Zekun Xi, Long Liao, Ziwei Yang, and Zheng Cao. 2025. Chain-of-alpha: Unleashing the power of large language models for alpha mining in quantitative trading. arXiv preprint arXiv:2508.06312. — [arXiv:2508.06312](https://arxiv.org/abs/2508.06312) find
  5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  6. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv prep…
  7. Marcos Lopez De Prado. 2018. Advances in financial machine learning. John Wiley & Sons. find
  8. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, H… find
  9. Ren, Zehui Ren, Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang, Zhenda Xie, Zhengyan Zhang, Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Ziyi Gao, and Zizheng… find
  10. Hongjun Ding, Binqi Chen, Jinsheng Huang, Taian Guo, Zhengyang Mao, Guoyi Shao, Lutong Zou, Luchen Liu, and Ming Zhang. 2025. Alphaeval: A comprehensive and efficient evaluation framework for formula alpha mining. arXiv preprint arXiv:2508.13174. — [arXiv:2508.13174](https://arxiv.org/abs/2508.13174) find
  11. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  12. Eugene F Fama and Kenneth R French. 1993. Common risk factors in the returns on stocks and bonds. Journal of financial economics, 33(1):3–56. find
  13. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, et al. 2025. Deepseek-r1 incentivizes reasoning in llms through reinforcement learning. Nature, 645(8081):633–638. find
  14. Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. 2024. A survey on large language models for code generation. arXiv preprint arXiv:2406.00515.
  15. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. find
  16. Yuante Li, Xu Yang, Xiao Yang, Minrui Xu, Xisen Wang, Weiqing Liu, and Jiang Bian. 2025. R&d-agent-quant: A multi-agent framework for data-centric factors and model joint optimization. arXiv preprint arXiv:2505.15155. — [arXiv:2505.15155](https://arxiv.org/abs/2505.15155) find
  17. Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Masson d’Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James M… — Competition-Level Code Generation with AlphaCode
  18. Xiaoming Lin, Ye Chen, Ziyu Li, and Kang He. 2019. Stock alpha mining based on genetic algorithm. Technical Report, Huatai Securities Research Center. find
  19. Fengyuan Liu, Huang Yi, Sichun Luo, Yuqi Wang, Yazheng Yang, Xinye Li, Zefa Hu, Junlan Feng, and Qi Liu. 2025. Cognitive alpha mining via llm-driven code-based evolution. arXiv preprint arXiv:2511.18850. find
  20. Haochen Luo, Ho Tin Ko, Jiandong Chen, David Sun, Yuan Zhang, and Chen Liu. 2026. Alphabench: Benchmarking large language models in formulaic alpha factor mining. In The Fourteenth International Conference on Learning Representations. find
  21. OpenAI, :, Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, Yu Bai, Bowen Baker, Haiming Bao, Boaz Barak, Ally Bennett, Tyler Bertao, Nivedita Brett, Eugene Brevdo, Greg Brockman, Sebastien Bubeck, Che Chang, Kai Chen, Mark Chen, Enoch Cheung, Aidan Cl… — gpt-oss-120b & gpt-oss-20b Model Card
  22. OpenAI. 2025. Introducing gpt-5.2. find
  23. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  24. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng… find
  25. Weizhe Ren, Yichen Qin, and Yang Li. 2024. Alpha mining and enhancing via warm start genetic programming for quantitative investment. arXiv preprint arXiv:2412.00896. — [arXiv:2412.00896](https://arxiv.org/abs/2412.00896) find
  26. Hao Shi, Weili Song, Xinting Zhang, Jiahe Shi, Cuicui Luo, Xiang Ao, Hamid Arian, and Luis Angel Seco. 2025a. Alphaforge: A framework to mine and dynamically combine formulaic alpha factors. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 12524–12532. find
  27. Yu Shi, Yitong Duan, and Jian Li. 2025b. Navigating the alpha jungle: An llm-powered mcts framework for formulaic factor mining. arXiv preprint arXiv:2505.11122. — [arXiv:2505.11122](https://arxiv.org/abs/2505.11122) find
  28. Zhaofan Su, Jianwu Lin, and Chengshan Zhang. 2022. Genetic algorithm based quantitative factors construction. In 2022 IEEE 20th International Conference on Industrial Informatics (INDIN), pages 650–655. IEEE. find
  29. Ziyi Tang, Zechuan Chen, Jiarui Yang, Jiayao Mai, Yongsen Zheng, Keze Wang, Jinrui Chen, and Liang Lin. 2025. Alphaagent: Llm-driven alpha mining with regularized exploration to counteract alpha decay. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, page… find
  30. Shuo Yu, Hongyan Xue, Xiang Ao, Feiyang Pan, Jia He, Dandan Tu, and Qing He. 2023. Generating synergistic formulaic alpha collections via reinforcement learning. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5476–5486. find
  31. Junjie Zhang, Shuoling Liu, Tongzhe Zhang, and Yuchen Shi. 2025. A survey on large language model-based alpha mining. Frontiers of Information Technology & Electronic Engineering, 26(10):1809– 1821. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/luo-2026-alphaqt-bench-diagnosing-gap-between/