Elamur Research⌕
☰
home / papers / 2026 / From Conversation to Evaluation: Benchmarking LLMs on Development Knowledge via SimpleDevQA

From Conversation to Evaluation: Benchmarking LLMs on Development Knowledge via SimpleDevQA

Paper · 2026 · Jing Zhang, Lianghong Guo, Yanlin Wang et al.

Summary

Developers often turn to LLMs for factual knowledge about tools, APIs and frameworks, but benchmarks for this Dev Knowledge QA task are narrow in the knowledge they cover and seldom rest on genuine user questions. The authors first analyze real user–LLM conversations from WildChat. They find that knowledge-seeking questions are a major part of how people use LLMs for software work, but the raw conversations are too noisy to serve as a benchmark directly. They therefore build a three-phase pipeline that turns these conversations into compact factual question–answer pairs, with generation and verification steps. The result is SimpleDevQA, a multilingual benchmark in English, Chinese and Russian. Each question has one brief, checkable answer, which keeps grading simple and reliable. In tests of 18 popular LLMs, closed-source systems generally score highest. Adding retrieved knowledge through RAG raises accuracy. Scores also track model confidence and code-generation ability. The benchmark grounds evaluation of developer-facing assistants in realistic queries.

Key points

  • Source data: real user–LLM dialogues from WildChat, converted into QA pairs by a three-phase pipeline (e.g., a Spring Boot question whose answer is JdbcTemplate).
  • SimpleDevQA covers three languages: English, Chinese and Russian. Answers are unique, short and verifiable.
  • 18 mainstream LLMs were evaluated, and closed-source models generally led.
  • RAG-based knowledge injection improved accuracy. Performance correlated with model confidence and with code-generation capability.
  • Data and code are released at github.com/DeepSoftwareAnalytics/SimpleDevQA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations🏢Huawei TechnologiesShenzhen · China96 works · 15,432 citations🎓Monash UniversityMelbourne · Australia64 works · 54,971 citations🎓Sun Yat-sen UniversityGuangzhou · China72 works · 9,502 citations🎓Chongqing UniversityChongqing · China14 works · 318 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Jing Zhang, Lianghong Guo, Yanlin Wang, Terry Yue Zhuo, Yong Wang, Mingwei Liu, Jiachi Chen, Ensheng Shi, Yuchi Ma, Hongyu Zhang and Zibin Zheng. 2026. From Conversation to Evaluation: Benchmarking LLMs on Development Knowledge via SimpleDevQA. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1877

@inproceedings{zhang2026conversation,
  title = {{From Conversation to Evaluation: Benchmarking LLMs on Development Knowledge via SimpleDevQA}},
  author = {Jing Zhang, Lianghong Guo, Yanlin Wang, Terry Yue Zhuo, Yong Wang, Mingwei Liu, Jiachi Chen, Ensheng Shi, Yuchi Ma, Hongyu Zhang and Zibin Zheng},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1877},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1877}
}

References

71 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2023. Confidence interval. Wikipedia, The Free Encyclopedia. [Online; accessed May 31, 2025]. 2023. Sample size calculator. https://www.surveysystem.com/sscalc.htm. Accessed: 2025-05-31. ↗
  2. Anthropic. 2024. Claude Haiku. Accessed: 2025-05-28. find
  3. BigModel Team. 2024. GLM-4 Usage Guide. Accessed: 2025-05-28. find
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  5. Vittorio Castelli, Rishav Chakravarti, Saswati Dana, Anthony Ferritto, Radu Florian, Martin Franz, Dinesh Garg, Dinesh Khandelwal, Scott McCarley, Mike McCawley, Mohamed Nasr, Lin Pan, Cezar Pendus, John Pitrelli, Saurabh Pujar, Salim Roukos, Andrzej Sakrajda, Avirup Sil, Rosario Uceda-Sosa, and 2 o… find
  6. Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or llms as the judge? a study on judgement biases. arXiv preprint arXiv:2402.10669.
  7. Jialiang Chen, Kaifa Zhao, Jie Liu, Chao Peng, Jierui Liu, Hang Zhu, Pengfei Gao, Ping Yang, and Shuiguang Deng. 2025. Coreqa: Uncovering potentials of language models in code repository question answering. Preprint, arXiv:2501.03447. — [arXiv:2501.03447](https://arxiv.org/abs/2501.03447) find
  8. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, and 1 others. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  9. I Chern, Steffi Chern, Shiqi Chen, Weizhe Yuan, Kehua Feng, Chunting Zhou, Junxian He, Graham Neubig, Pengfei Liu, and 1 others. 2023. Factool: Factuality detection in generative ai–a tool augmented framework for multi-task and multi-domain scenarios. arXiv preprint arXiv:2307.13528.
  10. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  11. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, and 1 others. 2024. A survey on llm-as-a-judge. arXiv preprint arXiv:2411.15594. find
  12. Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, and 1 others. 2024. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339.
  13. Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Yu Wu, YK Li, and 1 others. 2024. Deepseek-coder: When the large language model meets programming–the rise of code intelligence. arXiv preprint arXiv:2401.14196. find
  14. Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. 2025. From code to courtroom: Llms as the new software judges. arXiv preprint arXiv:2503.02246. — [arXiv:2503.02246](https://arxiv.org/abs/2503.02246) find
  15. Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, and 1 others. 2024. Chinese simpleqa: A chinese factuality evaluation for large language models. arXiv preprint arXiv:2411.07140.
  16. José Wellington Franco da Silva, Amanda Drielly Pires Venceslau, Juliano Efson Sales, José Gilvan Rodrigues Maia, Vládia Célia Monteiro Pinheiro, and Vânia Maria Ponte Vidal. 2020. A short survey on end-to-end simple question answering systems. Artificial Intelligence Review, 53(7):5429–5453. find
  17. Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large language models for software engineering: A systematic literature review. ACM Transactions on Software Engineering and Methodology, 33(8):1–79. find
  18. DeepSeek-AI, Aixin Liu, Aoxue Mei, Bangcai Lin, Bing Xue, Bingxuan Wang, Bingzheng Xu, Bochao Wu, Bowei Zhang, Chaofan Lin, Chen Dong, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenhao Xu, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, and 245 others. 2025. Deepseek-v3.2: Pushing the frontier of open…
  19. Ruida Hu, Chao Peng, Jingyi Ren, Bo Jiang, Xiangxin Meng, Qinyun Wu, Pengfei Gao, Xinchen Wang, and Cuiyun Gao. 2024. Coderepoqa: A large-scale benchmark for software engineering question answering. arXiv preprint arXiv:2412.14764. — [arXiv:2412.14764](https://arxiv.org/abs/2412.14764) find
  20. Bhuwan Dhingra, Kathryn Mazaitis, and William W Cohen. 2017. Quasar: Datasets for question answering by search and reading. arXiv preprint arXiv:1707.03904. find
  21. Google Developers. 2024. Custom Search API Documentation. Accessed: 2025-05-28. find
  22. Goose3 Contributors. 2024. Goose3: A Python Library for Web Content Extraction. Accessed: 2025-05-28. find
  23. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Yao Fu, and 1 others. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. Advances in Neural Information Processing Systems, 36:62991–63010…
  24. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. — Qwen2.5-Coder Technical Report
  25. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. find
  26. Changyoon Lee, Yeon Seonwoo, and Alice Oh. 2022. Cs1qa: A dataset for assisting code-based question answering in an introductory programming course. arXiv preprint arXiv:2210.14494. find
  27. Krystal M Lewis and Peter Hepburn. 2010. Open card sorting and factor analysis: a usability case study. The electronic library, 28(3):401–416. find
  28. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information processing systems, 33:9459–…
  29. Changmao Li and Jeffrey Flanigan. 2024. Rac: Efficient llm factuality correction with retrieval augmentation. arXiv preprint arXiv:2410.15667. find
  30. Haitao Li, Qian Dong, Junjie Chen, Huixue Su, Yujia Zhou, Qingyao Ai, Ziyi Ye, and Yiqun Liu. 2024a. Llms-as-judges: a comprehensive survey on llm-based evaluation methods. arXiv preprint arXiv:2412.05579.
  31. Jiarui Li, Ye Yuan, and Zehua Zhang. 2024b. Enhancing llm factual accuracy with rag to counter hallucinations: A case study on domain-specific queries in private knowledge-bases. arXiv preprint arXiv:2403.10446. — [arXiv:2403.10446](https://arxiv.org/abs/2403.10446) find
  32. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. Halueval: A large-scale hallucination evaluation benchmark for large language models. arXiv preprint arXiv:2305.11747.
  33. Linyi Li, Shijie Geng, Zhenwen Li, Yibo He, Hao Yu, Ziyue Hua, Guanghan Ning, Siwei Wang, Tao Xie, and Hongxia Yang. 2024c. Infibench: Evaluating the question-answering capabilities of code large language models. In Advances in Neural Information Processing Systems, volume 37, pages 128668– 128698.… find
  34. Linyi Li, Shijie Geng, Zhenwen Li, Yibo He, Hao Yu, Ziyue Hua, Guanghan Ning, Siwei Wang, Tao Xie, and Hongxia Yang. 2024d. Infibench: Evaluating the question-answering capabilities of code large language models. Advances in Neural Information Processing Systems, 37:128668–128698. find
  35. Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958.
  36. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  37. Chenxiao Liu and Xiaojun Wan. 2021. Codeqa: A question answering dataset for source code comprehension. arXiv preprint arXiv:2109.08365. find
  38. Jerry Liu. 2022. LlamaIndex. find
  39. Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang. 2024b. Repoqa: Evaluating long context code understanding. arXiv preprint arXiv:2406.06025. — [arXiv:2406.06025](https://arxiv.org/abs/2406.06025) find
  40. Dung Nguyen Manh, Thang Phan Chau, Nam Le Hai, Thong T Doan, Nam V Nguyen, Quang Pham, and Nghi DQ Bui. 2024. Codemmlu: A multi-task benchmark for assessing code understanding capabilities of codellms. arXiv preprint arXiv:2410.01999. — [arXiv:2410.01999](https://arxiv.org/abs/2410.01999) find
  41. Mary L Mchugh. 2012. Interrater reliability: the kappa statistic. Biochemia Medica, 22(3):276–282. find
  42. Meta AI. 2024. Meta llama-3-8b-instruct. Accessed: 2025-05-31. find
  43. Changan Niu, Chuanyi Li, Bin Luo, and Vincent Ng. 2022. Deep learning meets software engineering: A survey on pre-trained models of source code. arXiv preprint arXiv:2205.11739. — [arXiv:2205.11739](https://arxiv.org/abs/2205.11739) find
  44. OpenAI. 2024a. GPT-4o-mini. Accessed: 2025-05-29. find
  45. OpenAI. 2024b. Hello GPT-4o. Accessed: 2025-05-28. find
  46. OpenAI. 2024c. OpenAI o3 Mini. Accessed: 2025-05-28. find
  47. Jan Piasecki, Marcin Waligora, and Vilius Dranseika. 2018. Google search as an additional source in systematic reviews. Science and engineering ethics, 24:809–810. find
  48. Team Qwen. Qwen2. 5: A party of foundation models, september 2024. URL https://qwenlm.github. io/blog/qwen2, 5. ↗
  49. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, and 1 others. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.
  50. Ensheng Shi, Yanlin Wang, Fengji Zhang, Bei Chen, Hongyu Zhang, Yanli Wang, Daya Guo, Lun Du, Shi Han, Dongmei Zhang, and Hongbin Sun. 2025. Sotana: An open-source software engineering instruction-tuned model. In 2025 IEEE/ACM Second International Conference on AI Foundation Models and Software Engi… find
  51. Guijin Son, Hyunwoo Ko, Hoyoung Lee, Yewon Kim, and Seunghyeok Hong. 2024. Llm-as-a-judge & reward model: What they can and cannot do. arXiv preprint arXiv:2409.11239.
  52. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, and 1 others. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.…
  53. I Team. 2024. Internlm2 technical report. preprint arXiv:2403.17297. arXiv valpy. 2024. Prompt classification model. Accessed: 2025-05-28.
  54. Zhiruo Wang, Shuyan Zhou, Daniel Fried, and Graham Neubig. 2022. Execution-based evaluation for open-domain code generation. arXiv preprint arXiv:2212.10481.
  55. Jason Wei, Nguyen Karina, Hyung Won Chung, Yunxin Joy Jiao, Spencer Papay, Amelia Glaese, John Schulman, and William Fedus. 2024. Measuring short-form factuality in large language models. arXiv preprint arXiv:2411.04368.
  56. Philip Welsby and Bernard MY Cheung. 2023. Chatgpt. find
  57. Di Wu, Xiao-Yuan Jing, Hongyu Zhang, Yang Feng, Haowen Chen, Yuming Zhou, and Baowen Xu. 2023. Retrieving api knowledge from tutorials and stack overflow based on natural language queries. ACM Transactions on Software Engineering and Methodology, 32(5):1–36. find
  58. Peng Xia, Kangyu Zhu, Haoran Li, Hongtu Zhu, Yun Li, Gang Li, Linjun Zhang, and Huaxiu Yao. 2024. Rule: Reliable multimodal rag for factuality in medical vision language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 1081–1093. find
  59. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025a. Qwen3 technical report. Preprint, arXiv:2505.09388.
  60. Shuo Yang, Jiachi Chen, and Zibin Zheng. 2023. Definition and detection of defects in nft smart contracts. In Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis, pages 373–384. find
  61. Shuo Yang, Xingwei Lin, Jiachi Chen, Qingyuan Zhong, Lei Xiao, Renke Huang, Yanlin Wang, and Zibin Zheng. 2025b. Hyperion: Unveiling dapp inconsistencies using llm and dataflow-guided symbolic execution. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 2125–2137. find
  62. Ziyu Yao, Daniel S Weld, Wei-Peng Chen, and Huan Sun. 2018. Staqc: A systematically mined question-code dataset from stack overflow. In Proceedings of the 2018 World Wide Web Conference, pages 1693– 1703. find
  63. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, and 1 others. 2024. Justice or prejudice? quantifying biases in llm-as-a-judge. arXiv preprint arXiv:2410.02736.
  64. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, and 1 others. 2025a. Siren’s song in the ai ocean: A survey on hallucination in large language models. Computational Linguistics, pages 1–46.
  65. Ziyao Zhang, Chong Wang, Yanlin Wang, Ensheng Shi, Yuchi Ma, Wanjun Zhong, Jiachi Chen, Mingzhi Mao, and Zibin Zheng. 2025b. Llm hallucinations in practical code generation: Phenomena, mechanism, and mitigation. Proceedings of the ACM on Software Engineering, 2(ISSTA):481–503. find
  66. Ziyin Zhang, Chaoyu Chen, Bingchang Liu, Cong Liao, Zi Gong, Hang Yu, Jianguo Li, and Rui Wang. 2023. Unifying the perspectives of nlp and software engineering: A survey on language models for code. arXiv preprint arXiv:2311.07989. — [arXiv:2311.07989](https://arxiv.org/abs/2311.07989) find
  67. Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. 2024. Wildchat: 1m chatgpt interaction logs in the wild. arXiv preprint arXiv:2405.01470.
  68. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.
  69. Zibin Zheng, Kaiwen Ning, Qingyuan Zhong, Jiachi Chen, Wenqing Chen, Lianghong Guo, Weicheng Wang, and Yanlin Wang. 2025. Towards an understanding of large language models in software engineering tasks. Empirical Software Engineering, 30(2):50. find
  70. Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. 2024. Agieval: A human-centric benchmark for evaluating foundation models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2299–2314, Mexico City, Mexico.…
  71. Terry Yue Zhuo. 2024. Ice-score: Instructing large language models to evaluate code. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2232–2242. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-conversation-evaluation-benchmarking-llms-development/