Elamur Research⌕
☰
home / papers / 2026 / SWE-QA: Can Language Models Answer Repository-level Code Questions?

SWE-QA: Can Language Models Answer Repository-level Code Questions?

Paper · 2026 · Weihan Peng, Yuling Shi, Yuhang Wang et al.

Summary

Most code question-answering benchmarks, such as CoSQA and CodeQA, test models on short, isolated snippets. Real developer questions often require moving across files, following dependencies and understanding a project's architecture. SWE-QA is a benchmark built for this repository-level setting.

The authors crawled GitHub issues from the SWE-bench repositories and used an LLM to extract developer questions. They manually coded a sample of these questions to build a two-level taxonomy. Seed templates were created for each category. Questions were then instantiated from tree-sitter-derived code subgraphs, answered by an LLM using RAG over the repository, and checked by humans. The result is 720 QA pairs drawn from 15 Python repositories.

Six LLMs were tested under several context strategies. Direct prompting performs poorly, RAG helps considerably, and agent frameworks perform best. Human evaluation largely agrees with the LLM-as-a-Judge scores. The benchmark offers a realistic way to measure repository understanding, and the generation pipeline can be reused for new codebases.

Key points

  • Source data: 77,100 issues were crawled, and 41,955 remained after keeping only those with bodies of at least 1,000 characters. From these, 127,415 questions were extracted, of which 1,000 were manually coded to form the taxonomy.
  • Taxonomy:
    • Level 1 has four interrogative types: How (35.2%), Where (28.4%), Why (23.1%) and What (13.3%).
    • Level 2 has 12 intention categories, such as dependency tracing and design rationale.
  • Benchmark composition: 48 QA pairs per repository across 15 repositories, 12 from SWE-bench and 3 from SWE-Bench-Live (conan, reflex, streamlink). It is released on GitHub and Hugging Face (SWE-QA-Benchmark).
  • Best result: OpenHands with GPT-5.1 scores 70.79.
    • Models handle open-ended How and Why questions well.
    • What questions and location-focused questions are harder, especially Architecture exploration and Data / Control-flow.
  • Per-repository differences: pylint is the hardest repository. The SWE-Bench-Live repositories average 64.98, compared with 68.59 for the SWE-bench ones. The authors suggest this may reflect less data leakage in the newer repositories.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Shanghai Jiao Tong UniversityShanghai · China149 works · 16,161 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksQuestion answering · 441 works

Cite

Cite as Weihan Peng, Yuling Shi, Yuhang Wang, Xinyun Zhang, Beijun Shen and Xiaodong Gu. 2026. SWE-QA: Can Language Models Answer Repository-level Code Questions?. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.402

@inproceedings{peng2026swe,
  title = {{SWE-QA: Can Language Models Answer Repository-level Code Questions?}},
  author = {Weihan Peng, Yuling Shi, Yuhang Wang, Xinyun Zhang, Beijun Shen and Xiaodong Gu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.402},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.402}
}

References

35 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Claude Sonnet 4.5 model. https://www.anthropic.com/news/ claude-sonnet-4-5/. ↗
  2. Jialiang Chen, Kaifa Zhao, Jie Liu, Chao Peng, Jierui Liu, Hang Zhu, Pengfei Gao, Ping Yang, and Shuiguang Deng. 2025. Coreqa: uncovering potentials of language models in code repository question answering. arXiv preprint arXiv:2501.03447. — [arXiv:2501.03447](https://arxiv.org/abs/2501.03447) find
  3. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and 1 others. 2020. Codebert: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536– 1547.
  4. Mingyang Geng, Shangwen Wang, Dezun Dong, Haotian Wang, Ge Li, Zhi Jin, Xiaoguang Mao, and Xiangke Liao. 2024. Large language models are few-shot summarizers: Multi-intent comment generation via in-context learning. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering… find
  5. Google-AI. 2025. Gemini-2.5-Pro model. https://ai.google.dev/gemini-api/docs/ models?hl=en\#gemini-2.5-pro. ↗
  6. Junda He, Jieke Shi, Terry Yue Zhuo, Christoph Treude, Jiamou Sun, Zhenchang Xing, Xiaoning Du, and David Lo. 2025. From code to courtroom: Llms as the new software judges. arXiv preprint arXiv:2503.02246. — [arXiv:2503.02246](https://arxiv.org/abs/2503.02246) find
  7. Ruida Hu, Chao Peng, Jingyi Ren, Bo Jiang, Xiangxin Meng, Qinyun Wu, Pengfei Gao, Xinchen Wang, and Cuiyun Gao. 2024. Coderepoqa: A large-scale benchmark for software engineering question answering. arXiv preprint arXiv:2412.14764. — [arXiv:2412.14764](https://arxiv.org/abs/2412.14764) find
  8. Junjie Huang, Duyu Tang, Linjun Shou, Ming Gong, Ke Xu, Daxin Jiang, Ming Zhou, and Nan Duan. 2021. Cosqa: 20, 000+ web queries for code search and question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conferen… find
  9. Siming Huang, Tianhao Cheng, Jason Klein Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J Yang, Jiaheng Liu, Chenchen Zhang, Linzheng Chai, and 1 others. 2024. Opencoder: The open cookbook for top-tier code large language models. arXiv preprint arXiv:2411.04905.
  10. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. Swe-bench: Can language models resolve real-world github issues? In ICLR.
  11. Changyoon Lee, Yeon Seonwoo, and Alice Oh. 2022. CS1QA: A dataset for assisting code-based question answering in an introductory programming course. CoRR, abs/2210.14494. find
  12. Linyi Li, Shijie Geng, Zhenwen Li, Yibo He, Hao Yu, Ziyue Hua, Guanghan Ning, Siwei Wang, Tao Xie, and Hongxia Yang. 2024a. Infibench: Evaluating the question-answering capabilities of code large language models. Advances in Neural Information Processing Systems, 37:128668–128698. find
  13. Zehan Li, Jianfei Zhang, Chuantao Yin, Yuanxin Ouyang, and Wenge Rong. 2024b. Procqa: A large-scale community-based programming question answering dataset for code search. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-… find
  14. Chenxiao Liu and Xiaojun Wan. 2021. Codeqa: A question answering dataset for source code comprehension. In Findings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 16-20 November, 2021, pages 2618–2632. Association for Computational Lingu… find
  15. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522.
  16. Moonshot-AI. 2025. kimi-k2-0905-preview model. https://platform.moonshot.ai. ↗
  17. OpenAI. 2025. gpt-5.1-2025-11-13 model. https://openai.com/index/gpt-5-1-for-developers/. ↗
  18. Siru Ouyang, Wenhao Yu, Kaixin Ma, Zilin Xiao, Zhihan Zhang, Mengzhao Jia, Jiawei Han, Hongming Zhang, and Dong Yu. 2024. Repograph: Enhancing ai software engineering with repository-level code graph. In The Thirteenth International Conference on Learning Representations. find
  19. Qwen Team. 2025a. qwen3-coder-30b-a3b-instruct https://qwenlm.github.io/blog/ model. qwen3-coder/. ↗
  20. Qwen Team. 2025b. qwen3-coder-480b-a35binstruct model. https://qwenlm.github.io/ blog/qwen3-coder/. ↗
  21. Surya Prakash Sahu, Madhurima Mandal, Shikhar Bharadwaj, Aditya Kanade, Petros Maniatis, and Shirish K. Shevade. 2024. Codequeries: A dataset of semantic queries over code. In Proceedings of the 17th Innovations in Software Engineering Conference, ISEC 2024, Bangalore, India, February 22-24, 2024, p… find
  22. Hwanjun Song, Hang Su, Igor Shalyminov, Jason Cai, and Saab Mansour. 2024. Finesure: Fine-grained summarization evaluation using llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 906–922.
  23. Jan Strich, Florian Schneider, Irina Nikishina, and Chris Biemann. 2024. On improving repository-level code qa for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 209–244. find
  24. VoyageAI. 2024. voyage-code-3: more accurate code retrieval with lower dimensional, quantized embeddings. https://blog.voyageai.com/2024/12/ 04/voyage-code-3. ↗
  25. Ruiqi Wang, Jiyu Guo, Cuiyun Gao, Guodong Fan, Chun Yong Chong, and Xin Xia. 2025a. Can llms replace human evaluators? an empirical study of llm-as-a-judge in software engineering. Proceedings of the ACM on Software Engineering, 2(ISSTA):1955– 1977. find
  26. Xingyao Wang, Boxuan Li, Yufan Song, Frank F Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, and 1 others. 2024a. Openhands: An open platform for ai software developers as generalist agents. arXiv preprint arXiv:2407.16741.
  27. Yanlin Wang, Yanli Wang, Daya Guo, Jiachi Chen, Ruikai Zhang, Yuchi Ma, and Zibin Zheng. 2024b. Rlcoder: Reinforcement learning for repository-level code completion. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 165–177. IEEE Computer Society. find
  28. Yifei Wang, Feng Xiong, Yong Wang, Linjing Li, Xiangxiang Chu, and Daniel Dajun Zeng. 2025b. Position bias mitigates position bias: Mitigate position bias through inter-position knowledge distillation. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 1…
  29. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8696–8708. find
  30. John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024. Swe-agent: Agent-computer interfaces enable automated software engineering. Advances in Neural Information Processing Systems, 37:50528– 50652.
  31. Xinwei Yang, Zhaofeng Liu, Chen Huang, Jiashuai Zhang, Tong Zhang, Yifan Zhang, and Wenqiang Lei. 2025. Elaboration: A comprehensive benchmark on human-llm competitive programming. arXiv preprint arXiv:2505.16667. find
  32. Fengji Zhang, Bei Chen, Yue Zhang, Jacky Keung, Jin Liu, Daoguang Zan, Yi Mao, Jian-Guang Lou, and Weizhu Chen. 2023. Repocoder: Repository-level code completion through iterative retrieval and generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMN… find
  33. Linghao Zhang, Shilin He, Chaoyun Zhang, Yu Kang, Bowen Li, Chengxing Xie, Junhao Wang, Maoquan Wang, Yufan Huang, Shengyu Fu, and 1 others. 2025. Swe-bench goes live! arXiv preprint arXiv:2505.23419. — [arXiv:2505.23419](https://arxiv.org/abs/2505.23419) find
  34. Zhipu AI (Z.ai). 2025. glm-4.6 model. https://docs.bigmodel.cn/cn/guide/models/text/glm-4.6. ↗
  35. Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma, and 1 others. 2024. Deepseek-coder-v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXiv:2406.11931.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2026-swe-qa-can-language-models/