Elamur Research⌕
☰
home / papers / 2026 / Program Structure-aware Language Models: Targeted Software Testing beyond Textual Semantics

Program Structure-aware Language Models: Targeted Software Testing beyond Textual Semantics

Paper · 2026 · Khang Tran, Khoa Nguyen, Cristian Borcea et al.

Summary

LLM-based test generators usually expand coverage through prompt-engineered mutations. These methods cannot reliably steer generation toward a particular execution branch, so risky code paths that may hide bugs or vulnerabilities often go unexercised. The authors present GLMTest, a framework that conditions test generation on a developer-chosen target branch.

GLMTest extracts a code property graph (CPG) from the program. It marks the nodes belonging to the target branch with a binary mask and encodes the graph with a heterogeneous GNN. The resulting node-level structural embeddings are concatenated with the embedded text instruction and passed to an LLM. The GNN and the LLM are trained jointly, end to end. Training uses a new dataset built from real-world repositories, pairing human-written tests with the branches they execute.

On Python programs from TestGenEval, GLMTest on Qwen2.5-Coder-7B-Instruct reaches 50.2% branch accuracy, compared with 27.4% for state-of-the-art commercial LLMs. Coverage stays competitive. The approach makes targeted, interpretable testing of critical paths practical, including security-oriented testing and seed generation for fuzzers.

Key points

  • Comparison: GLMTest's branch accuracy (50.2%) is contrasted with the 27.4% reported for state-of-the-art commercial LLMs; Claude-Sonnet-4.5 and GPT-4o-mini are named as the commercial baselines.
  • Ablations on branch accuracy:
    • The default GAT encoder scores 0.502.
    • GraphSAGE scores 0.465.
    • Fine-tuning without a GNN scores 0.442.
    • Mean-pooling branch nodes into a single vector instead of keeping node-level embeddings scores 0.399.
  • Model size: a Qwen2.5-Coder-14B-Instruct backbone scores 49.3% BranchAcc and 17.63% BranchCov, compared with 50.2% and 16.91% for the 7B model. Scaling mainly broadens coverage rather than improving targeting.
  • Training cost: LoRA (rank 8) plus DeepSpeed on 4× A100 GPUs, about 192 GPU-hours, roughly 5.37 × 10⁸ tokens and about $119.56 on Vast.ai. The code and dataset are released on GitHub (khangtran2020/glmtest).
  • Limitations:
    • The model is trained on a small set of TestGenEval projects, and cross-project generalization is weak.
    • The static CPG may contain unreachable branches.
    • Per-branch inference becomes costly on very large systems.
    • CPGs are extracted with Joern, which supports other languages such as C/C++, so extension beyond Python is plausible.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓New Jersey Institute of TechnologyNewark · United States15 works · 1,333 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksInformation Systems · 1,437 worksSoftware · 92 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Khang Tran, Khoa Nguyen, Cristian Borcea and Hai Phan. 2026. Program Structure-aware Language Models: Targeted Software Testing beyond Textual Semantics. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.540

@inproceedings{tran2026program,
  title = {{Program Structure-aware Language Models: Targeted Software Testing beyond Textual Semantics}},
  author = {Khang Tran, Khoa Nguyen, Cristian Borcea and Hai Phan},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.540},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.540}
}

References

46 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Saranya Alagarsamy, Chakkrit Tantithamthavorn, and Aldeida Aleti. 2024. A3test: Assertion-augmented automated test case generation. Information and Software Technology, 176:107565. find
  2. Paul Ammann and Jeff Offutt. 2008. Introduction to Software Testing. Cambridge University Press. find
  3. Merve Astekin, Max Hort, and Leon Moonen. 2024. An exploratory study on how non-determinism in large language models affects log parsing. In Proceedings of the ACM/IEEE 2nd International Workshop on Interpretability, Robustness, and Benchmarking in Neural Software Engineering, pages 13–18. find
  4. Mohammad Baqar and Rajat Khanda. 2025. The future of software testing: Ai–powered test case generation and validation. In Intelligent Computing-Proceedings of the Computing Conference, pages 276–300. Springer. find
  5. Dhaya Sindhu Battina. 2019. Artificial intelligence in software test automation: A systematic literature review. International Journal of Emerging Technologies and Innovative Research (www. jetir. org\| UGC and issn Approved), ISSN, pages 2349–5162. find
  6. Tristan Bilot, Nour El Madhoun, Khaldoun Al Agha, and Anis Zouaoui. 2024. A survey on malware detection with graph representation learning. ACM Comput. Surv., 56(11). find
  7. Matteo Brunetto, Giovanni Denaro, Leonardo Mariani, and Mauro Pezzè. 2021. On introducing automatic test case generation in practice: A success story and lessons learned. Journal of Systems and Software, 176:110933. find
  8. Yinghao Chen, Zehao Hu, Chen Zhi, Junxiao Han, Shuiguang Deng, and Jianwei Yin. 2024. Chatunitest: A framework for llm-based test generation. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, FSE 2024, page 572–576, New York, NY, USA. Assoc… find
  9. Zeqi Chen, Zhaoyang Chu, Yi Gui, Feng Guo, Yao Wan, and Chuan Shi. 2025. Bridging code graphs and large language models for better code understanding. arXiv preprint arXiv:2512.07666. — [arXiv:2512.07666](https://arxiv.org/abs/2512.07666) find
  10. Arghavan Moradi Dakhel, Amin Nikanjam, Vahid Majdinasab, Foutse Khomh, and Michel C. Desmarais. 2024. Effective test generation using pre-trained large language models and mutation testing. Information and Software Technology, 171:107468. find
  11. Xiaotao Feng, Xiaogang Zhu, Kun Hu, Jincheng Wang, Yingjie Cao, Guang Gong, and Jianfeng Pan. 2025. Fuzzing: Randomness? reasoning! efficient directed fuzzing via large language models. arXiv preprint arXiv:2507.22065. — [arXiv:2507.22065](https://arxiv.org/abs/2507.22065) find
  12. Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. Advances in neural information processing systems, 30.
  13. Mark Harman, Jillian Ritchey, Inna Harper, Shubho Sengupta, Ke Mao, Abhishek Gulati, Christopher Foster, and Hervé Robert. 2025. Mutation-guided llm-based test generation at meta. In Proceedings of the 33rd ACM International Conference on the Foundations of Software Engineering, pages 180–191. find
  14. Linghan Huang, Peizhou Zhao, Lei Ma, and Huaming Chen. 2025. On the challenges of fuzzing techniques via large language models. In 2025 IEEE International Conference on Software Services Engineering (SSE), pages 162–171. IEEE. find
  15. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, and 1 others. 2024. Qwen2. 5-coder technical report. arXiv preprint arXiv:2409.12186. — Qwen2.5-Coder Technical Report
  16. Kush Jain, Gabriel Synnaeve, and Baptiste Roziere. Testgeneval: A real world unit test generation and test completion benchmark. In The Thirteenth International Conference on Learning Representations. find
  17. Wuxia Jin, Shuo Xu, Dawei Chen, Jiajun He, Dinghong Zhong, Ming Fan, Hongxu Chen, Huijia Zhang, and Ting Liu. 2024. Pyanalyzer: An effective and practical approach for dependency extraction from python code. In Proceedings of the IEEE/ACM 46th International Conference on Software Engineering, pages… find
  18. KPMG. 2024. Software testing: Market and insights report 2024. https://assets.kpmg.com/content/dam/kpmgsites/uk/pdf/2024/08/ software-testing-market-and-insights-report. pdf. Accessed: 2025-12-01. ↗
  19. Ahmed Lekssays, Hamza Mouhcine, Khang Tran, Ting Yu, and Issa Khalil. 2025. {LLMxCPG}:{Context-Aware} vulnerability detection through code property {Graph-Guided} large language models. In 34th USENIX Security Symposium (USENIX Security 25), pages 489–507. find
  20. Caroline Lemieux, Jeevana Priya Inala, Shuvendu K. Lahiri, and Siddhartha Sen. 2023. Codamosa: Escaping coverage plateaus in test generation with pre-trained large language models. In 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE), pages 919–931. find
  21. Youpeng Li, Fuxun Yu, and Xinda Wang. 2025. Vulpo: Context-aware vulnerability detection via on-policy llm optimization. arXiv preprint arXiv:2511.11896. — [arXiv:2511.11896](https://arxiv.org/abs/2511.11896) find
  22. Hongliang Liang, Xiaoxiao Pei, Xiaodong Jia, Wuwei Shen, and Jian Zhang. 2018. Fuzzing: State of the art. IEEE Transactions on Reliability, 67(3):1199–1218. find
  23. Ruitong Liu, Yanbin Wang, Haitao Xu, Jianguo Sun, Fan Zhang, Peiyue Li, and Zhenhao Guo. 2025a. Vul-lmgnns: Fusing language models and online-distilled graph neural networks for code vulnerability detection. Information Fusion, 115:102748. find
  24. Xiangyan Liu, Bo Lan, Zhiyuan Hu, Yang Liu, Zhicheng Zhang, Fei Wang, Michael Qizhe Shieh, and Wenmeng Zhou. 2025b. Codexgraph: Bridging large language models and code repositories via code graph databases. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associati… find
  25. Flemming Nielson, Hanne R. Nielson, and Chris Hankin. 2010. Principles of Program Analysis. Springer Publishing Company, Incorporated. find
  26. Rangeet Pan, Myeongsoo Kim, Rahul Krishna, Raju Pavuluri, and Saurabh Sinha. 2025. Aster: Natural and multi-language unit test generation with llms. In 2025 IEEE/ACM 47th International Conference on Software Engineering: Software Engineering in Practice (ICSE-SEIP), pages 413–424. IEEE. find
  27. Md Rizwan Parvez, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2018. Building language models for text with named entities. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2373–2383, Melbourne, Australia. Association… find
  28. Rajvardhan Patil and Venkat Gudivada. 2024. A review of current trends, techniques, and challenges in large language models (llms). Applied Sciences, 14(5):2074. find
  29. Nikitha Rao, Kush Jain, Uri Alon, Claire Le Goues, and Vincent J. Hellendoorn. 2024. Cat-lm training language models on aligned code and tests. In Proceedings of the 38th IEEE/ACM International Conference on Automated Software Engineering, ASE ’23, page 409–420. IEEE Press. find
  30. Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, and 1 others. 2023. Code llama: Open foundation models for code. arXiv preprint arXiv:2308.12950.
  31. Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. 2024a. Code-aware prompting: A study of coverage-guided test generation in regression setting using llm. Proc. ACM Softw. Eng., 1(FSE). find
  32. Gabriel Ryan, Siddhartha Jain, Mingyue Shang, Shiqi Wang, Xiaofei Ma, Murali Krishna Ramanathan, and Baishakhi Ray. 2024b. Code-aware prompting: A study of coverage-guided test generation in regression setting using llm. Proceedings of the ACM on Software Engineering, 1(FSE):951–971. find
  33. Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolutional networks. In European semantic web conference, pages 593–607. Springer. find
  34. Max Schäfer, Sarah Nadi, Aryaz Eghbali, and Frank Tip. 2024. An empirical evaluation of using large language models for automated unit test generation. IEEE Transactions on Software Engineering, 50(1):85–105. find
  35. Mohammed Latif Siddiq, Joanna Cecilia Da Silva Santos, Ridwanul Hasan Tanvir, Noshin Ulfat, Fahmid Al Rifat, and Vinícius Carvalho Lopes. 2024. Using large language models to generate junit tests: An empirical study. In Proceedings of the 28th International Conference on Evaluation and Assessment in… find
  36. Dokyung Song, Julian Lettner, Prabhu Rajasekaran, Yeoul Na, Stijn Volckaert, Per Larsen, and Michael Franz. 2019. Sok: Sanitizing for security. In 2019 IEEE Symposium on Security and Privacy (SP), pages 1275–1295. IEEE. find
  37. Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. 2025. The good, the bad, and the greedy: Evaluation of llms should not ignore non-determinism. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Techno…
  38. Michele Tufano, Dawn Drain, Alexey Svyatkovskiy, Shao Kun Deng, and Neel Sundaresan. 2021. Unit test case generation with transformers and focal context. Preprint, arXiv:2009.05617. — [arXiv:2009.05617](https://arxiv.org/abs/2009.05617) find
  39. Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2017. Graph attention networks. arXiv preprint arXiv:1710.10903.
  40. Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software testing with large language models: Survey, landscape, and vision. IEEE Transactions on Software Engineering, 50(4):911–936. find
  41. Wenhan Wang, Chenyuan Yang, Zhijie Wang, Yuheng Huang, Zhaoyang Chu, Da Song, Lingming Zhang, An Ran Chen, and Lei Ma. 2025. TestEval: Benchmarking large language models for test case generation. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3547–3562, Albuquerque,… find
  42. Felix Weissberg, Jonas Möller, Tom Ganz, Erik Im-grund, Lukas Pirch, Lukas Seidel, Moritz Schloegel, Thorsten Eisenhofer, and Konrad Rieck. 2024. Sok: Where to fuzz? assessing target selection methods in directed fuzzing. In Proceedings of the 19th ACM Asia Conference on Computer and Communications… find
  43. Martin White, Michele Tufano, Christopher Vendome, and Denys Poshyvanyk. 2016. Deep learning code fragments for code clone detection. In 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), pages 87–98. find
  44. Fabian Yamaguchi, Nico Golde, Daniel Arp, and Konrad Rieck. 2014. Modeling and discovering vulnerabilities with code property graphs. In 2014 IEEE symposium on security and privacy, pages 590–604. IEEE. find
  45. Gang Zhao and Jeff Huang. 2018. Deepsim: deep learning code functional similarity. In Proceedings of the 2018 26th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2018, page 141–151, New York, NY, USA. Association for C… find
  46. Xiaogang Zhu, Sheng Wen, Seyit Camtepe, and Yang Xiang. 2022. Fuzzing: a survey for roadmap. ACM Computing Surveys (CSUR), 54(11s):1–36. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tran-2026-program-structure-aware-language-models/