Elamur Research⌕
☰
home / papers / 2026 / Transformer-Assisted LLM-Based Source Code Summarisation: to Enable More Secure Software Development

Transformer-Assisted LLM-Based Source Code Summarisation: to Enable More Secure Software Development

Paper · 2026 · Jesse Phillips, Tracy Hall, Paul Rayson et al.

Summary

Source code summaries help maintainers understand software and catch bugs and vulnerabilities, but in practice they are often absent or stale. Earlier automatic approaches fall into two camps. Small task-specific Transformers reproduce developer vocabulary and score well on overlap-based NLG metrics. LLMs write fluent, semantically rich text but diverge lexically from reference summaries. The authors merge the two by placing a summary generated by CodeSumBART into a one-shot prompt for an LLM. They first refine the cleaning of the Funcom Java dataset and retrain CodeSumBART on it. They then compare a zero-shot prompt with three one-shot variants (implicit, explicit, explicit requesting improvement) across four open LLMs under 10B parameters, all run on a single workstation-class GPU. Using the Transformer's output as an example raised scores for every model on all metrics. CodeLlama with the explicit prompt performed best. The approach is significant because it delivers better automated documentation for secure development using locally runnable models.

Key points

  • The dataset is Funcom, cleaned to 499,997 Java method–summary pairs split 80/10/10. Better tokenisation reduced out-of-dictionary tokens from 146,905 to 40,941.
  • The models tested were Llama 3.1, Llama 3.2, CodeLlama and Deepseek Coder 1.5. Outputs were scored with BLEU-1, BLEU-4, smoothed BLEU-4, ROUGE-L and BERTScore.
  • With the explicit one-shot prompt, CodeLlama scored BLEU-1 39.96, BLEU-4 12.42 and BERTScore 70.79. Compared with zero-shot, this is a gain of about 7.8 BLEU-4 points and 5 BERTScore points.
  • Swapping words like "summary" for alternatives such as "description" gave a slight further improvement (CodeLlama: BLEU-1 40.61, BERTScore 71.13).
  • The two code-aware LLMs diverged sharply. CodeLlama led every comparison, while Deepseek Coder trailed and did best only with the improvement-request prompt. The authors conclude that the benefit of code awareness depends on the specific model.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Lancaster UniversityLancaster · United Kingdom35 works · 1,615 citations🎓VinUniversityHanoi · Vietnam11 works · 56 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksSoftware · 92 works

🧭 Topics 2Research subjects and languages the work is about.Summarization · 232 worksCode & programming languages · 129 works

Cite

Cite as Jesse Phillips, Tracy Hall, Paul Rayson and Mo El-Haj. 2026. Transformer-Assisted LLM-Based Source Code Summarisation: to Enable More Secure Software Development. Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{phillips2026transformer,
  title = {{Transformer-Assisted LLM-Based Source Code Summarisation: to Enable More Secure Software Development}},
  author = {Jesse Phillips, Tracy Hall, Paul Rayson and Mo El-Haj},
  booktitle = {Proceedings of the Second International Conference on Natural Language Processing and Artificial Intelligence for Cyber Security},
  year = {2026},
  url = {https://aclanthology.org/2026.nlpaics-1.4/}
}

References

4 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai-Wei Chang. 2020. A transformer-based approach for source code summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4998–5007, Online. Association for Computational Linguistics.
  2. Toufique Ahmed and Premkumar Devanbu. 2022. Few-shot training llms for project-specific code-summarization. In Proceedings of the 37th IEEE/ACM international conference on automated software engineering, pages 1–5. find
  3. Toufique Ahmed, Kunal Suresh Pai, Premkumar Devanbu, and Earl Barr. 2024. Automatic semantic augmentation of language model prompts (for code summarization). In Proceedings of the IEEE/ACM 46th international conference on software engineering, pages 1–13. find
  4. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code-BERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547,… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/phillips-2026-transformer-assisted-llm-based-source/