Elamur Research⌕
☰
home / papers / 2026 / FMISUYotkovaKastreva at SemEval-2026 Task 13: Lightweight Detection of LLM-Generated Code via Stylometric Signals

FMISUYotkovaKastreva at SemEval-2026 Task 13: Lightweight Detection of LLM-Generated Code via Stylometric Signals

Paper · 2026 · Elitsa Yotkova, Violeta Kastreva, Dimitar Dimitrov et al.

Summary

This paper presents a SemEval-2026 Task 13 Subtask A system that labels code snippets as human-written or LLM-generated. The setting is hard because the test data include programming languages and application domains never seen in training. The authors chose not to depend on large pretrained code encoders, which they found brittle under distribution shift. Instead, they built a cheap, interpretable pipeline from length-normalized stylometric ratios, mainly how much of a snippet is comments and how often verbs appear in those comments. Several helper components support feature extraction: Tree-Sitter for parsing, spaCy for spotting verbs, a programming-language identifier, and a line-level classifier that separates code from natural-language text. Pairing a depth-2 decision tree with heuristics derived from data analysis yielded a score of 67.35 in macro-F1 on the organizers' official evaluation set, which put the system within the top 15% of submissions. Training needs only a CPU and inference is almost instant. The work shows that simple, explainable signals can compete with heavy neural detectors.

Key points

  • Data: The training set has 500K snippets (238K human, 262K machine), about 457K of them in Python, written in C++, Python and Java. The test set adds Go, PHP, C#, C and JavaScript, plus research and production code.
  • Language identification: The language identifier was trained on Rosetta Code using character 3–8-gram TF.IDF features and Multinomial Naïve Bayes. It scored 95.10% accuracy on 1K test samples labeled with ChatGPT-5.2, with C/C++ and C#/Java as the main confusions.
  • Code-vs-text classifier: Built from Stack Overflow code and Twitch chat text, it uses character 3–5-gram TF.IDF features and a linear model. It reached 96.07 macro-F1 on validation and 86.43 on 30 manually annotated test samples.
  • Heuristics: One heuristic flags a standalone language-name line, a Markdown leak seen in about 5K samples, which added 0.05 macro-F1 over an all-human baseline. Another applies a text-like-ratio threshold of 0.3 and keeps 99.9% specificity on human code.
  • Classifier comparison: Logistic regression scored 63.16 macro-F1 (64.59 with threshold 0.65), and the depth-2 decision tree scored 65.62 (67.35 with heuristics). Features such as cyclomatic complexity, completeness score, and Statements-to-LOC contributed little.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates138 works · 6,590 citations🎓Sofia University "St. Kliment Ohridski"Sofia · Bulgaria23 works · 2,694 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

Cite

Cite as Elitsa Yotkova, Violeta Kastreva, Dimitar Dimitrov, Ivan Koychev and Preslav Nakov. 2026. FMISUYotkovaKastreva at SemEval-2026 Task 13: Lightweight Detection of LLM-Generated Code via Stylometric Signals. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.275

@inproceedings{yotkova2026fmisuyotkovakastreva,
  title = {{FMISUYotkovaKastreva at SemEval-2026 Task 13: Lightweight Detection of LLM-Generated Code via Stylometric Signals}},
  author = {Elitsa Yotkova, Violeta Kastreva, Dimitar Dimitrov, Ivan Koychev and Preslav Nakov},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.275},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.275}
}

References

18 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammad Bavarian, Heewoo Jun, Nikolas Tezak, John Schulman, Christine McLeavey, Jerry Tworek, and Mark Chen. 2022. Efficient training of language models to fill in the middle.
  2. Sufiyan Ahmed Bukhari. 2024. Issues in detection of AI-generated source code. Master’s thesis, University of Calgary, Calgary, Alberta, Canada. find
  3. Omer Dunay, Daniel Cheng, Adam Tait, Parth Thakkar, Peter C. Rigby, Andy Chiu, Imad Ahmad, Arun Ganesan, Chandra Maddila, Vijayaraghavan Murali, Ali Tayyebi, and Nachiappan Nagappan. 2024. Multiline AI-assisted code authoring. In Companion Proceedings of the 32nd ACM International Conference on the… find
  4. Alexander Froemmgen, Jacob Austin, Peter Choy, Nimesh Ghelani, Lera Kharatyan, Gabriela Surita, Elena Khrapko, Pascal Lamblin, Pierre-Antoine Manzagol, Marcus Revaj, Maxim Tabachnyk, Daniel Tarlow, Kevin Villela, Daniel Zheng, Satish Chandra, and Petros Maniatis. 2024. Resolving code review comments… find
  5. Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. Graphcodebert: Pre-training code representations with da…
  6. Kush Jain, Gabriel Synnaeve, and Baptiste Roziere. 2025. Testgeneval: A real world unit test generation and test completion benchmark. In International Conference on Learning Representations. michaelfumery. 2021. Stackoverflow questions filtered (2011–2021). Kaggle dataset. mowglii. 2020. Twitch cha… find
  7. Sebastian Nanz and Carlo A. Furia. 2014. A comparative study of programming languages in rosetta code. Preprint, arXiv:1409.0252. — [arXiv:1409.0252](https://arxiv.org/abs/1409.0252) find
  8. Phuong T. Nguyen, Juri Di Rocco, Claudio Di Sipio, Riccardo Rubei, Davide Di Ruscio, and Massimiliano Di Penta. 2024. GPTSniffer: A CodeBERT-based classifier to detect source code written by ChatGPT. Journal of Systems and Software, page 112059. find
  9. Daniil Orel, Dilshod Azizov, and Preslav Nakov. 2025a. CoDet-m4: Detecting machine-generated code in multi-lingual, multi-generator and multi-domain settings. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10570–10593, Vienna, Austria. Association for Computational Lin… find
  10. Daniil Orel, Dilshod Azizov, Indraneil Paul, Yuxia Wang, Iryna Gurevych, and Preslav Nakov. 2026. SemEval-2026 task 13: Detecting machine-generated code with multiple programming languages, generators, and application scenarios. In Proceedings of the 20th International Workshop on Semantic Evaluatio…
  11. Daniil Orel, Indraneil Paul, Iryna Gurevych, and Preslav Nakov. 2025b. Droid: A resource suite for AI-generated code detection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31263–31289, Suzhou, China. Association for Computational Linguistics. find
  12. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? Assessing the security of GitHub Copilot’s code contributions. Communications of the ACM, 68(2):96–105. find
  13. Musfiqur Rahman, SayedHassan Khatoonabadi, Ahmad Abdellatif, and Emad Shihab. 2024. Automatic detection of LLM-generated code: A comparative case study of contemporary models across function and class granularities. find
  14. Cynthia Rudin. 2019. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1:206–215.
  15. Miriam Sullivan, Andrew Kelly, and Paul McLaughlan. 2023. ChatGPT in higher education: Considerations for academic integrity and student learning. Journal of Applied Learning & Teaching, 6(1):31–40. find
  16. Tarun Suresh, Revanth Gangi Reddy, Yifei Xu, Zach Nussbaum, Andriy Mulyar, Brandon Duderstadt, and Heng Ji. 2025. CoRNStack: High-quality contrastive data for better code retrieval and reranking. In International Conference on Learning Representations. Poster. find
  17. Zhenyu Xu and Victor S. Sheng. 2025. CodeVision: Detecting LLM-generated code using 2D token probability maps and vision models. find
  18. An Yang, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoyan Huang, Jiandong Jiang, Jianhong Tu, Jianwei Zhang, Jingren Zhou, Junyang Lin, Kai Dang, Kexin Yang, Le Yu, Mei Li, Minmin Sun, Qin Zhu, Rui Men, Tao He, and 9 others. 2025. Qwen2.5-1m technical report.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yotkova-2026-fmisuyotkovakastreva-semeval-2026-task-13/