Elamur Research⌕
☰
home / papers / 2026 / SemEval-2026 Task 13: Detecting Machine-Generated Code with Multiple Programming Languages, Generators, and Application Scenarios

SemEval-2026 Task 13: Detecting Machine-Generated Code with Multiple Programming Languages, Generators, and Application Scenarios

Paper · 2026 · Daniil Orel, Dilshod Azizov, Indraneil Paul et al.

Summary

This shared task targets the problem of telling whether source code was written by people or produced by LLMs. This matters for academic integrity, hiring, and software security. Modern models can imitate human coding style, and existing detectors tend to break outside their training distribution. The organizers built a large benchmark on top of Droid and CoDet-M4. It draws on sources such as CodeNet, StarCoder-Data, Vault, TACO, LeetCode and CodeForces, and its synthetic code comes from many open model families via inverse prompting, comment- or problem-based generation, partial rewriting, and adversarial "human-like" generation. Samples were checked with ASTs and deduplicated with MinHash.

The task has three tracks:

  • Subtask A: binary detection, evaluated under language and domain shift.
  • Subtask B: attribution to a model family.
  • Subtask C: four-way labeling (human, machine, hybrid, adversarial).

Systems were ranked on Kaggle by macro-F1. Generalization to the shifted test data proved difficult for most teams. Family attribution was the hardest track, with the best score only around 0.51. The resulting resource is being extended into AICD Bench.

Key points

  • Participation: 81 teams in Subtask A, 34 in Subtask B, and 32 in Subtask C.
  • Subtask A trains only on algorithmic code in Python, Java and C++. Its test set (about 500K samples) adds C, C#, Go, JavaScript and PHP, plus research and generic deployed code.
  • Subtask B uses 11 classes: human plus 10 provider families, including DeepSeek, Qwen, Llama, IBM-Granite and OpenAI. The data covers 45 generators, and the test set includes generators unseen in training.
  • Subtask C's test set brings in out-of-distribution data:
    • human code from The Heap;
    • hybrid samples from SwallowCoder, rewritten with Llama 3.3 70B Instruct.
  • Best macro-F1 scores against the baselines:
SubtaskWinning teamBest macro-F1Baseline
ATeleAI0.9970.305
BTeleAI0.5080.229
CYoung DSMLKZ0.7850.481

In Subtask A, the next-best team reached 0.824, far behind the winner.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany109 works · 35,920 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Code & programming languages · 129 works

Cite

Cite as Daniil Orel, Dilshod Azizov, Indraneil Paul, Yuxia Wang, Iryna Gurevych and Preslav Nakov. 2026. SemEval-2026 Task 13: Detecting Machine-Generated Code with Multiple Programming Languages, Generators, and Application Scenarios. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.445

@inproceedings{orel2026semeval,
  title = {{SemEval-2026 Task 13: Detecting Machine-Generated Code with Multiple Programming Languages, Generators, and Application Scenarios}},
  author = {Daniil Orel, Dilshod Azizov, Indraneil Paul, Yuxia Wang, Iryna Gurevych and Preslav Nakov},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.445},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.445}
}

References

66 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Team 01.AI. 2024. Yi: Open foundation models by 01.ai. ArXiv, abs/2403.04652.
  2. Mistral AI. 2025. Mistral small – a new balance of performance and efficiency. Online. Available at https://mistral.ai/news/mistral-small-3 (Accessed: 1 April 2025). ↗
  3. Azher Ali and Mehwish Fatima. 2026. NUST-CodeIntel at SemEval-2026 task 13: Cross-domain detection of machine-generated code via stylometric features and transformer models. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association for Compu… find
  4. Abhishek Anand, Shifali Agrahari, Shubham Nilesh Kannaujiya, Sanasam Ranbir Singh, and Sujit Kumar. 2026. Osint at SemEval-2026 task 13: A distribution-aware framework for code generation detection and multi-source authorship attribution. In Proceedings of the 20th International Workshop on Semantic… find
  5. Hüseyin Arda Arslan, Evren Ayberk Munis, Timofei Khudonogov, Mert Akgün, Murat Beşli, Ayhan Meherrem, Claudio Savelli, and Flavio Giobergia. 2026. MALTO at SemEval-2026 task 13: Detecting human, AI, and hybrid code via hard negative mining and curriculum-driven ensembles. In Proceedings of the 20th… find
  6. Bowen Baker, Joost Huizinga, Leo Gao, Zehao Dou, Melody Y. Guan, Aleksander Madry, Wojciech Zaremba, Jakub Pachocki, and David Farhi. 2025. Monitoring reasoning models for misbehavior and the risks of promoting obfuscation. arXiv preprint arXiv:2503.11926. find
  7. Sebastian Balmus , and Bogdan Dura. 2026. ICI Innolabs at SemEval-2026 task 13: Sliding windows meet code transformers. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association for Computational Linguistics. find
  8. Sufiyan Ahmed Bukhari. 2024. Issues in detection of AI-generated source code. University of Calgary. find
  9. Shruti Chandrasekar, Vedajanaani R S, and Vijayalakshmi P. 2026. Medhastra at SemEval-2026 task 13: Stylometric ensembles and transformer fine-tuning for robust AI code detection, attribution, and adversarial analysis. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval… find
  10. Longfeng Chen and Xiao Zheng. 2026. SYSUpporter at SemEval-2026 task 13: Leveraging stylistic signals and language-aware truncation for code generation detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association for Computational Li… find
  11. Shiti Chowdhury and Adnan Faisal. 2026. CUET_Luminaries_0227 at SemEval-2026 task 13: Invariance-oriented representation learning for robust AI-generated code detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association for Computati… find
  12. Aldan Creo, Atharv Nair, Mohana Kannan Ravikumar, Vaishak Menon, Dario Wisznewer, and Vaibhav Jain. 2026. COODetect at SemEval-2026 task 13: Unsupervised latent domain adaptation for out-of-distribution AI-generated code detection. In Proceedings of the 20th International Workshop on Semantic Evalua… find
  13. Pham Quoc Cuong, Nguyen Ngoc Minh, Le Quang Minh, Nguyen Hoai Nguyet An, and Nguyen Trong Chinh. 2026. UIT-AMMC at SemEval-2026 task 13: Exploiting structural formatting signatures for robust AI-generated code detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEv…
  14. Team DeepSeek. 2024. Deepseek-coder: When the large language model meets programming – the rise of code intelligence. arXiv preprint arXiv:2401.14196. find
  15. Ruslan Berdichevsky, Shai Nahum-Gefen, and Elad Ben-Zaken. 2026. Dream at SemEval-2026 task 13: SALSA for single-pass machine-generated code detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association for Computational Linguistics. find
  16. Anıl Dervişoğlu and Atakan Site. 2026. Codexa at SemEval-2026 task 13: Loss engineering and diverse ensemble strategies for multi-class code authorship attribution. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association for Computationa… find
  17. Andrei Z. Broder. 1997. On the resemblance and containment of documents. In Compression and Complexity of SEQUENCES 1997, Positano, Amalfitan Coast, Salerno, Italy, June 11-13, 1997, Proceedings, pages 21–29. IEEE. find
  18. Vivek Dhayal. 2026. Team vivek dhayaal at SemEval-2026 task 13: Multi-class authorship detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association for Computational Linguistics. find
  19. How Chuen Yu Dmitry Sukhotin. 2026. Segmentation Fault at SemEval-2026 task 13: Various approaches to detecting machine- generated code. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association for Computational Linguistics. find
  20. Jany-Gabriel Ispas and Sergiu Nisioi. 2026. Archaeology at SemEval-2026 task 13: Fine-tuning pre-trained code models for AI-generated code detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association for Computational Linguistics. find
  21. Daniel-Antoniu Dumitru, Simina Lazăr, Nicoleta Danilă (Amargheoalei), Daniela Gîfu, and Diana Trăndăbăt , . 2026. CodeHunters at SemEval-2026 task 13: Detecting machine-generated code with multiple programming languages, generators, and application scenarios. In Proceedings of the 20th Internat…
  22. Bhuvana J, Ramanan Mahendran, Siddharth Chandrasekar S, Pragatheesh J, and Rethanya P. 2026. SSN-CSE-CODECATALYSTS at SemEval-2026 task 13: Integrating transformer semantics and AST-derived structural features for code classification. In Proceedings of the 20th International Workshop on Semantic Eva… find
  23. Fariska Zakhralativa Ruskanda Farhan Nafis Rayhan. 2026. Farhan nafis rayhan at SemEval-2026 task 13: Supervised contrastive learning approach with gated multiclass decomposition ensemble architecture for code authorship identification. In Proceedings of the 20th International Workshop on Semantic E…
  24. Kazuki Fujii, Yukito Tajima, Sakae Mizuki, Hinari Shimada, Taihei Shiotani, Koshiro Saito, Masanari Ohi, Masaki Kawamura, Taishi Nakamura, Takumi Okamoto, Shigeki Ishida, Kakeru Hattori, Youmi Ma, Hiroya Takamura, Rio Yokota, and Naoaki Okazaki. 2025. Rewriting pre-training data boosts LLM performan… find
  25. Yeraly Gainulla and Agzam Shamsadinov. 2026. YoungDSMLKZ at SemEval-2026 task 13: Mil-unixcoder with meta-stacking and handcrafted features for AI-generated code detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association for Comput… find
  26. Team Gemma. 2024. Gemma: Open models based on Gemini research and technology. arXiv preprint arXiv:2403.08295. find
  27. Sai Laasya Gorantla, Shreemithra Naveen, and Steven Bethard. 2026. Team_SLS at SemEval-2026 task 13: Detecting machine-generated code with Code-BERT and structural features. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association for Compu… find
  28. Shahir Habib and Dipankar Das. 2026. Königsberg at SemEval-2026 task 13: Beyond language models: A low-resource feature-driven and data-flow embedding approach for machine-generated code detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, US… find
  29. Team IBM. 2024. Granite code models: A family of open foundation models for code intelligence. ArXiv, abs/2405.04324. find
  30. Talukder Naemul Hasan Naem Jerin Romijah Tuli, MD. Sartaj Alam Pritom. 2026. Mind_Flayer at SemEval-2026 task 13: Calibration-aware ensemble routing for cross-language AI-generated code detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA… find
  31. Jonathan Katzy, Razvan Mihai Popescu, Arie van Deursen, and Maliheh Izadi. 2025. The Heap: A contamination-free multilingual code dataset for evaluating large language models. In IEEE/ACM Second International Conference on AI Foundation Models and Software Engineering, Forge@ICSE ’25, pages 151–155,… find
  32. Ryuto Koike, Masahiro Kaneko, and Naoaki Okazaki. 2024. OUTFOX: LLM-generated essay detection through in-context learning with adversarially generated examples. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Applications of Artificial In… find
  33. Linda Kupfer, Lisa Hader, Christian Jaumann, and Annemarie Friedrich. 2026. WWTC@UniA at SemEval-2026 task 13: BERT-based code authorship detection and qualitative analysis. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association for Compu… find
  34. Rongao Li, Jie Fu, Bo-Wen Zhang, Tao Huang, Zhihong Sun, Chen Lyu, Guang Liu, Zhi Jin, and Ge Li. 2023. TACO: topics in algorithmic code generation dataset. CoRR, abs/2312.14852. find
  35. Yongqiang Ma, Jiawei Liu, Fan Yi, Qikai Cheng, Yong Huang, Wei Lu, and Xiaozhong Liu. 2023. AI vs. human–differentiation analysis of scientific content generation. arXiv preprint arXiv:2301.10416. — [arXiv:2301.10416](https://arxiv.org/abs/2301.10416) find
  36. Dung Nguyen Manh, Nam Le Hai, Anh T. V. Dau, Anh Minh Nguyen, Khanh Nghiem, Jin Guo, and Nghi D. Q. Bui. 2023. The vault: A comprehensive multilingual dataset for advancing code understanding and generation. find
  37. Team Meta. 2024. The Llama 3 herd of models. arXiv preprint arXiv:2407.21783. of GitHub Copilot’s code contributions. Commun. ACM, 68(2):96–105.
  38. Microsoft. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905.
  39. Ruchir Puri, David S. Kung, Geert Janssen, Wei Zhang, Giacomo Domeniconi, Vladimir Zolotov, Julian Dolby, Jie Chen, Mihir Choudhury, Lindsey Decker, Veronika Thost, Luca Buratti, Saurabh Pujar, Shyam Ramji, Ulrich Finkler, Susan Malaika, and Frederick Reiss. 2021. CodeNet: A large-scale AI for code… find
  40. Ruwad Naswan, Dipit Saha, Md. Rafat Kabir, and Nabiha Tahseen. 2026. ASTraNet at SemEval-2026 task 13: Not all code looks the same: Multi-view structural and semantic detection of machine-generated code. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Die…
  41. Phuong T. Nguyen, Juri Di Rocco, Claudio Di Sipio, Riccardo Rubei, Davide Di Ruscio, and Massimiliano Di Penta. 2024. GPTSniffer: A CodeBERT-based classifier to detect source code written by Chat-GPT. J. Syst. Softw., 214:112059. find
  42. Henrique Gomes Nunes, Eduardo Figueiredo, Larissa Rocha Soares, Sarah Nadi, Fischer Ferreira, and Geanderson E. dos Santos. 2025. Evaluating the effectiveness of LLMs in fixing maintainability issues in real-world projects. CoRR, abs/2502.02368. — [arXiv:2502.02368](https://arxiv.org/abs/2502.02368) find
  43. Team Qwen. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  44. Musfiqur Rahman, SayedHassan Khatoonabadi, Ahmad Abdellatif, and Emad Shihab. 2024. Automatic detection of LLM-generated code: A case study of Claude 3 Haiku. CoRR, abs/2409.01382. — [arXiv:2409.01382](https://arxiv.org/abs/2409.01382) find
  45. Yuvan Ramesh and Tongtong Wu. 2026. Yuvan at SemEval-2026 task 13: Multi-strategy ensemble approaches for AI-generated code detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association for Computational Linguistics. find
  46. Daniil Orel, Dilshod Azizov, and Preslav Nakov. 2025a. Codet-m4: Detecting machine-generated code in multi-lingual, multi-generator and multi-domain settings. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10570–10593. find
  47. Oarisa Rebayet and Radiul Walee. 2026. CSMetro at SemEval-2026 task 13: A feature-driven weighted ensemble for AI-generated code detection. In Proceedings of the 20th Int. Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association for Computational Linguistics. find
  48. Daniil Orel, Dilshod Azizov, Indraneil Paul, Yuxia Wang, Iryna Gurevych, and Preslav Nakov. 2026. AICD bench: A challenging benchmark for AI-generated code detection. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics, Rabat, Morocco. Assoc…
  49. Lekkala Sai Teja, Kshitij Patiyal, Sangam Sai Anish, Annepaka Yadagiri, and Partha Pakray. 2026. CodeDet-NITS at SemEval-2026 task 13: AI code authorship detection beyond truncation. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA. Association…
  50. Daniil Orel, Indraneil Paul, Iryna Gurevych, and Preslav Nakov. 2025b. Droid: A resource suite for AI-generated code detection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31263–31289, Suzhou, China. Association for Computational Linguistics. find
  51. Nahid Niyaz Shovon and MD. Naim Parvez. 2026. Team_Omega at SemEval-2026 task 13: Frozen vs. trainable representations for out-of-distribution AI-generated code detection: A CodeBERT fine-tuning study. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), San Diego… find
  52. Avi Patel, Manthan Laddha, Pushti Sapovadiya, Pruthwik Mishra, and Shrikant Malviya. 2026. Transformertrio at SemEval-2026 task 13: Navigating domain shift and representation instability in machine-generated code detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (Se…
  53. Indraneil Paul, Haoyi Yang, Goran Glavaš, Kristian Kersting, and Iryna Gurevych. 2025. ObscuraCoder: Powering efficient code LM pre-training via obfuscation grounding. In The Thirteenth International Conference on Learning Representations. find
  54. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? assessing the security Parthib Sarkar Sruthi Santhanam and Yashvardhan Sharma. 2026. Stylometry at SemEval-2026 task 13: Clustered stylometric modeling for machine-generated code detect…
  55. Team StarCoder. 2023. Starcoder: may the source be with you! find
  56. D. Thenmozhi, M. Monesh, S. K. Paveethiran, K. S. Mayur, and K. Chandraprakash. 2026. SSN-CSE-CodeBrothers at SemEval-2026 task 13: Authorship attribution of LLM-generated code using fine-tuned CodeBERT. In Proceedings of the 20th Int. Workshop on Semantic Evaluation (SemEval-2026), San Diego, USA.… find
  57. Kareem Ashraf Elozeiri, Saad El Dine Ahmed El Etter, Maiya Goloburda, Tarek Mahmoud, Raj Vardhan Tomar, Nurkhan Laiyk, Osama Mohammed Afzal, Ryuto Koike, Masahiro Kaneko, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2025. GenAI content detection task 1: English and multilingual… find
  58. S Udaythalavesh, Rajalakshmi Sivanaiah, and Angel Deborah S. 2026. UdayThalavesh S at SemEval-2026 task 13: Fine-tuned CodeBERT with balanced sampling and adaptive threshold optimization for multi-language AI-generated code detection. In Proceedings of the 20th International Workshop on Semantic Eva… find
  59. Andric Valdez, Emmanuel Ancona, Sebastián Bernardino, Helena Gomez-Adorno, Fazlourrahman Balouchzahi, and Fabian Herrera. 2026. Late-iimas at SemEval-2026 task 13: Evaluating GNNs, PLMs, LLMs, and stylometry for automatic code identification. In Proceedings of the 20th International Workshop on Sema…
  60. Veniamin Veselovsky, Manoel Horta Ribeiro, and Robert West. 2023. Artificial artificial artificial intelligence: Crowd workers widely use large language models for text production tasks. CoRR, abs/2306.07899.
  61. Xiaodan Xu, Chao Ni, Xinrong Guo, Shaoxuan Liu, Xiaoya Wang, Kui Liu, and Xiaohu Yang. 2024. Distinguishing LLM-generated from human-written code by contrastive learning. CoRR, abs/2411.04704. — [arXiv:2411.04704](https://arxiv.org/abs/2411.04704) find
  62. Elitsa Yotkova, Violeta Kastreva, Dimitar Dimitrov, Ivan Koychev, and Preslav Nakov. 2026. FMI_SU_Yotkova_Kastreva at SemEval-2026 task 13: Lightweight detection of LLM-generated code via stylometric signals. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), Sa… find
  63. Shiquan Wang, Ruiyu Fang, Shuangyong Song, Yongxiang Li, and Xuelong Li. 2026. TeleAI at SemEval-2026 task 13: Data-centric full-parameter fine-tuning with multi-level ensembling for code generation detection. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026), S… find
  64. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Osama Mohammed Afzal, Tarek Mahmoud, Giovanni Puccetti, Thomas Arnold, Chenxi Whitehouse, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024a. Semeval-2024 task 8: Multigenerator, multidomain, a… find
  65. Yuxia Wang, Jonibek Mansurov, Petar Ivanov, Jinyan Su, Artem Shelmanov, Akim Tsvigun, Chenxi Whitehouse, Osama Mohammed Afzal, Tarek Mahmoud, Toru Sasaki, Thomas Arnold, Alham Fikri Aji, Nizar Habash, Iryna Gurevych, and Preslav Nakov. 2024b. M4: Multi-generator, multi-domain, and multi-lingual blac… find
  66. Yuxia Wang, Artem Shelmanov, Jonibek Mansurov, Akim Tsvigun, Vladislav Mikhailov, Rui Xing, Zhuohan Xie, Jiahui Geng, Giovanni Puccetti, Ekaterina Artemova, Jinyan Su, Minh Ngoc Ta, Mervat Abassy, find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/orel-2026-semeval-2026-task-13-detecting/