Elamur Research⌕
☰
home / papers / 2026 / mcdok at SemEval-2026 Task 13: Finetuning LLMs for Detection of Machine-Generated Code

mcdok at SemEval-2026 Task 13: Finetuning LLMs for Detection of Machine-Generated Code

Paper · 2026 · Adam Skurla, Dominik Macko and Jakub Simko

Summary

The paper addresses detecting machine-generated source code within SemEval-2026 Task 13. The task has three subtasks: binary human-vs-machine detection (A), attributing code to one of 10 LLM families or human authorship (B), and a four-way split into human, machine, hybrid and adversarial code (C). The authors adapt their earlier mdok detector, built for machine-generated text, into mcdok. They drop text preprocessing that would break code and keep the class-balancing data selection. Their main experiment is a comparison of base models for QLoRA sequence-classification fine-tuning, using only the organizers' data. Different backbones won in each subtask:

  • A: the general-purpose Gemma-3-27B-PT, but only with a strict decision threshold.
  • B: CodeGemma-7B.
  • C: Qwen2.5-Coder-14B.

All three systems beat the official CodeBERT baseline and placed in the upper part of the leaderboards. However, they remain well behind the leaders. The work suggests that generic machine-text detection pipelines can be moved to the code domain with modest changes, and the code and models are released for replication.

Key points

  • Fine-tuning setup: QLoRA with 4-bit quantization and LoRA r=64, alpha 16, dropout 0.1. Training used batch size 1, learning rate 2e-5 with a cosine schedule, and up to 3 epochs. The loss was cross-entropy weighted by inverse class frequency, and checkpoints were selected on test_sample.parquet as an out-of-distribution validation set.
  • Subtask A: Gemma-3-27B-PT scored only 0.48 Macro F1 at the default 0.5 threshold. Requiring full confidence for the machine class (threshold 1.0) raised this to about 0.698, against 0.305 for the CodeBERT baseline.
  • Subtask B: CodeGemma-7B trained on the full, unbalanced data reached 0.396 Macro F1, versus 0.229 for the baseline. Undersampled "balanced" variants scored lower.
  • Subtask C: Qwen2.5-Coder-14B achieved 0.686 Macro F1, versus 0.481 for the baseline. Hybrid code was the hardest class and was most often confused with machine-generated code.
  • Rankings and further analysis: The unofficial Kaggle ranks were 10th of 81 (A), 13th of 34 (B) and 5th of 32 (C). The authors also analysed results on a leakage-filtered Droid test set. Performance was fairly stable across languages, including unseen ones such as C and C#, with Rust and PHP as outliers due to sparse data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Brno University of TechnologyBrno · Czechia11 works · 32,326 citations🤝Kempelen Institute of Intelligent TechnologiesBratislava · Slovakia11 works · 82 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

Cite

Cite as Adam Skurla, Dominik Macko and Jakub Simko. 2026. mcdok at SemEval-2026 Task 13: Finetuning LLMs for Detection of Machine-Generated Code. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.47

@inproceedings{skurla2026mcdok,
  title = {{mcdok at SemEval-2026 Task 13: Finetuning LLMs for Detection of Machine-Generated Code}},
  author = {Adam Skurla, Dominik Macko and Jakub Simko},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.47},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.47}
}

References

12 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Janek Bevendorff, Yuxia Wang, Jussi Karlgren, Matti Wiegmann, Maik Fröbe, Akim Tsivgun, Jinyan Su, Zhuohan Xie, Mervat T. Abassy, Jonibek Mansurov, Rui Xing, Minh Ngoc Ta, Kareem Ashraf Elozeiri, Tianle Gu, Raj Vardhan Tomar, Jiahui Geng, Ekaterina Artemova, Artem Shelmanov, Nizar Habash, and 5 othe… find
  2. Lucio La Cava, Dominik Macko, Róbert Móro, Ivan Srba, and Andrea Tagarelli. 2025. Authorship attribution in multilingual machine-generated texts. Preprint, arXiv:2508.01656. find
  3. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient fine-tuning of quantized LLMs. In Advances in Neural Information Processing Systems, volume 36, pages 10088–10115. Curran Associates, Inc. find
  4. Binyuan Hui, Jian Yang, Zeyu Cui, Jiaxi Yang, Dayiheng Liu, Lei Zhang, Tianyu Liu, Jiajun Zhang, Bowen Yu, Keming Lu, Kai Dang, Yang Fan, Yichang Zhang, An Yang, Rui Men, Fei Huang, Bo Zheng, Yibo Miao, Shanghaoran Quan, and 5 others. 2024. Qwen2.5-Coder technical report. Preprint, arXiv:2409.12186.
  5. Dominik Macko. 2025. mdok of KInIT: Robustly fine-tuned LLM for binary and multiclass AI-generated text detection. In Working Notes of CLEF 2025 – Conference and Labs of the Evaluation Forum, CEUR-WS.org. find
  6. Dominik Macko, Robert Moro, and Ivan Srba. 2025. Increasing the robustness of the fine-tuned multilingual machine-generated text detectors. Preprint, arXiv:2503.15128. — [arXiv:2503.15128](https://arxiv.org/abs/2503.15128) find
  7. Daniil Orel, Indraneil Paul, Iryna Gurevych, and Preslav Nakov. 2025. Droid: A resource suite for AI-generated code detection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31263–31289, Suzhou, China. Association for Computational Linguistics. find
  8. Daniil Orel, Indraneil Paul, Iryna Gurevych, and Preslav Nakov. 2026. SemEval-2026 task 13: Detecting machine-generated code with multiple programming languages, generators, and application scenarios. In Proceedings of the 20th International Workshop on Semantic Evaluation (SemEval-2026).
  9. Michal Spiegel and Dominik Macko. 2024a. IMGTB: A framework for machine-generated text detection benchmarking. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pages 172–179, Bangkok, Thailand. Association for Computational… find
  10. Michal Spiegel and Dominik Macko. 2024b. KInIT at SemEval-2024 task 8: Fine-tuned LLMs for multilingual machine-generated text detection. In Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), pages 558–564, Mexico City, Mexico. Association for Computational Linguis… find
  11. CodeGemma Team, Heri Zhao, Jeffrey Hui, Joshua Howland, Nam Nguyen, Siqi Zuo, Andrea Hu, Christopher A. Choquette-Choo, Jingyue Shen, Joe Kelley, Kshitij Bansal, Luke Vilnis, Mateo Wirth, Paul Michel, Peter Choy, Pratik Joshi, Ravin Kumar, Sarmad Hashmi, Shubham Agrawal, and 8 others. 2024. CodeGemm…
  12. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/skurla-2026-mcdok-semeval-2026-task-13/