HyperparameterOmens at SemEval-2026 Task 13: Various approaches to detecting machine- generated code
Paper · 2026 · Dmitry Sukhotin and How Yu
Summary
The paper describes the team's systems for SemEval-2026 Task 13, which asks whether source code was written by humans or by AI. The task is built on the Droid benchmark. In Subtask A (binary detection), training data is LeetCode-style code in three languages, while the test set contains production and research code in eight languages. Token- and character-level classifiers transferred poorly to this test set.
The authors found a parsing bug that silently labelled about half of the test samples "Human". After fixing it, they built tree-sitter AST features. They then filtered out features that reverse direction across domains or drift strongly, and pruned correlated ones. The result is a compact five-feature logistic regression that scores 0.57 macro F1.
In Subtask C (four-way authorship: human, AI, hybrid, adversarial), lexical models collapsed under vocabulary shift. CodeBERT and GraphCodeBERT stayed robust. A per-class weighted ensemble reached 0.85 on the 1K labelled sample but only 0.57 on the full test set.
The work shows that pipeline validation and shift-aware feature selection matter for code-authorship detection.
Key points
- Subtask A data: 500K training, 100K validation and 500K test examples. Training is 91% Python, and five test languages never appear in training. Lexical baselines (NBSVM, TF–IDF, char n-grams, CharCNN) scored only 0.22–0.41 on the public test.
- Parsing bug: A trivial-tree gate rejected 49.4% of test samples, versus 12.4% of training samples, and rejected samples defaulted to "Human". Removing the gate raised parse coverage above 99.9%.
- Feature selection: Sign-flip and Wasserstein-drift diagnostics flagged 15 of 25 AST features as unstable. After these were removed, any two features correlated at |r| > 0.6 were reduced to a single member, leaving 5 features. Test F1 rose from 0.38 to 0.57, the validation–test gap shrank from 0.52 to 0.13, and the private score was 0.56. A post-deadline five-model ensemble reached 0.62.
- Subtask C robustness: Validation-to-test drop on the 1K sample was about 55% for TF–IDF/NBSVM, 5–6% for XGBoost/LightGBM on AST features, and 0.3% for CharCNN (0.74 F1). CodeBERT reached about 0.83 F1, and GraphCodeBERT dropped only 0.04%.
- Subtask C ensemble: Weights for each class were tuned with Optuna (15,000 trials per class), giving 0.8521 macro F1 on the 1K sample and 0.5722 on the full test set. Hybrid and adversarial code remained hardest, averaging about 0.61 and 0.58 F1 across models, with only 9.4% and 13.1% of training data.
Institutions
GermanyFields, topics and concepts
📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works
Cite
Cite as Dmitry Sukhotin and How Yu. 2026. HyperparameterOmens at SemEval-2026 Task 13: Various approaches to detecting machine- generated code. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).
@inproceedings{sukhotin2026hyperparameteromens,
title = {{HyperparameterOmens at SemEval-2026 Task 13: Various approaches to detecting machine- generated code}},
author = {Dmitry Sukhotin and How Yu},
booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
year = {2026},
doi = {10.18653/v1/2026.semeval-1.177},
url = {https://doi.org/10.18653/v1/2026.semeval-1.177}
}References
- Takuya Akiba, Shotaro Sano, Toshiki Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 2623–2631.
- Aylin Caliskan-Islam, Richard Harang, Andrew Liu, Arvind Narayanan, Clare Voss, Fabian Yamaguchi, and Rachel Greenstadt. 2015. De-anonymizing programmers via code stylometry. In 24th USENIX Security Symposium (USENIX Security 15), pages 255–270. USENIX Association. find
- Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code-BERT: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547.… find
- FUNCTION_DEF, CALL, LOOP) before extraction. The five features used in the final model are: Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. 2020. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2… find
- Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, and Shengyu Fu. 2020. GraphCode-BERT: Pre-training code representations with data flow. arXiv preprint arXiv:2009.08366. branching_std. Standard deviation of the number of children across al…
- Daniil Orel, Dilshod Azizov, Indraneil Paul, Yuxia Wang, Iryna Gurevych, and Preslav Nakov. 2026. SemEval-2026 task 13: Detecting machine-generated code with multiple programming languages, generators, and application scenarios. In Proceedings of the 20th International Workshop on Semantic Evaluatio…
- Daniil Orel, Indraneil Paul, Iryna Gurevych, and Preslav Nakov. 2025. Droid: A resource suite for AI-generated code detection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31263–31289, Suzhou, China. Association for Computational Linguistics. find
- Adarsh Subbaswamy, Bryant Chen, and Suchi Saria. 2022. A unifying causal framework for analyzing dataset shift-stable learning algorithms. Journal of Causal Inference, 10(1):64–89. find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/sukhotin-2026-hyperparameteromens-semeval-2026-task-13/