📄 contestant001 at SemEval-2026 Task 13 Stylometric and TF-IDF-Based Detection of Machine-Generated Code

Summary

This SemEval-2026 Task 13 (Subtask A) system addresses binary detection of machine-generated code when training and test data differ. The training data has 500K LeetCode-style solutions in C++, Python and Java (48% human). The test set adds unseen languages (Go, PHP, C#, JavaScript, C), research and production domains, and a 78% human share. The authors first analyse stylistic traits. Generated code tends to have more comments and docstrings, more blank lines, more spaces and longer text, while human code uses tabs more often. They then build a two-branch ensemble. One branch feeds word-level TF-IDF n-grams with language tags into Logistic Regression. The other feeds 23 parser-free stylometric features into a Random Forest. The outputs are combined by weighted averaging with a raised decision threshold. On the practice test set the ensemble reaches 0.53 Macro F1 (0.5175 in the official submission). Fine-tuned CodeBERT and UniXcoder score about 0.99 on validation but drop to 0.24–0.27 on test. The work suggests that cheap, interpretable surface features generalise better than semantic code models under distribution shift.

Key points

• The ensemble score is 0.4·p_tfidf + 0.6·p_feat with a threshold of 0.7 instead of 0.5, to offset the class-ratio shift. Weights and threshold were chosen by manual grid search on the 100K validation split.
• Standalone results: the stylometric Random Forest reaches 0.51 Macro F1 and the improved TF-IDF model 0.44 (basic TF-IDF: 0.33). In the ablation, equal weights give 0.48 and the default threshold 0.44. The ablation table lists 0.51 for the final configuration, while the text reports 0.53.
• Per-language F1 on seen languages: Python 0.49, C++ 0.41, Java 0.25. Unseen languages range from 0.08 (PHP) to 0.29 (C).
• The authors attribute the transformer failure to two causes. Tokenizers normalise whitespace and formatting, which are the most discriminative cues. The models also exploit spurious training-set shortcuts.
• Setup: scikit-learn TF-IDF with 1–5-grams, 100K features and inverse language-frequency sample weights, run on Google Colab. The feature models train in minutes, while transformer fine-tuning takes 2–3 hours per model. The paper's byline lists Banafsheh Gholinejad as co-author.