Elamur Research⌕
☰
home / papers / 2026 / Stylometry at SemEval-2026 Task 13: Clustered Stylometric Modeling for Machine-Generated Code Detection

Stylometry at SemEval-2026 Task 13: Clustered Stylometric Modeling for Machine-Generated Code Detection

Paper · 2026 · Sruthi Santhanam, Parthib Sarkar and Yashvardhan Sharma

Summary

The paper addresses whether a program was written by a person or produced by an LLM, as posed in SemEval-2026 Task 13 Subtask A. Training data cover only a few languages and one algorithmic domain, while the test set adds unseen languages and domains. Rather than fine-tuning large neural models, the authors build a lightweight stylometric pipeline. Each snippet is represented by character-level TF-IDF n-grams plus standardized structural descriptors such as length, line count, comment, digit and punctuation ratios, and character diversity. K-means clustering then routes snippets to cluster-specific experts, acting as a simple mixture-of-experts. Each expert averages logistic regression and multinomial Naïve Bayes probabilities, and a decision threshold is tuned on validation data to maximize macro F1. The system scores 0.6125 macro F1 on the official test set and places 23rd. This shows that cheap, interpretable signals tied to verbosity, formatting and commenting carry useful cross-domain information for spotting generated code.

Key points

  • Training data: roughly 91% Python, with C++ and Java making up the rest. AI samples come from 34 generators. Because of memory limits, a stratified subset of 80,000 training samples was used.
  • Exploratory analysis: generated code is longer than human code (mean 1052 vs. about 600 characters) and has more comments (comment ratio 0.105 vs. 0.03). Human code is more punctuation-dense.
  • Configuration:
    • Features: char 3–5-grams, vocabulary capped at 120,000, min_df of 5.
    • Clustering: MiniBatch K-means with K = 4.
    • Classifiers: L2 logistic regression and Naïve Bayes with α = 0.5.
    • Threshold: searched over [0.70, 0.99].
    • Tools: scikit-learn on Colab.
  • Ablation: removing threshold calibration causes the largest drop, to 0.338. A single logistic regression performs almost identically to the full ensemble (0.640 vs. 0.639).
  • Error analysis: on 1000 validation samples the error rate is 24.1%. Long, regular human code is often mislabeled as machine-generated. Code: github.com/parthib-s/semeval-2026-task13-stylometry.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Birla Institute of Technology and Science, PilaniPilāni · India16 works · 41,879 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Sruthi Santhanam, Parthib Sarkar and Yashvardhan Sharma. 2026. Stylometry at SemEval-2026 Task 13: Clustered Stylometric Modeling for Machine-Generated Code Detection. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.172

@inproceedings{santhanam2026stylometry,
  title = {{Stylometry at SemEval-2026 Task 13: Clustered Stylometric Modeling for Machine-Generated Code Detection}},
  author = {Sruthi Santhanam, Parthib Sarkar and Yashvardhan Sharma},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.172},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.172}
}

References

15 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aylin Caliskan-Islam, Richard Harang, Andrew Liu, Arvind Narayanan, Clare Voss, Fabian Yamaguchi, and Rachel Greenstadt. 2015. De-anonymizing programmers via code stylometry. In Proceedings of the 24th USENIX Security Symposium (USENIX Security 2015), pages 255–270, Washington, D.C. USENIX Associati… find
  2. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, and Ting Liu. 2020. Codebert: A pre-trained model for programming and natural languages. In Findings of EMNLP.
  3. Sebastian Gehrmann, Hendrik Strobelt, and Alexander Rush. 2019. Gltr: Statistical detection and visualization of generated text. Proceedings of ACL System Demonstrations.
  4. Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. 1991. Adaptive mixtures of local experts. Neural Computation, 3(1):79–87. find
  5. Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks V.S. Lakshmanan. 2020. Automatic detection of machine generated text: A critical survey. In Proceedings of COLING.
  6. Moshe Koppel, Jonathan Schler, and Shlomo Argamon. 2009. Computational methods in authorship attribution. Journal of the American Society for Information Science and Technology, 60(1):9–26. find
  7. Stuart Lloyd. 1982. Least squares quantization in PCM. IEEE Transactions on Information Theory, 28(2):129–137. find
  8. Andrew McCallum and Kamal Nigam. 1998. A comparison of event models for naive bayes text classification. In AAAI Workshop.
  9. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. 2023. DetectGPT: Zero-shot machine-generated text detection using probability curvature. In Proceedings of the 40th International Conference on Machine Learning (ICML 2023), volume 202 of Proceedings of Machine…
  10. Daniil Orel, Dilshod Azizov, Indraneil Paul, Yuxia Wang, Iryna Gurevych, and Preslav Nakov. 2026a. AICD bench: A challenging benchmark for AI-generated code detection. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Paper…
  11. Daniil Orel, Dilshod Azizov, Indraneil Paul, Yuxia Wang, Iryna Gurevych, and Preslav Nakov. 2026b. Semeval-2026 task 13: Detecting machine-generated code with multiple programming languages, generators, and application scenarios. In Proceedings of the 20th International Workshop on Semantic Evaluati…
  12. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. In OpenAI Technical Report. find
  13. Juan Ramos. 2003. Using tf-idf to determine word relevance in document queries. In ML Conference. find
  14. Efstathios Stamatatos. 2009. A survey of modern authorship attribution methods. Journal of the American Society for Information Science and Technology, 60(3):538–556. find
  15. Yue Wang, Weishi Wang, Shafiq Joty, and Steven Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In EMNLP. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/santhanam-2026-stylometry-semeval-2026-task-13/