Elamur Research⌕
☰
home / papers / 2026 / YNU-HPCC at SemEval-2026 Task 13: Robust Machine-Generated Code Detection under Distribution Shifts

YNU-HPCC at SemEval-2026 Task 13: Robust Machine-Generated Code Detection under Distribution Shifts

Paper · 2026 · Lixian Xing, Jin Wang and Xuejie Zhang

Summary

LLM-written code is now common, so separating it from human code matters for authorship, academic integrity and software reliability. SemEval-2026 Task 13 tests detectors under hard conditions: unseen programming languages and domains, many generator families, and snippets that mix human and machine writing. Using only the data provided by the organizers, the YNU-HPCC team compared three kinds of approaches. The first was classifiers built on code encoders (UniXcoder, GraphCodeBERT), either fully fine-tuned or kept frozen. The second was LightGBM trained on handcrafted stylometric features, Tree-Sitter AST features and LLM-logit features. The third was strategies tailored to each subtask.

  • Binary detection (Subtask A): a small set of simple surface features beat both the encoders and richer feature combinations. SHAP analysis showed indentation entropy was the strongest cue.
  • Generator attribution (Subtask B): a two-stage pipeline first separates human from machine code, then identifies the model family.
  • Hybrid detection (Subtask C): UniXcoder improved when trained on synthetic spliced human/machine samples together with a Supervised Contrastive Learning objective and Focal Loss.

The system ranked 12th of 81 teams, 14th of 34 and 8th of 32 in the three subtasks. The main lesson is that under distribution shift, cheap formatting signals and frozen pretrained weights can generalize better than deeper or fully adapted models.

Key points

  • Subtask A, encoders (macro F1): frozen GraphCodeBERT scored 0.51, frozen UniXcoder 0.47 and fully fine-tuned UniXcoder 0.43.
  • Subtask A, LightGBM: the 21-dimensional Shallow features scored 0.69, AST features 0.58 and Logits features 0.48. Combining Shallow with Logits dropped to 0.43. The logit features came from CodeLlama-7B, Llama-3.1-8B, CodeQwen-1.5-7B and Nxcode-orpo-7B.
  • Subtask B: the data is highly imbalanced (about 442k Human samples versus roughly 2k for some generators). The Stage 1 GraphCodeBERT classifier reached 0.97 F1 on validation. The full two-stage system scored 0.39 against 0.37 for a single-stage model. Adding Focal Loss or a language-prediction multitask head brought no further gain.
  • Subtask C, augmentation: synthetic hybrid samples were built from human and machine snippets in the same language, split at a random ratio α ∈ [0.25, 0.75], joined in random order and fixed at 512 tokens.
  • Subtask C, results: F1 rose from 0.57 for the UniXcoder baseline to 0.60 with SupCon and Focal Loss, and to 0.62 with the augmentation added.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Yunnan UniversityKunming · China20 works · 40 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

Cite

Cite as Lixian Xing, Jin Wang and Xuejie Zhang. 2026. YNU-HPCC at SemEval-2026 Task 13: Robust Machine-Generated Code Detection under Distribution Shifts. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.205

@inproceedings{xing2026ynu,
  title = {{YNU-HPCC at SemEval-2026 Task 13: Robust Machine-Generated Code Detection under Distribution Shifts}},
  author = {Lixian Xing, Jin Wang and Xuejie Zhang},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.205},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.205}
}

References

29 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Álvarez-Fidalgo and Francisco Ortin. 2025. Clave: A deep learning model for source code authorship verification with contrastive learning and transformer encoders. Information Processing & Management, 62(3):104005. find
  2. Maor Ashkenazi, Ofir Brenner, Tal Furman Shohet, and Eran Treister. 2025. Zero-shot detection of llm-generated code via approximated task conditioning. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 187– 204. Springer. find
  3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — Qwen Technical Report
  4. Tamas Bisztray, Bilel Cherif, Richard A Dubniczky, Nils Gruschka, Bertalan Borsos, Mohamed Amine Ferrag, Attila Kovacs, Vasileios Mavroeidis, and Norbert Tihanyi. 2025. I know which llm wrote your code last summer: Llm generated code stylometry for authorship attribution. In Proceedings of the 18th… find
  5. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhai… — Evaluating Large Language Models Trained on Code
  6. Atish Kumar Dipongkor, Ziyu Yao, and Kevin Moran. 2025. Reassessing code authorship attribution in the era of language models. arXiv preprint arXiv:2506.17120. — [arXiv:2506.17120](https://arxiv.org/abs/2506.17120) find
  7. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. Code-BERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16… find
  8. Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified Cross-Modal Pre-training for Code Representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 202…
  9. Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin B. Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. GraphCodeBERT: Pre-training Code Representations with Da…
  10. Hanxi Guo, Siyuan Cheng, Kaiyuan Zhang, Guangyu Shen, and Xiangyu Zhang. 2025. Codemirage: A multi-lingual benchmark for detecting ai-generated and paraphrased source code from production-level llms. arXiv preprint arXiv:2506.11059. find
  11. Jiwoo Hong, Noah Lee, and James Thorne. 2024. ORPO: Monolithic Preference Optimization without Reference Model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 12-16, 2024, pages 11170–11189. Association for Computationa… find
  12. Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. LightGBM: A Highly Efficient Gradient Boosting Decision Tree. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  13. Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Supervised Contrastive Learning. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS… find
  14. Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. 2017. Focal Loss for Dense Object Detection. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2999–3007. IEEE Computer Society.
  15. Scott Lundberg and Su-In Lee. 2017. A Unified Approach to Interpreting Model Predictions. ArXiv:1705.07874 [cs].
  16. Phuong T. Nguyen, Juri Di Rocco, Claudio Di Sipio, Riccardo Rubei, Davide Di Ruscio, and Massimiliano Di Penta. 2024. GPTSniffer: A CodeBERT-based classifier to detect source code written by ChatGPT. J. Syst. Softw., 214:112059. find
  17. Daniil Orel, Dilshod Azizov, and Preslav Nakov. 2025a. CoDet-M4: Detecting Machine-Generated Code in Multi-Lingual, Multi-Generator and Multi-Domain Settings. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 - August 1, 2025, volume ACL 2025 of Finding… find
  18. Daniil Orel, Dilshod Azizov, and Preslav Nakov. 2025b. Codet-m4: Detecting machine-generated code in multi-lingual, multi-generator and multi-domain settings. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10570–10593. find
  19. Daniil Orel, Dilshod Azizov, Indraneil Paul, Yuxia Wang, Iryna Gurevych, and Preslav Nakov. 2026a. Aicd bench: A challenging benchmark for ai-generated code detection. arXiv preprint arXiv:2602.02079.
  20. Conference on Software Engineering (ICSE), pages 1628–1639. IEEE. find
  21. Llama Team. 2024. The Llama 3 Herd of Models. CoRR, abs/2407.21783. ArXiv: 2407.21783.
  22. Yue Wang, Hung Le, Akhilesh Gotmare, Nghi Bui, Junnan Li, and Steven Hoi. 2023. Codet5+: Open code large language models for code understanding and generation. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 1069–1088. find
  23. Yang Xianjun, Zhang Kexun, Chen Haifeng, Petzold Linda, Wang William Yang, and Cheng Wei. 2023. Zero-shot detection of machine-generated codes. arXiv preprint arXiv:2310.05103. — [arXiv:2310.05103](https://arxiv.org/abs/2310.05103) find
  24. Tong Ye, Yangkai Du, Tengfei Ma, Lingfei Wu, Xuhong Zhang, Shouling Ji, and Wenhai Wang. 2025. Uncovering llm-generated code: A zero-shot synthetic code detector via code rewriting. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 968–976. find
  25. Daniil Orel, Dilshod Azizov, Indraneil Paul, Yuxia Wang, Iryna Gurevych, and Preslav Nakov. 2026b. SemEval-2026 task 13: Detecting machine-generated code with multiple programming languages, generators, and application scenarios. In Proceedings of the 20th International Workshop on Semantic Evaluati…
  26. Daniil Orel, Indraneil Paul, Iryna Gurevych, and Preslav Nakov. 2025c. Droid: A Resource Suite for AI-Generated Code Detection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, EMNLP 2025, Suzhou, China, November 4-9, 2025, pages 31263–31289. Association for… find
  27. Daniil Orel, Indraneil Paul, Iryna Gurevych, and Preslav Nakov. 2025d. Droid: A resource suite for ai-generated code detection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31251–31277. find
  28. Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton-Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal… — Code Llama: Open Foundation Models for Code
  29. Yuling Shi, Hongyu Zhang, Chengcheng Wan, and Xiaodong Gu. 2025. Between lines of code: Unraveling the distinct patterns of machine and human programmers. In 2025 IEEE/ACM 47th International find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xing-2026-ynu-hpcc-semeval-2026-task/