Elamur Research⌕
☰
home / papers / 2026 / TransformerTrio at SemEval-2026 Task 13: Navigating Domain Shift and Representation Instability in Machine-Generated Code Detection

TransformerTrio at SemEval-2026 Task 13: Navigating Domain Shift and Representation Instability in Machine-Generated Code Detection

Paper · 2026 · Avi Patel, Manthan Laddha, Pushti Sapovadiya et al.

Summary

This paper describes the TransformerTrio systems for SemEval-2026 Task 13, which asks whether source code was written by a person or an LLM when programming languages, domains and generators shift between training and test. The task has three subtasks: binary human/machine detection (A), 11-way attribution to human or one of ten LLM families (B), and four-way labelling as human, machine, hybrid or adversarial (C). The data come from the AICD Bench benchmark. The authors compare five model types:

  • XGBoost on five handcrafted stylometric features
  • XGBoost on frozen ModernBERT-Base embeddings
  • adversarially filtered UniXcoder embeddings
  • fine-tuned ModernBERT-Large
  • a UniXcoder logit-stacking meta-classifier

The simple feature model wins Subtask A, while embedding-based variants generalise worse. About 40% of embedding dimensions flip their correlation with the label across domains, and removing domain-separable dimensions does not help. End-to-end fine-tuning works best for B and C, though B suffers from severe class imbalance. Overall, the results suggest that robust, well-aligned features matter more than model complexity for detecting machine-written code.

Key points

  • Subtask A (macro F1): the five-feature XGBoost model (Model A1) scored 0.6724, against 0.6031 with ModernBERT-Base embeddings and 0.2906 with adversarially filtered UniXcoder embeddings. Tuning the decision threshold on the dev set mattered a lot: Model A1 improved from 0.6160 to 0.6724 at threshold 0.84.
  • Fine-tuning UniXcoder-Base raised in-domain validation F1 while dev-set F1 steadily dropped. An adversarial classifier separated train and test embeddings with ROC–AUC 0.94, and filtering only lowered this to 0.87.
  • Subtask B: fine-tuned ModernBERT-Large reached macro F1 0.3997. Human (F1 0.98) and OpenAI (0.75) were recognised reliably, but the other families ranged from 0.07 to 0.52.
  • Subtask C: single-stage ModernBERT-Large (Model C1, 0.6337) slightly beat UniXcoder logit stacking with an XGBoost meta-classifier (Model C2, 0.6251). Dev scores were much higher (0.8565 and 0.8254), which points to strong distribution shift.
  • In Subtask C, the main confusions were among the machine-generated, hybrid and adversarial classes. PHP was the weakest language (F1 0.554) and C the strongest (0.896). Code is released at github.com/Avi4306/SemEval-2026-Task-13.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sardar Vallabhbhai National Institute of Technology SuratSurat · India5 works · 4 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

Cite

Cite as Avi Patel, Manthan Laddha, Pushti Sapovadiya, Pruthwik Mishra and Shrikant Malviya. 2026. TransformerTrio at SemEval-2026 Task 13: Navigating Domain Shift and Representation Instability in Machine-Generated Code Detection. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.140

@inproceedings{patel2026transformertrio,
  title = {{TransformerTrio at SemEval-2026 Task 13: Navigating Domain Shift and Representation Instability in Machine-Generated Code Detection}},
  author = {Avi Patel, Manthan Laddha, Pushti Sapovadiya, Pruthwik Mishra and Shrikant Malviya},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.140},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.140}
}

References

23 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tianqi Chen and Carlos Guestrin. 2016a. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 785–794, New York, NY, USA. Association for Computing Machinery. find
  2. Tianqi Chen and Carlos Guestrin. 2016b. Xgboost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’16, page 785–794. ACM. find
  3. Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified cross-modal pre-training for code representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7212–7225, Dublin, Ireland. Associa…
  4. Hanxi Guo, Siyuan Cheng, Kaiyuan Zhang, Guangyu Shen, and Xiangyu Zhang. 2025. Codemirage: A multi-lingual benchmark for detecting ai-generated and paraphrased source code from production-level llms. Preprint, arXiv:2506.11059. find
  5. Maurice H Halstead. 1977. Elements of Software Science (Operating and programming systems series). Elsevier Science Inc. find
  6. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2020. Codesearchnet challenge: Evaluating the state of semantic code search. Preprint, arXiv:1909.09436.
  7. Oseremen Joy Idialu, Noble Saji Mathews, Rungroj Maipradit, Joanne M. Atlee, and Mei Nagappan. 2024. Whodunit: Classifying code as human authored or gpt-4 generated - a case study on codechef problems. In Proceedings of the 21st International Conference on Mining Software Repositories, MSR ’24, page… find
  8. Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, page 7167–7177, Red Hook, NY, USA. Curran Ass… find
  9. Ke Li, Sheng Hong, Cai Fu, Yunhe Zhang, and Ming Liu. 2023. Discriminating human-authored from chatgpt-generated code via discernable feature analysis. In 2023 IEEE 34th International Symposium on Software Reliability Engineering Workshops (ISS-REW), pages 120–127. find
  10. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. Preprint, arXiv:1711.05101. find
  11. Phuong T. Nguyen, Juri Di Rocco, Claudio Di Sipio, Riccardo Rubei, Davide Di Ruscio, and Massimiliano Di Penta. 2024. Gptsniffer: A codebert-based classifier to detect source code written by chatgpt. Journal of Systems and Software, 214:112059. find
  12. Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. 2023. Towards stable test-time adaptation in dynamic wild world. In Internetional Conference on Learning Representations. find
  13. Daniil Orel, Dilshod Azizov, and Preslav Nakov. 2025a. CoDet-m4: Detecting machine-generated code in multi-lingual, multi-generator and multi-domain settings. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10570–10593, Vienna, Austria. Association for Computational Lin… find
  14. Daniil Orel, Dilshod Azizov, Indraneil Paul, Yuxia Wang, Iryna Gurevych, and Preslav Nakov. 2026a. AICD bench: A challenging benchmark for ai-generated code detection. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Paper…
  15. Daniil Orel, Dilshod Azizov, Indraneil Paul, Yuxia Wang, Iryna Gurevych, and Preslav Nakov. 2026b. SemEval-2026 task 13: Detecting machine-generated code with multiple programming languages, generators, and application scenarios. In Proceedings of the 20th International Workshop on Semantic Evaluati…
  16. Daniil Orel, Indraneil Paul, Iryna Gurevych, and Preslav Nakov. 2025b. Droid: A resource suite for ai-generated code detection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31251–31277. find
  17. Damien Teney, Yong Lin, Seong Joon Oh, and Ehsan Abbasnejad. 2023. Id and ood performance are sometimes inversely correlated on real-world datasets. Preprint, arXiv:2209.00613. — [arXiv:2209.00613](https://arxiv.org/abs/2209.00613) find
  18. Eduard Tulchinskii, Kristian Kuznetsov, Laida Kushnareva, Daniil Cherniavskii, Sergey Nikolenko, Evgeny Burnaev, Serguei Barannikov, and Irina Piontkovskaya. 2023. Intrinsic dimension estimation for robust detection of ai-generated texts. In Advances in Neural Information Processing Systems, volume…
  19. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  20. Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. 2021. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations. find
  21. Yanwen Wang, Mahdi Khodadadzadeh, and Raúl Zurita-Milla. 2025. On the use of adversarial validation for quantifying dissimilarity in geospatial machine learning prediction. GIScience & Remote Sensing, 62(1). find
  22. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Griffin Thomas Adams, Jeremy Howard, and Iacopo Poli. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory e…
  23. Xiaodan Xu, Chao Ni, Xinrong Guo, Shaoxuan Liu, Xiaoya Wang, Kui Liu, and Xiaohu Yang. 2025. Distinguishing llm-generated from human-written code by contrastive learning. ACM Trans. Softw. Eng. Methodol., 34(4). find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/patel-2026-transformertrio-semeval-2026-task-13/