Elamur Research⌕
☰
home / papers / 2026 / MIUN BiasPatrol at SemEval-2026 Task 13: Why TF-IDF can Beat Transformers for OOD Code Detection

MIUN BiasPatrol at SemEval-2026 Task 13: Why TF-IDF can Beat Transformers for OOD Code Detection

Paper · 2026 · Loviza Sahlen, Thomas Springfeldt, Mehwish Fatima et al.

Summary

This work tackles SemEval-2026 Task 13:A: deciding whether a code snippet was written by a human or by an AI, with a focus on generalizing to unseen programming languages and domains. Training and validation data contain only algorithmic Python, Java and C++ code. The test set adds new languages and research/production code. The authors cleaned the data by dropping mislabeled, code-free, very short and very long samples. They then paired four classifiers (LogReg, SVM, Random Forest, XGBoost) with three kinds of input: nine handcrafted statistical features, TF-IDF over GraphCodeBERT subword tokens, and frozen [CLS] embeddings from UniXcoder and GraphCodeBERT.

Embedding-based pipelines scored highest on in-distribution validation but lost up to 77% of their Macro F1 on the out-of-distribution test. TF-IDF with tree-based models held up far better. Feature analysis traced much of this brittleness to whitespace cues. Normalizing spaces helped the classical models but did not close the gap for embeddings. The study argues that simple, well-normalized lexical features can be a more trustworthy basis for real-world detectors than larger representations.

Key points

  • Data: 500,000 training samples and 100,000 validation samples, plus a 1,000-sample test set that is 77.7% human-written.
    • Python makes up over 91% of the training data.
    • The test set adds C#, JavaScript, Go, C and PHP.
  • Best out-of-distribution system: RF + TF-IDF reached 56.90 Macro F1 on the test set after space normalization, despite only 68.72 on validation.
  • Embedding collapse: XGB + GraphCodeBERT scored 89.47 on validation but only 23.09 on the test set after normalization. Before normalization, its validation F1 was about 94 and its test F1 was 40.80.
  • Whitespace bias: Whitespace ratio was the most predictive statistical feature, which pointed to a formatting shortcut in the dataset.
  • Zero-shot baselines: Prompted zero-shot classification with DeBERTa-v3-large and GraphCodeBERT performed close to chance on out-of-distribution data, so these approaches were left out of the main comparison.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of Sciences and TechnologyIslamabad · Pakistan2 works

Also on the byline: Mid Sweden University.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

💡 Concepts 1Key ideas and methods explained on our concept pages.TF-IDF · 26 works

Cite

Cite as Loviza Sahlen, Thomas Springfeldt, Mehwish Fatima and Raja Khurram Shahzad. 2026. MIUN BiasPatrol at SemEval-2026 Task 13: Why TF-IDF can Beat Transformers for OOD Code Detection. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.312

@inproceedings{sahlen2026miun,
  title = {{MIUN BiasPatrol at SemEval-2026 Task 13: Why TF-IDF can Beat Transformers for OOD Code Detection}},
  author = {Loviza Sahlen, Thomas Springfeldt, Mehwish Fatima and Raja Khurram Shahzad},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.312},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.312}
}

References

22 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammed Abuhamad, Changhun Jung, David Mohaisen, and DaeHun Nyang. 2025. SHIELD: Thwarting code authorship attribution. IEEE Transactions on Dependable and Secure Computing, 22(5):4753– 4767. find
  2. Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning. Springer. find
  3. Tamas Bisztray, Bilel Cherif, Richard A. Dubniczky, Nils Gruschka, Bertalan Borsos, Mohamed Amine Ferrag, Attila Kovacs, Vasileios Mavroeidis, and Norbert Tihanyi. 2026. I know which llm wrote your code last summer: Llm generated code stylometry for authorship attribution. In Proceedings of the 18th… find
  4. Egor Bogomolov, Vladimir Kovalenko, Yurii Rebryk, Alberto Bacchelli, and Timofey Bryksin. 2021. Authorship attribution of source code: a language-agnostic approach and applicability in software engineering. In Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and… find
  5. Haroon Elahi and Guojun Wang. 2024. Forward-porting and its limitations in fuzzer evaluation. Information Sciences, 662:120142. find
  6. Tom Fawcett. 2006. An introduction to roc analysis. Pattern recognition letters, 27(8):861–874. find
  7. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Ming Feng, Ming Gong, Li Shou, Daxin Jiang, Ming Liu, and Ming Zhou. 2020. Codebert: A pre-trained model for programming and natural languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1536–1547. Association for Com…
  8. Daya Guo, Zhangyin Feng, Lili Zhang, Xiaodong Wang, Yufeng Wu, Lei Chen, Dongmei Yu, and Ming Zhou. 2021. Graphcodebert: Pre-training code representations with data flow. In Findings of the Association for Computational Linguistics: ACL 2021, pages 3306–3317.
  9. Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified cross-modal pre-training for code representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7212–7225. Association for Computat…
  10. Trevor Hastie, Robert Tibshirani, and Jerome Friedman. 2009. The Elements of Statistical Learning. Springer. find
  11. Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. 2023. Copyright violations and large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7403–7412. Association for Computational Linguistics.
  12. Jooyoung Lee, Toshini Agrawal, Adaku Uchendu, Thai Le, Jinghui Chen, and Dongwon Lee. 2025. Plag-Bench: Exploring the duality of large language models in plagiarism generation and detection. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computati… find
  13. Daniil Orel, Dilshod Azizov, and Preslav Nakov. 2025a. CoDet-m4: Detecting machine-generated code in multi-lingual, multi-generator and multi-domain settings. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10570–10593. Association for Computational Linguistics. find
  14. Daniil Orel, Indraneil Paul, Iryna Gurevych, and Preslav Nakov. 2025b. Droid: A resource suite for ai-generated code detection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31251–31277. find
  15. Shinwoo Park, Hyundong Jin, Jeong-Won Cha, and Yo-Sub Han. 2025. Detecting code paraphrased by large language models using coding style features. Engineering Applications of Artificial Intelligence, 162:112454. find
  16. Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2022. Asleep at the keyboard? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy, pages 754–768. find
  17. Norbert Tihanyi, Tamas Bisztray, Mohamed Amine Ferrag, Ridhi Jain, and Lucas Cordeiro. 2024. How secure is ai-generated code: a large-scale comparison of large language models. Empirical Software Engineering, 30. find
  18. Antonio Torralba and Alexei Efros. 2011. Unbiased look at dataset bias. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition, page 1521–1528. IEEE Computer Society. find
  19. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, and Jamie Brew. 2020. Transformers: State-of-the-art natural language processing. Proceedings of the 2020 Conference on Empirical Methods in Natural Lang…
  20. David Álvarez Fidalgo and Francisco Ortin. 2025. CLAVE: A deep learning model for source code authorship verification with contrastive learning and transformer encoders. Information Processing & Management, 62:104005. find
  21. Bc. Richard Čerňanský and Ing. Juraj Petrík. 2025. Comparing machine learning and deep learning models with attention mechanisms for multiple source code authorship attribution. In 2025 International Conference on Computer Systems and Technologies, pages 01–09. find
  22. Zhen Li, Guenevere Qian Chen, Chen Chen, Yayi Zou, and Shouhuai Xu. 2022. Ropgen: Towards robust code authorship attribution via automatic coding style transformation. In 2022 IEEE/ACM 44th International Conference on Software Engineering (, pages 1906–1918. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sahlen-2026-miun-biaspatrol-semeval-2026-task/