Elamur Research⌕
☰
home / papers / 2026 / HybridCodeAuthorship: A Benchmark Dataset for Line-Level Code Authorship Detection

HybridCodeAuthorship: A Benchmark Dataset for Line-Level Code Authorship Detection

Summary

As AI coding assistants spread, real-world source files increasingly mix lines written by people and lines produced by LLMs. Existing detection benchmarks mostly use LeetCode-style exercises and label whole snippets as entirely human or entirely machine-written, so they cannot test fine-grained localisation. The authors built HybridCodeAuthorship, a benchmark of Python files in which AI-generated segments are embedded inside genuine human-written code from repositories linked in CodeSearchNet.

Their pipeline has two stages. In the testing stage, original and modified files are run against the project's unit tests, and pass/fail labels are kept. In the interleaving stage, an LLM picks self-contained code blocks, replaces them with descriptive placeholder comments that act as proxies for developer prompts, and a model then regenerates those blocks. Syntax validity is checked afterwards.

Two state-of-the-art detectors were adapted to line- and chunk-level classification. Even the best one reached only moderate F1. This shows the task is hard and that the resource can help evaluate tools for measuring AI usage and managing risk in industrial codebases.

Key points

  • The source data is CodeSearchNet (GitHub and Microsoft Research), which links to roughly 2 million presumably human-written files. Only Python is used for now, and other languages are planned for later.
  • Replaced segments are marked with "GENERATE CODE:" comments and are then filled in by an LLM. The amount replaced varies to simulate different assistant acceptance rates.
  • Unit-test pass/fail labels for both human and AI-modified files are released, so users can filter out non-functional code.
  • AIGCode Detector, a DetectGPT-style method using CodeBERT perturbations, performed best. It reached F1 0.48 at line level and 0.56 at chunk level, where chunks are runs of consecutive lines by the same author type.
  • Error analysis showed that AI-modified code introduced Attribute, Type, Name and Value errors not seen in the human originals, and it had a higher share of Syntax errors. The dataset is released on GitHub (CapitalOne-Research/c1-hybrid-code-authorship).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4edsbxrqe8na

@inproceedings{patterson2026hybridcodeauthorship,
  title = {{HybridCodeAuthorship: A Benchmark Dataset for Line-Level Code Authorship Detection}},
  author = {Patterson, Luke S. and Wang, Li and Faulkner, Adam},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4edsbxrqe8na},
  url = {https://doi.org/10.63317/4edsbxrqe8na}
}

References

51 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mohammed Abuhamad, Tamer Abuhmed, Dae-Hun Nyang, and David Mohaisen. 2020. Multi-χ: Identifying multiple authors from source code files. Proceedings on Privacy Enhancing Technologies. find
  2. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K. Arora, et al. 2025. gpt-oss-120b & gpt-oss-20b model card. find our card
  3. Ajmain I Alam, Chanchal K Roy, and Kevin A Schneider. 2023. GPTCloneBench: A comprehensive benchmark of semantic clones and cross-language clones using GPT-3 model and SemanticCloneBench. In 2023 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 1–12. IEEE. find
  4. Ekaterina Artemova, Jason Lucas, Saranya Venkatraman, Jooyoung Lee, Sergei Tilga, Adaku Uchendu, and Vladislav Mikhailov. 2024. Beemo: Benchmark of expert-edited machine-generated outputs. ArXiv preprint:2411.04032. find
  5. Gal Bakal, Ali Dasdan, Yaniv Katz, Michael Kaufman, and Guy Levin. 2025. Experience with github copilot for developer productivity at zoom-info. arXiv preprint arXiv:2501.13282. — [arXiv:2501.13282](https://arxiv.org/abs/2501.13282) find
  6. Joel Becker, Nate Rush, Elizabeth Barnes, and David Rein. 2025. Measuring the impact of early-2025 ai on experienced open-source developer productivity. arXiv preprint arXiv:2507.09089. — [arXiv:2507.09089](https://arxiv.org/abs/2507.09089) find
  7. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  8. Luana Bulla, Alessandro Midolo, Misael Mongiovì, and Emiliano Tramontana. 2024. Ex-Code: A robust and explainable model to detect AI-generated code. Information, 15(12):819. find
  9. Aylin Çalışkan İslam, Richard Davison, Richard Liu, Rachel Greenstadt, and Arvind Narayanan. 2015. De-anonymizing programmers via code stylometry. In 24th USENIX Security Symposium (USENIX Security 15), pages 255–270. find
  10. Sayan Chatterjee, Ching Louis Liu, Gareth Rowland, and Tim Hogarth. 2024. The impact of ai tool on engineering at anz bank an empirical study on github copilot within corporate environment. arXiv preprint arXiv:2402.05636. — [arXiv:2402.05636](https://arxiv.org/abs/2402.05636) find
  11. Soohyeon Choi and David Mohaisen. 2025. Attributing ChatGPT-generated source codes. IEEE Transactions on Dependable and Secure Computing. find
  12. Copyleaks. 2023. AI Content Detector. https://copyleaks.com/ai-content-detector. ↗
  13. Zheyuan Kevin Cui, Mert Demirer, Sonia Jaffe, Leon Musolff, Sida Peng, and Tobias Salz. 2025. The effects of generative ai on high-skilled work: Evidence from three field experiments with software developers. Available at SSRN 4945566. find
  14. Basak Demirok and Mucahid Kutlu. 2024. AIG-CodeSet: A new annotated dataset for AI generated code detection. In 2025 33rd Signal Processing and Communications Applications Conference, pages 1–4. find
  15. Basak Demirok, Mucahid Kutlu, and Selin Mergen. 2025. MultiAIGCD: A comprehensive dataset for ai generated code detection covering multiple languages, models, prompts, and scenarios. arXiv preprint arXiv:2507.21693. — [arXiv:2507.21693](https://arxiv.org/abs/2507.21693) find
  16. Begum Karaci Deniz, Chandra Gnanasambandam, Martin Harrysson, Alharith Hussin, and Shivam Srivastava. 2023. Unleashing developer productivity with generative ai. McKinsey Digital, 7. find
  17. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv– 2407. find
  18. Liam Dugan, Alyssa Hwang, Filip Trhlik, Josh Magnus Ludan, Andrew Zhu, Hainiu Xu, Daphne Ippolito, and Chris Callison-Burch. 2024. RAId: A shared benchmark for robust evaluation of machine-generated text detectors. ArXiv preprint:2405.07940. find
  19. Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, et al. 2020. Codebert: A pre-trained model for programming and natural languages. In Findings of the association for computational linguistics: EMNLP 2020, pages 1536–1547. find our card
  20. GPTZero. 2023. GPTZero. https://gptzero.me/. ↗
  21. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. 2023. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. arXiv preprint arXiv:2301.07597. find our card
  22. Hanxi Guo et al. 2025. Codemirage: A multi-lingual benchmark for detecting ai-generated and paraphrased source code from production-level llms. arXiv preprint arXiv:2506.11059. find
  23. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. CodeSearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436. find our card
  24. Oseremen Joy Idialu, Noble Saji Mathews, Rungroj Maipradit, Joanne M. Atlee, and Mei Nagappan. 2024. Whodunit: Classifying code as human authored or GPT-4 generated-a case study on CodeChef problems. In Proceedings of the 21st International Conference on Mining Software Repositories, pages 394–406. find
  25. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues? arXiv preprint arXiv:2310.06770. find our card
  26. Ram Mohan Rao Kadiyala et al. 2025. Robust and fine-grained detection of AI generated texts. ArXiv preprint:2504.11952. — [arXiv:2504.11952](https://arxiv.org/abs/2504.11952) find
  27. Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. 2023. Starcoder: may the source be with you! arXiv preprint arXiv:2305.06161. find
  28. Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021. Codexglue: A machine learning benchmark dataset for code understanding and generation. arXiv preprint arXiv:2102.04664. find
  29. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. 2023. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In International conference on machine learning, pages 24950–24962. PMLR. find our card
  30. OpenAI. 2023. AI Text Classifier. https://pl atform.openai.com/ai-text-classifi er. find
  31. Daniil Orel, Indraneil Paul, Iryna Gurevych, and Preslav Nakov. 2025. Droid: A resource suite for ai-generated code detection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31251– 31277. find
  32. Wei Hung Pan, Ming Jie Chok, Jonathan Leong Shan Wong, Yung Xin Shin, Yeong Shian Poon, Zhou Yang, Chun Yong Chong, David Lo, and Mei Kuan Lim. 2024. Assessing AI detectors in identifying AI-generated code: Implications for education. In Proceedings of the 46th International Conference on Software E… find
  33. Elise Paradis, Kate Grey, Quinn Madison, Daye Nam, Andrew Macvean, Vahid Meimand, Nan Zhang, Ben Ferrari-Church, and Satish Chandra. 2025. How much does ai impact development speed? an enterprise-based randomized controlled trial. In 2025 IEEE/ACM 47th International Conference on Software Engineerin… find
  34. Sida Peng, Eirini Kalliamvakou, Peter Cihon, and Mert Demirer. 2023. The impact of ai on developer productivity: Evidence from github copilot. arXiv preprint arXiv:2302.06590. — [arXiv:2302.06590](https://arxiv.org/abs/2302.06590) find
  35. Musfiqur Rahman, SayedHassan Khatoonabadi, and Emad Shihab. 2025. Beyond synthetic benchmarks: Evaluating llm performance on real-world class-level code generation. arXiv preprint arXiv:2510.26130. — [arXiv:2510.26130](https://arxiv.org/abs/2510.26130) find
  36. Agnia Sergeyuk, Yaroslav Golubev, Timofey Bryksin, and Iftekhar Ahmed. 2025. Using AI-based coding assistants in practice: State of affairs, perceptions, and ways forward. Information and Software Technology, 178:107610. find
  37. Yuling Shi, Hongyu Zhang, Chengcheng Wan, and Xiaodong Gu. 2025. Between lines of code: Unraveling the distinct patterns of machine and human programmers. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 51–62. IEEE Computer Society. find
  38. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. 2019. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203. find our card
  39. Zhixiong Su, Yichen Wang, Herun Wan, Zhaohan Zhang, and Minnan Luo. 2025. HACo-Det: A study towards fine-grained machine-generated text detection under human-AI coauthoring. ArXiv preprint:2506.02959. find
  40. Florian Tambon, Arghavan Moradi-Dakhel, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Giuliano Antoniol. 2025. Bugs in large language models generated code: An empirical study. Empirical Software Engineering, 30(3):65. find
  41. Ningfei Wang, Shouling Ji, and Ting Wang. 2018. Integration of static and dynamic code stylometry analysis for programmer de-anonymization. In Proceedings of the 11th ACM Workshop on Artificial Intelligence and Security, pages 3–14. find
  42. Yue Wang, Hung Le, Akhilesh Deepak Gotmare, Nghi DQ Bui, Junnan Li, and Steven CH Hoi. 2023. Codet5+: Open code large language models for code understanding and generation. arXiv preprint arXiv:2305.07922. find
  43. Yue Wang, Weishi Wang, Shafiq Joty, and Steven CH Hoi. 2021. Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. arXiv preprint arXiv:2109.00859. find
  44. Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, et al. 2025. Smarter, better, faster, longer: A modern bidirectional encoder for fast, memory efficient, and long context finetuning and infer… find our card
  45. Hao Wen, Yueheng Zhu, Chao Liu, Xiaoxue Ren, Weiwei Du, and Meng Yan. 2024. Fixing function-level code generation errors for foundation large language models. arXiv preprint arXiv:2409.00676. — [arXiv:2409.00676](https://arxiv.org/abs/2409.00676) find
  46. Writer. 2023. AI Content Detector. https://wr iter.com/ai-content-detector. find
  47. Xiaodan Xu, Chao Ni, Xinrong Guo, Shaoxuan Liu, Xiaoya Wang, Kui Liu, and Xiaohu Yang. 2025. Distinguishing llm-generated from human-written code by contrastive learning. ACM Transactions on Software Engineering and Methodology, 34(4):1–31. find
  48. Zhenyu Xu and Victor S Sheng. 2024. Detecting ai-generated code assignments using perplexity of large language models. In Proceedings of the aaai conference on artificial intelligence, volume 38, pages 23155–23162. find
  49. Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, et al. 2023. Natural language to code generation in interactive data science notebooks. In Proceedings of the 61st Annual Meeting of the Association for Co… find
  50. Qihui Zhang, Chujie Gao, Dongping Chen, Yue Huang, Yixin Huang, Zhenyang Sun, Shilin Zhang, Weiye Li, Zhengyan Fu, Yao Wan, et al. 2024. LLM-as-a-coauthor: Can mixed human-written and machine-generated text be detected? In Findings of the Association for Computational Linguistics: NAACL 2024, pages… find
  51. Albert Ziegler, Eirini Kalliamvakou, X Alice Li, Andrew Rice, Devon Rifkin, Shawn Simister, Ganesh Sittampalam, and Edward Aftandilian. 2024. Measuring github copilot’s impact on productivity. Communications of the ACM, 67(3):54– 63. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/patterson-2026-hybridcodeauthorship-benchmark-dataset-line-level/