Elamur Research⌕
☰
home / papers / 2026 / MindFlayer at SemEval-2026 Task 13:LACR-ENS: Calibration-Aware Ensemble Routing for Cross-Language AI-Generated Code Detection

MindFlayer at SemEval-2026 Task 13:LACR-ENS: Calibration-Aware Ensemble Routing for Cross-Language AI-Generated Code Detection

Paper · 2026 · Jerin Romijah Tuli, Talukder Naemul Hasan Naem and Md. Sartaj Alam Pritom

Summary

This system paper for SemEval-2026 Task 13 (Subtask A) addresses binary detection of human-written versus machine-generated code across eight programming languages. Only Python, Java and C++ have strong training coverage. The authors argue that the main difficulty is trustworthiness under language shift: fine-tuned encoders become overconfident on languages they have not seen. They fine-tune UniXCoder and GraphCodeBERT (about 125M parameters each) on AICD Bench and merge their pre-softmax logits with weights 0.6 and 0.4. On top of this ensemble they add Language-Aware Confidence Routing (LACR), which picks a decision threshold by language group and domain. The paper shows this thresholding is equivalent to implicit per-language temperature scaling, but it needs only a lookup table rather than calibration data for each language. LACR lowers calibration error on unseen languages while leaving F1 almost unchanged. The final system reaches macro-F1 0.538 and ranks 37th of 82. Two further contributions are a label-free risk signal based on vocabulary overlap with the training languages, and a documented HuggingFace pitfall that silently inverted labels.

Key points

  • Data: AICD Bench (HuggingFace DaniilOr/SemEval-2026-Task13) has 500K training, 100K validation and 1K test samples. It spans eight languages in total: C, C++, C#, Go, Java, JavaScript, PHP and Python.
  • Calibration results:
    • Out-of-distribution ECE is 0.18 versus 0.09 on seen languages, and LACR reduces it to 0.11.
    • At confidence above 0.80, accuracy on unseen languages is only 0.61, compared with 0.84 on seen ones.
  • Ablation on the test set (macro-F1):
    • Single models score 0.491 (UniXCoder) and 0.476 (GraphCodeBERT).
    • The ensemble reaches 0.525, rising to 0.536 with per-language temperature scaling and 0.538 with LACR.
    • LACR improved the score in 78% of 1,000 bootstrap resamples, though the gain is not formally significant.
  • Language proximity: Subword-overlap proximity to the seen languages correlates with OOD F1 (r=+0.94) and inversely with ECE (r=−0.89), based on only 5 languages. C, the closest language, performs best; Go and C# perform worst. The authors propose a continuous, proximity-weighted threshold as an extension of LACR.
  • Reproducibility pitfall: Without an explicit id2label mapping, the first submission scored 0.251 because its labels were inverted. Flipping the labels recovered 0.538. Encoder-only systems all stayed below 0.55, while three LoRA-tuned LLM systems exceeded 0.73.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Jerin Romijah Tuli, Talukder Naemul Hasan Naem and Md. Sartaj Alam Pritom. 2026. MindFlayer at SemEval-2026 Task 13:LACR-ENS: Calibration-Aware Ensemble Routing for Cross-Language AI-Generated Code Detection. Proceedings of the 20th International Workshop on Semantic Evaluation (2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.semeval-1.294

@inproceedings{tuli2026mindflayer,
  title = {{MindFlayer at SemEval-2026 Task 13:LACR-ENS: Calibration-Aware Ensemble Routing for Cross-Language AI-Generated Code Detection}},
  author = {Jerin Romijah Tuli, Talukder Naemul Hasan Naem and Md. Sartaj Alam Pritom},
  booktitle = {Proceedings of the 20th International Workshop on Semantic Evaluation (2026)},
  year = {2026},
  doi = {10.18653/v1/2026.semeval-1.294},
  url = {https://doi.org/10.18653/v1/2026.semeval-1.294}
}

References

13 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. We presented LACR-E NS as an empirical probe of OOD failure in AI-generated code detection. Cross-language shift induces a calibration failure that is partially addressable by post-hoc routing and a representation failure that is not; language-family proximity grounds the LACR partition in a measura… find
  2. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321– 1330. PMLR. find
  3. Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, and Jian Yin. 2022. UniXcoder: Unified cross-modal pre-training for code representation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7212–7225, Dublin, Ireland. Associa…
  4. Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Shujie Liu, Long Zhou, Nan Duan, Alexey Svyatkovskiy, Shengyu Fu, Michele Tufano, Shao Kun Deng, Colin Clement, Dawn Drain, Neel Sundaresan, Jian Yin, Daxin Jiang, and Ming Zhou. 2021. Graph-CodeBERT: Pre-training code representations with data… find
  5. Xiaomeng Hu, Pin-Yu Chen, and Tsung-Yi Ho. 2023. RADAR: Robust AI-text detection via adversarial learning. In Advances in Neural Information Processing Systems. find
  6. Daniil Orel, Dilshod Azizov, and Preslav Nakov. 2025a. CoDet-M4: Detecting machine-generated code in multi-lingual, multi-generator and multi-domain settings. In Findings of the Association for Computational Linguistics: ACL 2025, pages 10570–10593, Vienna, Austria. Association for Computational Lin… find
  7. Daniil Orel, Dilshod Azizov, Indraneil Paul, Yuxia Wang, Iryna Gurevych, and Preslav Nakov. 2026a. AICD bench: A challenging benchmark for AI-generated code detection. In Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Paper…
  8. Daniil Orel, Dilshod Azizov, Indraneil Paul, Yuxia Wang, Iryna Gurevych, and Preslav Nakov. 2026b. SemEval-2026 task 13: Detecting machine-generated code with multiple programming languages, generators, and application scenarios. In Proceedings of the 20th International Workshop on Semantic Evaluati…
  9. Daniil Orel, Indraneil Paul, Iryna Gurevych, and Preslav Nakov. 2025b. Droid: A resource suite for AI-generated code detection. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 31263–31289, Suzhou, China. Association for Computational Linguistics. find
  10. Claudio Di Sipio Riccardo Rubei Davide Di Ruscio Massimiliano Di Penta Phuong T. Nguyen, Juri Di Rocco. 2024. GPTSniffer: A binary classifier to detect ChatGPT-generated code. In Journal of Systems and Software. find
  11. Xiao Pu, Jingyu Zhang, and Yulia Tsvetkov. 2023. On the zero-shot generalization of machine-generated text detectors. arXiv preprint arXiv:2310.05165. find
  12. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yann Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Transfor…
  13. Xinrong Guo Shaoxuan Liu Xiaoya Wang Kui Liu Xiaohu Yang Xiaodan Xu, Chao Ni. 2025. Distinguishing LLM-generated from human-written code: A contrastive approach. In ACM Transactions on Software Engineering and Methodology. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tuli-2026-mindflayer-semeval-2026-task-13/