Elamur Research⌕
☰
home / papers / 2026 / Learning on Imbalanced Noisy Data via Debiased Sample Selection and LLM-Driven Annotation

Learning on Imbalanced Noisy Data via Debiased Sample Selection and LLM-Driven Annotation

Paper · 2026 · Bo Yuan, Yulin Chen and Yin Zhang

Summary

Text classification data gathered through web crawling or crowdsourcing often has wrong labels, and real collections are usually class-imbalanced too. The authors show that common small-loss sample selection, which uses one fixed global threshold, fails in this setting. Tail-class samples have high losses and get discarded (selection bias), and self-training lets the model reinforce its own errors (training bias). They propose DeCo, an active-learning framework that pairs small language models (SLMs) with LLMs. The SLM side has two parts:

  • A Multiple Expert Model (MEM) adds several expert layers to the classifier, so samples are selected on layers separate from the one being trained.
  • A dynamic class-wise threshold, based on momentum-smoothed confidence and set per class, decides which samples count as clean.

Clean examples become in-context demonstrations for an LLM, which relabels the suspected noisy ones. The corrected labels feed back into SLM training, and each subset is trained with its own loss. On synthetic and real-world noisy benchmarks, DeCo beats existing LNL methods without needing human re-annotation.

Key points

  • The SLM splits the data into a clean set, a hard set and a noisy set. Only the noisy set is sent to the LLM (GPT-3.5-Turbo; GPT-4-0613 is also mentioned) for ICL-based relabeling.
  • The per-sample dynamic threshold follows τ(t;x) = λ·p(t;x) + (1−λ)·τ(t−1;x), where p(t;x) is the sample's maximum confidence at epoch t. The threshold therefore rises as the model memorizes more.
  • On R8 with 40% asymmetric noise, DeCo outperforms BERT, CoTeaching, CL, CR, NPC, SelfMix and DyGen, and comes close to training on ground-truth labels. SelfMix gradually drops class 2 and never picks clean class-7 samples, while DeCo covers every class.
  • On Trec with 20% asymmetric noise, only 264 of 1108 noisy-set labels are correct before LLM querying. After relabeling at epoch 3, label accuracy in the noisy set is 83.84%, and it reaches 99.59% by epoch 6.
  • The main stated limitation is dependence on LLM annotation quality, which may be insufficient in very difficult or unusual domains.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China224 works · 13,587 citations🎓National University of Defense TechnologyChangsha · China20 works · 1,339 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Bo Yuan, Yulin Chen and Yin Zhang. 2026. Learning on Imbalanced Noisy Data via Debiased Sample Selection and LLM-Driven Annotation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1526

@inproceedings{yuan2026learning,
  title = {{Learning on Imbalanced Noisy Data via Debiased Sample Selection and LLM-Driven Annotation}},
  author = {Bo Yuan, Yulin Chen and Yin Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1526},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1526}
}

References

54 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Görkem Algan and Ilkay Ulusoy. 2020. Label noise types and their effects on deep learning. CoRR, abs/2003.10471. — [arXiv:2003.10471](https://arxiv.org/abs/2003.10471) find
  2. Abhijeet Awasthi, Sabyasachi Ghosh, Rasna Goyal, and Sunita Sarawagi. 2020. Learning from rules generalizing labeled exemplars. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  3. HeeSun Bae, Seungjae Shin, Byeonghu Na, JoonHo Jang, Kyungwoo Song, and Il-Chul Moon. 2022. From noisy prediction to true label: Noisy prediction calibration via generative model. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Pro… find
  4. Ruichu Cai, Zhenjie Zhang, Anthony K. H. Tung, Chenyun Dai, and Zhifeng Hao. 2014. A general framework of hierarchical clustering and its applications. Inf. Sci., 272:29–48. find
  5. Duo Chai, Wei Wu, Qinghong Han, Fei Wu, and Jiwei Li. 2020. Description based text classification with reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, p… find
  6. Pengfei Chen, Benben Liao, Guangyong Chen, and Shengyu Zhang. 2019. Understanding and utilizing deep neural networks trained with noisy labels. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of… find
  7. PMLR. Chen Cheng, Xinzhi Yu, Haodong Wen, Jingsong Sun, Guanzhang Yue, Yihao Zhang, and Zeming Wei. 2024. Exploring the robustness of in-context learning with noisy labels. CoRR, abs/2404.18191. — [arXiv:2404.18191](https://arxiv.org/abs/2404.18191) find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  9. Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. 2023. Mitigating label biases for in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 14014–14031. Association…
  10. Hongfu Gao, Feipeng Zhang, Wenyu Jiang, Jun Shu, Feng Zheng, and Hongxin Wei. 2024. On the noise robustness of in-context learning for text generation. Preprint, arXiv:2405.17264. — [arXiv:2405.17264](https://arxiv.org/abs/2405.17264) find
  11. Jiahui Gao, Renjie Pi, Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. 2023a. Self-guided noise-free data generation for efficient zero-shot learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rw… find
  12. OpenReview.net. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023b. Retrieval-augmented generation for large language models: A survey. CoRR, abs/2312.10997.
  13. Aritra Ghosh, Himanshu Kumar, and P. S. Sastry. 2017. Robust loss functions under label noise for deep neural networks. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pages 1919– 1925. find
  14. AAAI Press. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd-workers for text-annotation tasks. CoRR, abs/2303.15056.
  15. Antonio Gulli. 2005. The anatomy of a news search engine. In Proceedings of the 14th international conference on World Wide Web, WWW 2005, Chiba, Japan, May 10-14, 2005 - Special interest tracks and posters, pages 880–881. find
  16. ACM. Bo Han, Quanming Yao, Xingrui Yu, Gang Niu, Miao Xu, Weihua Hu, Ivor W. Tsang, and Masashi Sugiyama. 2018. Co-teaching: Robust training of deep neural networks with extremely noisy labels. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processin… find
  17. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In Proceedings of the 5th International Wo…
  18. Yingsong Huang, Bing Bai, Shengwei Zhao, Kun Bai, and Fei Wang. 2022. Uncertainty-aware learning against label noise on imbalanced datasets. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 202… find
  19. Shenwang Jiang, Jianan Li, Ying Wang, Bo Huang, Zhang Zhang, and Tingfa Xu. 2022. Delving into sample loss curve to embrace noisy and imbalanced data. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence… find
  20. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  21. Jannik Kossen, Tom Rainforth, and Yarin Gal. 2023. In-context learning in large language models learns label relationships but is not conventional learning. CoRR, abs/2307.12375. — [arXiv:2307.12375](https://arxiv.org/abs/2307.12375) find
  22. Martin Krallinger, Obdulia Rabal, Saber A Akhondi, Martın Pérez Pérez, Jesús Santamaría, Gael Pérez Rodríguez, Georgios Tsatsaronis, Ander Intxaurrondo, José Antonio López, Umesh Nandal, et al. 2017. Overview of the biocreative vi chemical-protein interaction track. In Proceedings of the sixth BioCr… find
  23. Ken Lang. 1995. Newsweeder: Learning to filter netnews. In Machine Learning, Proceedings of the Twelfth International Conference on Machine Learning, Tahoe City, California, USA, July 9-12, 1995, pages 331–339. Morgan Kaufmann. find
  24. David D. Lewis. 1992. An evaluation of phrasal and clustered representations on a text categorization task. In Proceedings of the 15th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. Copenhagen, Denmark, June 21-24, 1992, pages 37–50. ACM. find
  25. Junnan Li, Richard Socher, and Steven C. H. Hoi. 2020. Dividemix: Learning with noisy labels as semi-supervised learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  26. Wen Li, Limin Wang, Wei Li, Eirikur Agustsson, and Luc Van Gool. 2017. Webvision database: Visual learning and understanding from web data. CoRR, abs/1708.02862. — [arXiv:1708.02862](https://arxiv.org/abs/1708.02862) find
  27. Xin Li and Dan Roth. 2002. Learning question classifiers. In 19th International Conference on Computational Linguistics, COLING 2002, Howard International House and Academia Sinica, Taipei, Taiwan, August 24 - September 1, 2002.
  28. Yifan Li, Hu Han, Shiguang Shan, and Xilin Chen. 2023. DISC: learning from noisy labels via dynamic instance-specific selection and correction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC, Canada, June 17-24, 2023, pages 24070–24079. IEEE. find
  29. Huafeng Liu, Mengmeng Sheng, Zeren Sun, Yazhou Yao, Xian-Sheng Hua, and Heng Tao Shen. 2024. Learning with imbalanced noisy data by preventing bias in sample selection. IEEE Trans. Multim., 26:7426–7437. find
  30. Danqing Luo, Chen Zhang, Yan Zhang, and Haizhou Li. 2024. Crosstune: Black-box few-shot classification with label enhancement. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC/COLING 2024, 20-25 May, 2024, Torino, Italy,… find
  31. Xingjun Ma, Hanxun Huang, Yisen Wang, Simone Romano, Sarah M. Erfani, and James Bailey. 2020. Normalized loss functions for deep learning with noisy labels. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings o… find
  32. Curtis G. Northcutt, Lu Jiang, and Isaac L. Chuang. 2021. Confident learning: Estimating uncertainty in dataset labels. J. Artif. Intell. Res., 70:1373–1411. find
  33. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. Cornell University -arXiv,Cornell University - arXiv.
  34. Dan Qiao, Chenchen Dai, Yuyang Ding, Juntao Li, Qiang Chen, Wenliang Chen, and Min Zhang. 2022. Selfmix: Robust learning against textual label noise with self-mixup training. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, O… find
  35. Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. 2018. Learning to reweight examples for robust deep learning. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learnin… find
  36. PMLR. Lior Rokach. 2010. Ensemble-based classifiers. Artificial Intelligence Review, page 1–39. find
  37. Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. 2019. Meta-weightnet: Learning an explicit mapping for sample weighting. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-… find
  38. Antti Tarvainen and Harri Valpola. 2017. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 20… find
  39. Brendan van Rooyen, Aditya Krishna Menon, and Robert C. Williamson. 2015. Learning with symmetric label noise: The importance of being unhinged. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Qu… find
  40. Song Wang, Zhen Tan, Ruocheng Guo, and Jundong Li. 2023. Noise-robust fine-tuning of pretrained language models via external guidance. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, December 6-10, 2023, pages 12528–12540. Association for Computational Linguistic… find
  41. Qi Wei, Lei Feng, Haobo Wang, and Bo An. 2024. Debiased sample selection for combating noisy labels. CoRR, abs/2401.13360. — [arXiv:2401.13360](https://arxiv.org/abs/2401.13360) find
  42. Ruixuan Xiao, Yiwen Dong, Junbo Zhao, Runze Wu, Minmin Lin, Gang Chen, and Haobo Wang. 2023. Freeal: Towards human-free active learning in the era of large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10… find
  43. Yilun Xu, Peng Cao, Yuqing Kong, and Yizhou Wang. 2019. L_dmi: A novel information-theoretic loss function for training deep nets robust to label noise. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-… find
  44. Yan Yan, Rómer Rosales, Glenn Fung, Subramanian Ramanathan, and Jennifer G. Dy. 2014. Learning from multiple annotators with varying expertise. Mach. Learn., 95(3):291–327. find
  45. Taraneh Younesian, Zilong Zhao, Amirmasoud Ghiassi, Robert Birke, and Lydia Y. Chen. 2021. Qactor: Active learning on noisy labels. In Asian Conference on Machine Learning, ACML 2021, 17-19 November 2021, Virtual Event, volume 157 of Proceedings of Machine Learning Research, pages 548–563. PMLR. find
  46. Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. 2018. mixup: Beyond empirical risk minimization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings. OpenReview.net.
  47. Yivan Zhang, Gang Niu, and Masashi Sugiyama. 2021. Learning noise transition matrix from only noisy labels via total variation regularization. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Lear… find
  48. Zhilu Zhang and Mert R. Sabuncu. 2018. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada, pag… find
  49. Liyue Zhao, Gita Sukthankar, and Rahul Sukthankar. 2011. Incremental relabeling for active learning with noisy crowdsourced annotations. In PASSAT/Social-Com 2011, Privacy, Security, Risk and Trust (PAS-SAT), 2011 IEEE Third International Conference on and 2011 IEEE Third International Conference on… find
  50. Wenxuan Zhou and Muhao Chen. 2021. Learning from noisy labels for entity-centric information extraction. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 5381–5392. Associ… find
  51. Wenxuan Zhou, Hongtao Lin, Bill Yuchen Lin, Ziqi Wang, Junyi Du, Leonardo Neves, and Xiang Ren. 2020. NERO: A neural rule grounding framework for label-efficient relation extraction. In WWW ’20: The Web Conference 2020, Taipei, Taiwan, April 20-24, 2020, pages 2166–2176. ACM / IW3C2. find
  52. Dawei Zhu, Michael A. Hedderich, Fangzhou Zhai, David Ifeoluwa Adelani, and Dietrich Klakow. 2022. Is BERT robust to label noise? A study on learning with noisy labels in text classification. In Proceedings of the Third Workshop on Insights from Negative Results in NLP, Insights@ACL 2022, Dublin, Ir… find
  53. Yuchen Zhuang, Yue Yu, Lingkai Kong, Xiang Chen, and Chao Zhang. 2023. Dygen: Learning from noisy labels via dynamics-enhanced generative modeling. CoRR, abs/2305.19395. — [arXiv:2305.19395](https://arxiv.org/abs/2305.19395) find
  54. The loss distribution of clean samples and noisy samples, whether in the head class or tail class, dynamically evolves during the training process; • Both tail class samples and noisy samples exhibit large losses; find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yuan-2026-learning-imbalanced-noisy-data-debiased/