Elamur Research⌕
☰
home / papers / 2026 / Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition

Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition

Paper · 2026 · Qi Zhang, Fangping Lan, Cornelia Caragea et al.

Summary

NER systems built on fine-tuned small models such as BERT still perform best, but they need large amounts of human-labeled data. Prompting LLMs with only a few examples has so far fallen short of them. The authors test what happens when an LLM prompt contains hundreds of demonstrations instead of a handful. They frame NER as a generation task: the model rewrites each sentence with XML-like entity tags, which are then mapped to BIO labels for comparison with earlier systems. They evaluated seven open and closed LLMs on four English benchmarks covering news, social media, movies and restaurants. Demonstrations were picked either at random or by BM25 retrieval, up to 500 per prompt. With enough examples, the LLMs matched or outperformed BERT trained on the complete training sets. Building on this, the authors propose In-Context Annotation (ICA), in which about 100 human-labeled seeds guide an LLM to label unlabeled text. That output is cleaned up and used to train BERT. In low-resource settings, ICA beats prior state-of-the-art methods by roughly 10 F1 points, which makes it a cheap way to produce compact, deployable NER models.

Key points

  • Benchmarks and models: The scaling study uses CoNLL2003, WNUT 2017, MIT-Movie (12 entity types) and MIT-Restaurant (8 types). The models are Qwen-2.5 (7B/32B/72B), LLaMA-3.1, GPT-4o and DeepSeekV3, all with a context window of at least 32K tokens.
  • Scaling behavior: BM25-retrieved demonstrations improve results quickly and level off near 100 examples (around 50 for CoNLL2003). Random demonstrations improve more slowly, but by 500 shots they nearly catch up, while being much cheaper because one fixed set serves the whole test set.
  • Comparison with BERT: Many-shot ICL reached or exceeded BERT-FT trained on the full data in almost every setting. The one exception was LLaMA3.1-70B on MIT-Restaurant.
  • ICA pipeline: About 100 seed examples are used to label 2k unlabeled sentences. The labels are refined with self-consistency, self-correction and error-aware self-refinement (ERA), and the result is used to fine-tune BERT. Tested on five low-resource datasets, this gives an average gain of about 10 F1 points.
  • Evaluation and limitations: Performance is measured as micro-averaged F1 over entities, counting a prediction as correct only when its span matches exactly, computed with seqeval. A BERT-FT (target-only) baseline trained on 100–200 samples serves as the lower bound. All experiments cover English only, and repeated LLM calls add compute cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois ChicagoChicago · United States70 works · 20,363 citations🎓Temple UniversityPhiladelphia · United States11 works · 24,319 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,894 worksNamed entity recognition · 214 worksLow-resource languages · 287 works

Cite

Cite as Qi Zhang, Fangping Lan, Cornelia Caragea, Longin Jan Latecki and Eduard Dragut. 2026. Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1431

@inproceedings{zhang2026scaling,
  title = {{Scaling Performance and Low-Resource Annotation with Many-Shot In-Context Learning for Named Entity Recognition}},
  author = {Qi Zhang, Fangping Lan, Cornelia Caragea, Longin Jan Latecki and Eduard Dragut},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1431},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1431}
}

References

67 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rishabh Agarwal, Avi Singh, Lei Zhang, Bernd Bohnet, Luis Rosias, Stephanie Chan, Biao Zhang, Ankesh Anand, Zaheer Abbas, Azade Nova, and 1 others. 2024. Many-shot in-context learning. Advances in Neural Information Processing Systems, 37:76930– 76966.
  2. Dhananjay Ashok and Zachary C Lipton. 2023. Promptner: Prompting for named entity recognition. arXiv preprint arXiv:2305.15444.
  3. Amanda Bertsch, Maor Ivgi, Emily Xiao, Uri Alon, Jonathan Berant, Matthew R. Gormley, and Graham Neubig. 2025. In-context learning with long-context models: An in-depth exploration. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Ling…
  4. Sergei Bogdanov, Alexandre Constantin, Timothée Bernard, Benoit Crabbé, and Etienne P Bernard. 2024. NuNER: Entity recognition encoder pre-training via LLM-annotated data. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 11829–11841, Miami, Florida, US… find
  5. Jiong Cai, Shen Huang, Yong Jiang, Zeqi Tan, Pengjun Xie, and Kewei Tu. 2023. Improving low-resource named entity recognition with graph propagated data augmentation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 110–118, T… find
  6. Jiuhai Chen, Lichang Chen, Chen Zhu, and Tianyi Zhou. 2023. How many demonstrations do you need for in-context learning? In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 11149–11159, Singapore. Association for Computational Linguistics. find
  7. Shivani Choudhary, Niladri Chatterjee, and Subir Saha. 2023. IITD at SemEval-2023 task 2: A multi-stage information retrieval approach for fine-grained named entity recognition. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 800–806, Toronto, Canada. A… find
  8. Xiang Dai and Heike Adel. 2020. An analysis of simple data augmentation for named entity recognition. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3861–3867, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  9. DeepSeek. 2024. Introducing deepseek-v3. find
  10. Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2024. Investigating data contamination in modern benchmarks for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol…
  11. Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140–147, Copenhagen, Denmark. Association for Computational Linguistics.
  12. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL, pages 4171–4186.
  13. Bosheng Ding, Linlin Liu, Lidong Bing, Canasai Kruengkrai, Thien Hai Nguyen, Shafiq Joty, Luo Si, and Chunyan Miao. 2020. DAGA: Data augmentation with a generation approach for low-resource tagging tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNL…
  14. Bosheng Ding, Chengwei Qin, Linlin Liu, Yew Ken Chia, Boyang Li, Shafiq Joty, and Lidong Bing. 2023. Is GPT-3 a good data annotator? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11173–11195, Toronto, Canada. Association for…
  15. Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Haitao Zheng, and Zhiyuan Liu. 2021. Few-NERD: A few-shot named entity recognition dataset. In ACL, pages 3198–3213.
  16. Yuyang Ding, Juntao Li, Pinzheng Wang, Zecheng Tang, Yan Bowen, and Min Zhang. 2024. Rethinking negative instances for generative named entity recognition. In Findings of the Association for Computational Linguistics: ACL 2024, pages 3461–3475, Bangkok, Thailand. Association for Computational Lingui… find
  17. Hao Fei, Shengqiong Wu, Jingye Li, Bobo Li, Fei Li, Libo Qin, Meishan Zhang, Min Zhang, and Tat-Seng Chua. 2022. Lasuie: Unifying information extraction with latent adaptive structure-aware generative language model. Advances in Neural Information Processing Systems, 35:15460–15475. find
  18. Yao Fu, Rameswar Panda, Xinyao Niu, Xiang Yue, Hannaneh Hajishirzi, Yoon Kim, and Hao Peng. 2024. Data engineering for scaling language models to 128k context. In Proceedings of the 41st International Conference on Machine Learning, pages 14125–14134.
  19. Tianyu Gao, Alexander Wettig, Howard Yen, and Danqi Chen. 2024. How to train long-context language models (effectively). arXiv preprint arXiv:2410.02660.
  20. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  21. Yifan Hao, Xingyuan Pan, Hanning Zhang, Chenlu Ye, Rui Pan, and Tong Zhang. 2025. Understanding over-adaptation in supervised fine-tuning: The role of ensemble methods. arXiv preprint arXiv:2506.01901. — [arXiv:2506.01901](https://arxiv.org/abs/2506.01901) find
  22. Yuzhao Heng, Chunyuan Deng, Yitong Li, Yue Yu, Yinghao Li, Rongzhi Zhang, and Chao Zhang. 2024. ProgGen: Generating named entity recognition datasets step-by-step with self-reflexive large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15992– 16030, Ba… find
  23. Clyde Highmore. 2024. In-context learning in large language models: A comprehensive survey. Preprints, 202407:v1. find
  24. Jinpeng Hu, He Zhao, Dan Guo, Xiang Wan, and Tsung-Hui Chang. 2022. A label-aware autoregressive framework for cross-domain NER. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2222–2232, Seattle, United States. Association for Computational Linguistics. find
  25. Xuming Hu, Zhaochen Hong, Yong Jiang, Zhichao Lin, Xiaobin Wang, Pengjun Xie, and Philip S Yu. 2024a. Three heads are better than one: improving cross-domain ner with progressive decomposed network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18261–18269. find
  26. Yan Hu, Qingyu Chen, Jingcheng Du, Xueqing Peng, Vipina Kuttichi Keloth, Xu Zuo, Yujia Zhou, Zehan Li, Xiaoqian Jiang, Zhiyong Lu, and 1 others. 2024b. Improving large language models for clinical named entity recognition via prompt engineering. Journal of the American Medical Informatics Associatio… find
  27. Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet. arXiv preprint arXiv:2310.01798.
  28. Bernal Jimenez Gutierrez, Nikolas McNeal, Clayton Washington, You Chen, Lang Li, Huan Sun, and Yu Su. 2022. Thinking about GPT-3 in-context learning for biomedical IE? think again. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4497–4512, Abu Dhabi, United Arab Emira…
  29. Dan Jurafsky. 2000. Speech & language processing. Pearson Education India. find
  30. Ryo Kamoi, Yusen Zhang, Nan Zhang, Jiawei Han, and Rui Zhang. 2024. When can LLMs actually correct their own mistakes? a critical survey of self-correction of LLMs. Transactions of the Association for Computational Linguistics, 12:1417–1440.
  31. Jingjing Liu, Panupong Pasupat, Scott Cyphers, and Jim Glass. 2013. Asgard: A portable architecture for multilingual dialogue systems. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, pages 8386–8390. IEEE. find
  32. Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2021. Crossner: Evaluating cross-domain named entity recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 13452–13460. find
  33. Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Unified structure generation for universal information extraction. arXiv preprint arXiv:2203.12277. find
  34. Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, and 1 others. 2023. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36:46534–46594. find
  35. Stephen Mayhew, Snigdha Chaturvedi, Chen-Tse Tsai, and Dan Roth. 2019. Named entity recognition with partially annotated training data. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 645–655, Hong Kong, China. Association for Computational Linguistics… find
  36. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Ab…
  37. Subhadip Nandi and Neeraj Agrawal. 2024. Improving few-shot cross-domain named entity recognition by instruction tuning a word-embedding based retrieval augmented large language model. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 68… find
  38. Huitong Pan, Mustapha Adamu, Qi Zhang, Eduard Dragut, and Longin Jan Latecki. 2025a. ClimateIE: A dataset for climate science information extraction. In Proceedings of the 2nd Workshop on Natural Language Processing Meets Climate Change (ClimateNLP 2025), pages 76–98, Vienna, Austria. Association fo… find
  39. Huitong Pan, Qi Zhang, Mustapha Adamu, Eduard Dragut, and Longin Jan Latecki. 2025b. Taxonomy-driven knowledge graph construction for domain-specific scientific applications. In Findings of the Association for Computational Linguistics: ACL 2025, pages 4295–4320, Vienna, Austria. Association for Com… find
  40. Huitong Pan, Qi Zhang, Cornelia Caragea, Eduard Dragut, and Longin Jan Latecki. 2024a. SciDMT: A large-scale corpus for detecting scientific mentions. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1… find
  41. Huitong Pan, Qi Zhang, Eduard Dragut, Cornelia Caragea, and Longin Jan Latecki. 2023. DMDD: A large-scale dataset for dataset mentions detection. Transactions of the Association for Computational Linguistics, 11:1132–1146. find
  42. Liangming Pan, Michael Saxon, Wenda Xu, Deepak Nathani, Xinyi Wang, and William Yang Wang. 2024b. Automatically correcting large language models: Surveying the landscape of diverse automated correction strategies. Transactions of the Association for Computational Linguistics, 12:484–506.
  43. Reid Pryzant, Ziyi Yang, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2022. Automatic rule induction for efficient semi-supervised learning. In Findings of the EMNLP, pages 28–44. find
  44. Esteban Safranchik, Shiying Luo, and Stephen Bach. 2020. Weakly supervised sequence tagging from noisy rules. Proceedings of the AAAI Conference on Artificial Intelligence, 34(04):5570–5578. find
  45. Oscar Sainz, Iker García-Ferrero, Rodrigo Agerri, Oier Lopez de Lacalle, German Rigau, and Eneko Agirre. 2023. Gollie: Annotation guidelines improve zero-shot information-extraction. arXiv preprint arXiv:2310.03668.
  46. Jingbo Shang, Liyuan Liu, Xiaotao Gu, Xiang Ren, Teng Ren, and Jiawei Han. 2018. Learning named entity tagger using domain-specific dictionary. In EMNLP, pages 2054–2064.
  47. Amanpreet Singh, Mike D’Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. 2023. SciRepEval: A multi-format benchmark for scientific document representations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5548–5566, Singapore. Association for Compu… find
  48. Mingyang Song, Mao Zheng, and Xuan Luo. 2025. Can many-shot in-context learning help LLMs as evaluators? a preliminary empirical study. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8232–8241, Abu Dhabi, UAE. Association for Computational Linguistics.
  49. Mo Sun, Siheng Xiong, Yuankai Cai, and Bowen Zuo. 2025. Positional attention for efficient bert-based named entity recognition. arxiv preprint arXiv:2505.01868. — [arXiv:2505.01868](https://arxiv.org/abs/2505.01868) find
  50. Minghao Tang, Peng Zhang, Yongquan He, Yongxiu Xu, Chengpeng Chao, and Hongbo Xu. 2022. DoSEA: A domain-specific entity-aware framework for cross-domain named entity recogition. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2147–2156, Gyeongju, Republic of K… find
  51. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142– 147.
  52. Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, Guoyin Wang, and Chen Guo. 2025. GPT-NER: Named entity recognition via large language models. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 4257–4275, Albuquerque, New Mexico. Assoc…
  53. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Ed H Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations.
  54. Xiang Wei, Yufeng Chen, Ning Cheng, Xingyu Cui, Jinan Xu, and Wenjuan Han. 2024. CollabKG: A learnable human-machine-cooperative information extraction toolkit for (event) knowledge graph construction. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language R… find
  55. Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang, Zuozhu Liu, and Hongwei Wang. 2023. Empirical study of zero-shot NER with ChatGPT. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7935–7956, Singapore. Association for Computational Linguistics. find
  56. Tingyu Xie, Qi Li, Yan Zhang, Zuozhu Liu, and Hongwei Wang. 2024. Self-improving for zero-shot named entity recognition with large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volum… find
  57. Jingyun Xu and Yi Cai. 2023. Decoupled hyperbolic graph attention network for cross-domain named entity recognition. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 591– 600. find
  58. Zikang Zhang, Wangjie You, Tianci Wu, Xinrui Wang, Juntao Li, and Min Zhang. 2025b. A survey of generative information extraction. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4840–4870, Abu Dhabi, UAE. Association for Computational Linguistics. find
  59. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025a. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  60. Junhao Zheng, Haibin Chen, and Qianli Ma. 2022. Cross-domain named entity recognition via graph matching. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2670–2680, Dublin, Ireland. Association for Computational Linguistics. find
  61. Yuming Yang, Wantong Zhao, Caishuang Huang, Junjie Ye, Xiao Wang, Huiyuan Zheng, Yang Nan, Yuran Wang, Xueying Xu, Kaixin Huang, Yunke Zhang, Tao Gui, Qi Zhang, and Xuanjing Huang. 2025b. Beyond boundaries: Learning a universal entity taxonomy across datasets and languages for open named entity reco…
  62. Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. Universalner: Targeted distillation from large language models for open named entity recognition. In The Twelfth International Conference on Learning Representations.
  63. Urchade Zaratiana, Nadi Tomeh, Pierre Holat, and Thierry Charnois. 2024. GLiNER: Generalist model for named entity recognition using bidirectional transformer. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol…
  64. Qi Zhang, Zhijia Chen, Huitong Pan, Cornelia Caragea, Longin Jan Latecki, and Eduard Dragut. 2024a. SciER: An entity and relation extraction dataset for datasets, methods, and tasks in scientific documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pa… find
  65. Qi Zhang, Huitong Pan, Zhijia Chen, Longin Jan Latecki, Cornelia Caragea, and Eduard Dragut. 2025a. DynClean: Training dynamics-based label cleaning for distantly-supervised named entity recognition. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2540–2556, Albuquerq… find
  66. Xinghua Zhang, Bowen Yu, Xin Cong, Taoyu Su, Quangang Li, Tingwen Liu, and Hongbo Xu. 2024b. Cross-domain ner under a divide-and-transfer paradigm. ACM Transactions on Information Systems, 42(5):1–32. find
  67. Xinghua Zhang, Bowen Yu, Yubin Wang, Tingwen Liu, Taoyu Su, and Hongbo Xu. 2022. Exploring modular task decomposition in cross-domain named entity find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-scaling-performance-low-resource-annotation/