Elamur Research⌕
☰
home / papers / 2023 / PromptNER: Prompting For Named Entity Recognition

PromptNER: Prompting For Named Entity Recognition

Paper · 2023 · Dhananjay Ashok and Zachary C. Lipton

Summary

Named Entity Recognition (NER) systems usually need large amounts of in-domain labelled data and a fixed notion of what an entity is. LLM prompting has transformed few-shot learning on many NLP tasks, but it had lagged behind specialised methods on NER. The authors present PromptNER, which requires no parameter updates. It frames NER as conditional generation with a backbone LLM (T5XXL, GPT-3.5 or GPT-4) and gives the model three things: a swappable natural-language definition of which entity types count and which do not, a few target-domain examples, and a structured output template. In that template, each line names a candidate phrase, states whether it is an entity, gives its type and adds a short justification, similar to chain-of-thought reasoning. Adapting to a new domain only means editing the definition and the examples. With GPT-4, PromptNER sets new few-shot results on CoNLL, GENIA and FewNERD. On CrossNER, it beats cross-domain methods on 3 of 5 target domains while using about 1–2% of the target data those methods use. This makes customised NER feasible for users who have very little annotated data.

Key points

  • Benchmark results (GPT-4): PromptNER reaches 83.48 micro-F1 on few-shot CoNLL, about 4 points above prior methods. On GENIA it scores 58.44, against 49.58 for the best baseline (PACL). On FewNERD-Intra it gains about 4 points.
  • CrossNER (CoNLL as source): PromptNER uses only 2 target examples, while competitors such as CP-NER and LANER use 100–200. It is best on Politics (78.61), Literature (74.44) and Music (84.26), but trails on AI and Sciences.
  • Ablation on GPT-4: The full setup outperforms plain few-shot prompting and standard chain-of-thought prompting. Removing the definitions, the examples, the explanations or the candidate listing generally lowers scores.
  • Human study of disagreements: The authors sampled 20 disagreeing examples per CrossNER domain, each judged by 3 of 10 annotators. Many of the "errors" were rated as acceptable alternatives. In Politics, annotators preferred PromptNER's output over the gold labels in 60% of cases. This suggests the reported F1 scores underestimate true performance.
  • Limitations:
    • The LLM output carries no span information, so entities must be recovered with hand-crafted parsing, which struggles with repeated phrases.
    • Training-data contamination cannot be ruled out.
    • The generated explanations are not a reliable source of interpretability.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 1,343 worksNamed entity recognition · 101 works

Cite

Cite as Dhananjay Ashok and Zachary C. Lipton. 2023. PromptNER: Prompting For Named Entity Recognition. arXiv:2305.15444.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.15444 · arXiv 2305.15444

@misc{ashok2023promptner,
  title = {{PromptNER: Prompting For Named Entity Recognition}},
  author = {Dhananjay Ashok and Zachary C. Lipton},
  year = {2023},
  doi = {10.48550/arxiv.2305.15444},
  eprint = {2305.15444},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.15444}
}

References

51 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sajid Ali, Khalid Masood, Anas Riaz, and Amna Saud. 2022. Named entity recognition using deep learning: A review. In 2022 International Conference on Business Analytics for Technology and Security (ICBATS), pages 1–7. IEEE. find
  2. Andrew Borthwick, John Sterling, Eugene Agichtein, and Ralph Grishman. 1998. Exploiting diverse knowledge sources via maximum entropy in named entity recognition. In Sixth Workshop on Very Large Corpora. find
  3. Andrew Eliot Borthwick. 1999. A maximum entropy approach to named entity recognition. New York University. find
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  5. Xiang Chen, Lei Li, Qiaoshuo Fei, Ningyu Zhang, Chuanqi Tan, Yong Jiang, Fei Huang, and Huajun Chen. 2023. One model for all domains: Collaborative domain-prefix tuning for cross-domain ner. arXiv preprint arXiv:2301.10410. — [arXiv:2301.10410](https://arxiv.org/abs/2301.10410) find
  6. Yanru Chen, Yanan Zheng, and Zhilin Yang. 2022. Prompt-based metric learning for few-shot ner. arXiv preprint arXiv:2211.04337. find
  7. Nancy Chinchor. 1995. Muc-6 named entity task definition (version 2.1). In 6th Message Understanding Conference, Columbia, Maryland. find
  8. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
  9. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.
  10. Kenneth Ward Church, Zeyu Chen, and Yanjun Ma. 2021. Emerging trends: A gentle introduction to fine-tuning. Natural Language Engineering, 27(6):763– 778. find
  11. Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. 2021. Template-based named entity recognition using bart. arXiv preprint arXiv:2106.01760.
  12. Sarkar Snigdha Sarathi Das, Arzoo Katiyar, Rebecca J Passonneau, and Rui Zhang. 2022. Container: Few-shot named entity recognition via contrastive learning. In ACL.
  13. Wim De Mulder, Steven Bethard, and Marie-Francine Moens. 2015. A survey on the application of recurrent neural networks to statistical language modeling. Computer Speech & Language, 30(1):61–98. find
  14. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2023. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, pages 1–16.
  15. Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Hai-Tao Zheng, and Zhiyuan Liu. 2021. Few-nerd: A few-shot named entity recognition dataset. arXiv preprint arXiv:2105.07464.
  16. Tome Eftimov, Barbara Koroušić Seljak, and Peter Korošec. 2017. A rule-based named-entity recognition method for knowledge extraction of evidence-based dietary recommendations. PloS one, 12(6):e0179488. find
  17. Dimitra Farmakiotou, Vangelis Karkaletsis, John Kout-sias, George Sigletos, Constantine D Spyropoulos, and Panagiotis Stamatopoulos. 2000. Rule-based named entity recognition for greek financial texts. In Proceedings of the Workshop on Computational lexicography and Multimedia Dictionaries (COMLEX 2… find
  18. Jinpeng Hu, He Zhao, Dan Guo, Xiang Wan, and Tsung-Hui Chang. 2022a. A label-aware autoregressive framework for cross-domain ner. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 2222–2232. find
  19. Xuming Hu, Yong Jiang, Aiwei Liu, Zhongqiang Huang, Pengjun Xie, Fei Huang, Lijie Wen, and Philip S Yu. 2022b. Entda: Entity-to-text based data augmentation approach for named entity recognition tasks. arXiv preprint arXiv:2210.10343. — [arXiv:2210.10343](https://arxiv.org/abs/2210.10343) find
  20. Yucheng Huang, Kai He, Yige Wang, Xianli Zhang, Tieliang Gong, Rui Mao, and Chen Li. 2022a. Copner: Contrastive learning with prompt guiding for few-shot named entity recognition. In Proceedings of the 29th International conference on computational linguistics, pages 2515–2527. find
  21. Yucheng Huang, Kai He, Yige Wang, Xianli Zhang, Tieliang Gong, Rui Mao, and Chen Li. 2022b. COP-NER: Contrastive learning with prompt guiding for few-shot named entity recognition. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2515–2527, Gyeongju, Republic o… find
  22. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
  23. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361.
  24. J-D Kim, Tomoko Ohta, Yuka Tateisi, and Jun’ichi Tsujii. 2003. Genia corpus—a semantically annotated corpus for bio-textmining. Bioinformatics, 19(suppl_1):i180–i182. find
  25. Dong-Ho Lee, Akshen Kadakia, Kangmin Tan, Mahak Agarwal, Xinyu Feng, Takashi Shibuya, Ryosuke Mitani, Toshiyuki Sekiya, Jay Pujara, and Xiang Ren. 2021. Good examples make a faster learner: Simple demonstration-based learning for low-resource ner. arXiv preprint arXiv:2110.08454. find
  26. Jingye Li, Hao Fei, Jiang Liu, Shengqiong Wu, Meishan Zhang, Chong Teng, Donghong Ji, and Fei Li. 2022. Unified named entity recognition as word-word relation classification. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 10965–10973. find
  27. Nut Limsopatham and Nigel Henry Collier. 2016. Bidirectional lstm for named entity recognition in twitter messages. find
  28. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35.
  29. Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2021. Crossner: Evaluating cross-domain named entity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13452–13460. find
  30. Jinghui Lu, Rui Zhao, Brian Mac Namee, and Fei Tan. 2022a. Punifiedner: a prompting-based unified ner system for diverse datasets. arXiv preprint arXiv:2211.14838. — [arXiv:2211.14838](https://arxiv.org/abs/2211.14838) find
  31. Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022b. Unified structure generation for universal information extraction. arXiv preprint arXiv:2203.12277. find
  32. Ruotian Ma, Xin Zhou, Tao Gui, Yiding Tan, Qi Zhang, and Xuanjing Huang. 2021. Template-free prompt tuning for few-shot ner. arXiv preprint arXiv:2109.13532.
  33. Tingting Ma, Huiqiang Jiang, Qianhui Wu, Tiejun Zhao, and Chin-Yew Lin. 2022. Decomposed meta-learning for few-shot named entity recognition. arXiv preprint arXiv:2204.05751. find
  34. Takaki Makino, Yoshihiro Ohta, Jun’ichi Tsujii, et al. 2002. Tuning support vector machines for biomedical named entity recognition. In Proceedings of the ACL-02 workshop on Natural language processing in the biomedical domain, pages 1–8. find
  35. James Mayfield, Paul McNamee, and Christine Piatko. 2003. Named entity recognition using hundreds of thousands of features. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003, pages 184–187. find
  36. Andrei Mikheev, Marc Moens, and Claire Grover. 1999. Named entity recognition without gazetteers. In Ninth Conference of the European Chapter of the Association for Computational Linguistics, pages 1– 8.
  37. Hong Ming, Jiaoyun Yang, Lili Jiang, Yan Pan, and Ning An. 2022. Few-shot nested named entity recognition. arXiv preprint arXiv:2212.00953. — [arXiv:2212.00953](https://arxiv.org/abs/2212.00953) find
  38. OpenAI. 2023. Gpt-4 technical report.
  39. Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. arXiv preprint cs/0306050.
  40. Abhishek Sharma, Sudeshna Chakraborty, and Shivam Kumar. 2022. Named entity recognition in natural language processing: A systematic review. In Proceedings of Second Doctoral Symposium on Computational Intelligence: DoSCI 2021, pages 817–828. Springer. find
  41. Yongliang Shen, Xinyin Ma, Zeqi Tan, Shuai Zhang, Wen Wang, and Weiming Lu. 2021. Locate and label: A two-stage identifier for nested named entity recognition. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Na…
  42. Yiqi Tong, Fuzhen Zhuang, Deqing Wang, Haochao Ying, and Binling Wang. 2022. Improving biomedical named entity recognition with a unified multi-task mrc framework. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 8332–8336. IEEE. find
  43. Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. 2022a. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001.
  44. Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. 2022b. Deepstruct: Pre-training of language models for structure prediction. arXiv preprint arXiv:2205.10475. — [arXiv:2205.10475](https://arxiv.org/abs/2205.10475) find
  45. Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2020. Automated concatenation of embeddings for structured prediction. arXiv preprint arXiv:2010.05006.
  46. Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652. find
  47. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903.
  48. Weiwen Xu, Xin Li, Wenxuan Zhang, Meng Zhou, Lidong Bing, Wai Lam, and Luo Si. 2022. From cloz-ing to comprehending: Retrofitting pre-trained language model to pre-trained machine reader. arXiv preprint arXiv:2212.04755. — [arXiv:2212.04755](https://arxiv.org/abs/2212.04755) find
  49. Linyi Yang, Lifan Yuan, Leyang Cui, Wenyang Gao, and Yue Zhang. 2022. Factmix: Using a few labeled in-domain examples to generalize to cross-domain named entity recognition. arXiv preprint arXiv:2208.11464. find
  50. Juntao Yu, Bernd Bohnet, and Massimo Poesio. 2020. Named entity recognition as dependency parsing. arXiv preprint arXiv:2005.07150.
  51. GuoDong Zhou and Jian Su. 2002. Named entity recognition using an hmm-based chunk tagger. In Proceedings of the 40th annual meeting of the association for computational linguistics, pages 473–480.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ashok-2023-promptner-prompting-named-entity-recognition/