Elamur Research⌕
☰
home / papers / 2022 / Prompt-learning for Fine-grained Entity Typing

Prompt-learning for Fine-grained Entity Typing

Paper · 2022 · Ning Ding, Yulin Chen, Xu Han et al.

Summary

The paper asks whether prompt-learning, which recasts a downstream task as a cloze-style masked-word prediction, can replace classifier-based fine-tuning for fine-grained entity typing. It focuses on cases with little or no labelled data. The authors build a simple pipeline called PLET with two parts:

  • Verbalizers: each hierarchical type, such as LOCATION/CITY, is mapped to the words in its label path, and the scores of those words at the [MASK] position are averaged.
  • Templates: either hand-written natural-language templates or learnable soft-token templates are wrapped around a copy of the entity mention.

For the zero-shot case, where no labels exist, they add a self-supervised variant, PLET (S). It learns type information from unlabelled text and the label schema by making the predicted distributions of paired examples similar, rather than fitting individual labels. Experiments on FEW-NERD, OntoNotes, BBN and Open Entity show clear gains over fine-tuning in few-shot settings and further gains from self-supervision in zero-shot settings. This makes prompting a practical option for typing entities when annotation is scarce.

Key points

  • Zero-shot results with BERT-base (Table 4): PLET (S) improves on plain PLET across all four benchmarks.
    • FEW-NERD micro-F1 rises from 28.39 to 47.98.
    • OntoNotes micro-F1 (with "Other" removed) rises from 33.61 to 49.79.
    • BBN accuracy rises from 55.82 to 57.79.
    • Open Entity accuracy rises from 11.37 to 18.78.
  • Templates (FEW-NERD, 8-shot): Among hard templates, adding the phrase "in this sentence" helps noticeably. Among soft templates, the shortest one (l=2) scores best, with 59.25 accuracy and 69.58 micro-F1.
  • Backbone comparison: In zero-shot mode, GPT-2 does best and T5-base does worst (FEW-NERD micro-F1: 41.83 for GPT-2, 28.39 for BERT-base, 14.30 for T5-base). With full supervision the three models perform similarly, with BERT-base reaching about 85.8 micro-F1 on FEW-NERD.
  • Random verbalizer: Mapping types to random label words hurts badly at very low shot counts. The gap almost disappears by 128 shots.
  • Code: Released at github.com/thunlp/PromptTyping and also integrated into OpenPrompt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China137 works · 83,262 citations🏢Alibaba GroupHangzhou · China64 works · 14,113 citations🎓Seoul National UniversitySeoul · South Korea20 works · 45,282 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Named entity recognition · 101 works

Cite

Cite as Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Xiaobin Wang, Pengjun Xie, Haitao Zheng, Zhiyuan Liu, Juanzi Li and Hong-Gee Kim. 2022. Prompt-learning for Fine-grained Entity Typing. Findings of the Association for Computational Linguistics: EMNLP 2022.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.findings-emnlp.512

@inproceedings{ding2022prompt,
  title = {{Prompt-learning for Fine-grained Entity Typing}},
  author = {Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Xiaobin Wang, Pengjun Xie, Haitao Zheng, Zhiyuan Liu, Juanzi Li and Hong-Gee Kim},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2022},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.512},
  url = {https://doi.org/10.18653/v1/2022.findings-emnlp.512}
}

References

47 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2895– 2905, Florence, Italy. Association for Com…
  2. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proceedings of NIPS, pages 1877–1901.
  3. Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. 2018a. Ultra-fine entity typing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 87–96, Melbourne, Australia. Association for Computational Linguistics. find
  4. Eunsol Choi, Omer Levy, Yejin Choi, and Luke Zettlemoyer. 2018b. Ultra-fine entity typing. In Proceedings of ACL, pages 87–96. find
  5. Hongliang Dai, Yangqiu Song, and Haixun Wang. 2021. Ultra-fine entity typing with weak supervision from a masked language model. In Proceedings of ACL, pages 1790–1799. find
  6. Joe Davison, Joshua Feldman, and Alexander M Rush. 2019. Commonsense knowledge mining from pre-trained models. In Proceedings of EMNLP-IJCNLP, pages 1173–1178.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019a. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171–4186.
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019b. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
  9. Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Hai-Tao Zheng, and Maosong Sun. 2021a. Openprompt: An open-source framework for prompt-learning. arXiv preprint arXiv:2111.01998.
  10. Ning Ding, Xiaobin Wang, Yao Fu, Guangwei Xu, Rui Wang, Pengjun Xie, Ying Shen, Fei Huang, Hai-Tao Zheng, and Rui Zhang. 2021b. Prototypical representation learning for relation extraction. In Proceedings of ICLR. find
  11. Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Hai-Tao Zheng, and Zhiyuan Liu. 2021c. Few-nerd: A few-shot named entity recognition dataset. In Proceedings of ACL, pages 3198–3213.
  12. Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012.15723.
  13. Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, et al. 2021a. Pre-trained models: Past, present and future. arXiv preprint arXiv:2106.07139. find
  14. Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2021b. Ptr: Prompt tuning with rules for text classification. arXiv preprint arXiv:2105.11259. find
  15. John Hewitt and Christopher D Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of NAACL, pages 4129–4138.
  16. Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Juanzi Li, and Maosong Sun. 2021. Knowledgeable prompt-tuning: Incorporating knowledge into prompt verbalizer for text classification. arXiv preprint arXiv:2108.02035.
  17. Ganesh Jawahar, Benoît Sagot, and Djamé Seddah. 2019. What does bert learn about the structure of language? In Proceedings of ACL, pages 3651–3657.
  18. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691. find
  19. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. find
  20. Thomas Lin, Oren Etzioni, et al. 2012. No noun phrase left behind: detecting and typing unlinkable entities. In Proceedings of EMNLP-CoNLL, pages 893–903. find
  21. Xiao Ling and Daniel S. Weld. 2012. Fine-grained entity recognition. In AAAI. find
  22. Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too. arXiv preprint arXiv:2103.10385.
  23. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  24. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In Proceedings of ICLR. find
  25. Yasumasa Onoe, Michael Boratko, Andrew McCallum, and Greg Durrett. 2021. Modeling fine-grained entity types with box embeddings. arXiv preprint arXiv:2101.00345. find
  26. Yasumasa Onoe and Greg Durrett. 2019. Learning to denoise distantly-labeled data for entity typing. arXiv preprint arXiv:1905.01566. find
  27. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and…
  28. Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2020. Learning from Context or Names? An Empirical Study on Neural Relation Extraction. In Proceedings of EMNLP, pages 3661–3672.
  29. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of EMNLP, pages 2463– 2473.
  30. Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: A survey. Science China Technological Sciences, pages 1–26.
  31. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. OpenAI. find
  32. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. find
  33. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21:1–67.
  34. Xiang Ren, Wenqi He, Meng Qu, Lifu Huang, Heng Ji, and Jiawei Han. 2016a. AFET: Automatic fine-grained entity typing by hierarchical partial-label embedding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1369–1378, Austin, Texas. Association for Com… find
  35. Xiang Ren, Wenqi He, Meng Qu, Clare R. Voss, Heng Ji, and Jiawei Han. 2016b. Label noise reduction in entity typing by heterogeneous partial-label embedding. In Proceedings of SIGKDD, page 1825–1834. find
  36. Timo Schick, Helmut Schmid, and Hinrich Schütze. 2020. Automatically identifying words that can serve as labels for few-shot text classification. In Proceedings of COLING, pages 5569–5578.
  37. Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of EACL, pages 255–269.
  38. Sonse Shimaoka, Pontus Stenetorp, Kentaro Inui, and Sebastian Riedel. 2017. Neural architectures for fine-grained entity type classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 1271–1280, Valenc… find
  39. Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models using automatically generated prompts. In Proceedings of EMNLP, pages 4222–4235. find
  40. Dong Wang, Ning Ding, Piji Li, and Haitao Zheng. 2021a. CLINE: Contrastive learning with semantic negative examples for natural language understanding. In Proceedings of ACL. find
  41. Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021b. K-adapter: Infusing knowledge into pre-trained models with adapters. In Findings of ACL/IJCNLP, pages 1405–1418.
  42. Ralph Weischedel and Ada Brunstein. 2005. BBN Pronoun Coreference and Entity Type Corpus. Linguistic Data Consortium, Philadelphia. find
  43. Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Nianwen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, Mohammed El-Bachouti, Robert Belvin, and Ann Houston. 2013. OntoNotes Release 5.0. Abacus Data Network. find
  44. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho…
  45. Wenhan Xiong, Jiawei Wu, Deren Lei, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2019. Imposing label-relational inductive bias for extremely fine-grained entity typing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics:… find
  46. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of ACL, pages 1441–1451. find
  47. Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improving few-shot performance of language models. arXiv preprint arXiv:2102.09690.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ding-2022-prompt-learning-fine-grained-entity/