Elamur Research
☰
home / papers / 2023 / InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction

InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2304.08085

Summary

Large language models handle many tasks well when given prompts, but they still struggle with information extraction (IE). For example, gpt-3.5-turbo scores only 18.22 F1 on Ontonotes. Earlier unified approaches also have drawbacks. UIE must be fine-tuned separately for each downstream task. USM relies on per-token semantic matching, which is slow and hard to combine with generative models.

The authors present InstructUIE, which treats named entity recognition, relation extraction and event extraction as text generation. Each example has four parts:

  • an expert-written instruction,
  • a list of candidate labels (options),
  • the input text,
  • a target answer written in natural language.

Paired auxiliary subtasks are trained alongside each main task, such as span extraction and entity typing for NER.

For training and evaluation, they release IE INSTRUCTIONS, a benchmark of 32 public datasets with harmonized label names. In supervised experiments, the model performs on par with Bert. In zero-shot tests, it beats the strongest prior systems. This shows that one instruction-tuned model can cover many IE tasks and generalize to unseen label sets.

Key points

  • IE INSTRUCTIONS: 32 datasets covering NER, RE and EE across science, healthcare, social media, transportation, news and Wikidata. Labels were unified and rewritten in plain language, e.g. "people person place_of_birth" became "place of birth".
  • Auxiliary tasks: entity span extraction and entity typing (NER), entity pair extraction and relation classification (RE), trigger extraction and argument extraction (EE).
  • Zero-shot NER: InstructUIE outperforms USM on every dataset except CrossNER_Literature, by 5.21 to 25.27 Micro-F1 points. The gain exceeds 20 points on MIT Movie and CrossNER_AI. It still trails ChatGPT on NER.
  • Zero-shot RE: with 5 unseen labels and 5 random seeds, it beats ZETT by 5.84 points on FewRel and 3.46 points on Wiki-ZSL. Davinci and ChatGPT score very low here (ChatGPT: 9.96 and 13.14).
  • Availability: data, code and models are at github.com/BeyonderXX/InstructUIE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Topics: Named entity recognition

Cited by

146 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 145 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2304.08085 · arXiv 2304.08085

@misc{wang2023instructuie,
  title = {{InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction}},
  author = {Wang, Xiao and Zhou, Weikang and Zu, Can and Xia, Han and Chen, Tianze and Zhang, Yuansen and Zheng, Rui and Ye, Junjie and Zhang, Qi and Gui, Tao and Kang, Jihua and Yang, Jingsheng and Li, Siyuan and Du, Chunsai},
  year = {2023},
  doi = {10.48550/arxiv.2304.08085},
  eprint = {2304.08085},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2304.08085}
}

References

77 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Al-Rfou, Vivek Kulkarni, Bryan Perozzi, and Steven Skiena. 2014. POLYGLOT-NER: massive multilingual named entity recognition. CoRR, abs/1410.3791. — [arXiv:1410.3791](https://arxiv.org/abs/1410.3791) find
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, C… find
  3. Chih-Yao Chen and Cheng-Te Li. 2021. Zs-bert: Towards zero-shot relation extraction with attribute representation learning. find
  4. Pei Chen, Haotian Xu, Cheng Zhang, and Ruihong Huang. 2022a. Crossroads, buildings and neighborhoods: A dataset for fine-grained location recognition. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa… find
  5. Pei Chen, Haotian Xu, Cheng Zhang, and Ruihong Huang. 2022b. Crossroads, buildings and neighborhoods: A dataset for fine-grained location recognition. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa… find
  6. Xuanting Chen, Junjie Ye, Can Zu, Nuo Xu, Rui Zheng, Minlong Peng, Jie Zhou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. How robust is gpt-3.5 to predecessors? a comprehensive study on language understanding tasks. arXiv preprint arXiv:2303.00293. — [arXiv:2303.00293](https://arxiv.org/abs/2303.00293) find
  7. Yew Ken Chia, Lidong Bing, Soujanya Poria, and Luo Si. 2022. Relationprompt: Leveraging prompts to generate synthetic data for zero-shot relation triplet extraction. find
  8. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416. find
  9. Leon Derczynski, Kalina Bontcheva, and Ian Roberts. 2016. Broad Twitter corpus: A diverse named entity recognition resource. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1169– 1179, Osaka, Japan. find
  10. The COLING 2016 Organizing Committee. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  12. Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: A resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1–10. find
  13. Runwei Guan. 2022. Findvehicle and vehiclefinder: A ner dataset for a text-image cross-modal vehicle retrieval system. find
  14. Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann-Apitius, and Luca Toldo. 2012. Development of a benchmark corpus to support the automatic extraction of drug-related adverse effects from medical case reports. Journal of Biomedical Informatics, 45(5):885–892. Tex… find
  15. Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2018. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the-art evaluation. find
  16. Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. 2010. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. In *SEMEVAL. find our card
  17. Eduard H. Hovy, Mitchell P. Marcus, Martha Palmer, Lance A. Ramshaw, and Ralph M. Weischedel. 2006. Ontonotes: The 90% solution. In North American Chapter of the Association for Computational Linguistics. find
  18. Pere-Lluís Huguet Cabot and Roberto Navigli. 2021. REBEL: Relation extraction by end-to-end language generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2370– 2381, Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  19. Sharmistha Jat, Siddhesh Khandelwal, and Partha Pratim Talukdar. 2018. Improving distantly supervised relation extraction using word and entity based attention. ArXiv, abs/1804.06987. — [arXiv:1804.06987](https://arxiv.org/abs/1804.06987) find
  20. Bosung Kim, Hayate Iso, Nikita Bhutani, Estevam Hruschka, and Ndapa Nakashole. 2022. Zero-shot triplet extraction by template infilling. find
  21. Jin-Dong Kim, Tomoko Ohta, Yuka Tateisi, and Junichi Tsujii. 2003a. Genia corpus - a semantically annotated corpus for bio-textmining. Bioinformatics, 19 Suppl 1:i180–2. find
  22. Jin-Dong Kim, Tomoko Ohta, Yuka Tateisi, and Jun’ichi Tsujii. 2003b. Genia corpus—a semantically annotated corpus for bio-textmining. Bioinformatics (Oxford, England), 19 Suppl 1:i180–2. find
  23. Veysel Kocaman and David Talby. 2020a. Biomedical named entity recognition at scale. In ICPR Workshops. find
  24. Veysel Kocaman and David Talby. 2020b. Biomedical named entity recognition at scale. CoRR, abs/2011.06315. find
  25. Veysel Kocaman and David Talby. 2020c. Biomedical named entity recognition at scale. CoRR, abs/2011.06315. find
  26. Veysel Kocaman and David Talby. 2022a. Accurate clinical and biomedical named entity recognition at scale. Software Impacts, 13:100373. find
  27. Veysel Kocaman and David Talby. 2022b. Accurate clinical and biomedical named entity recognition at scale. Software Impacts, 13:100373. find
  28. Martin Krallinger, Obdulia Rabal, Florian Leitner, Miguel Vazquez, David Salgado, Zhiyong lu, Robert Leaman, Yanan Lu, Donghong Ji, Daniel Lowe, Roger Sayle, Riza Batista-Navarro, Rafal Rak, Torsten Huber, Tim Rocktäschel, Sérgio Matos, David Campos, Buzhou Tang, Wang Qi, and Alfonso Valencia. 2015.… find
  29. Aman Kumar and Binil Starly. 2021a. “fabner”: information extraction from manufacturing process science domain literature using named entity recognition. Journal of Intelligent Manufacturing, 33:2393 – 2407. find
  30. Aman Kumar and Binil Starly. 2021b. “fabner”: information extraction from manufacturing process science domain literature using named entity recognition. Journal of Intelligent Manufacturing, 33. find
  31. Sunny Lai, Kwong Sak Leung, and Yee Leung. 2018. SUNNYNLP at SemEval-2018 task 10: A support-vector-machine-based method for detecting semantic difference using taxonomy and word embedding features. In Proceedings of the 12th International Workshop on Semantic Evaluation, pages 741–746, New Orleans,… find
  32. Jiao Li, Yueping Sun, Robin J. Johnson, Daniela Sci-aky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J. Mattingly, Thomas C. Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database: The Journal of Biological Databases… find
  33. Xiaoya Li, Xiaofei Sun, Yuxian Meng, Junjun Liang, Fei Wu, and Jiwei Li. 2019. Dice loss for data-imbalanced NLP tasks. CoRR, abs/1911.02855. find
  34. Yijin Liu, Fandong Meng, Jinchao Zhang, Jinan Xu, Yufeng Chen, and Jie Zhou. 2019. GCDT: A global context enhanced deep transition architecture for sequence labeling. CoRR, abs/1906.02437. find
  35. Zihan Liu, Yan Xu, Tiezheng Yu, Wenliang Dai, Ziwei Ji, Samuel Cahyawijaya, Andrea Madotto, and Pascale Fung. 2020. Crossner: Evaluating cross-domain named entity recognition. find
  36. S. Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning. ArXiv, abs/2301.13688. find our card
  37. Jie Lou, Yaojie Lu, Dai Dai, Wei Jia, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2023. Universal information extraction as unified semantic matching. find
  38. Yaojie Lu, Hongyu Lin, Jin Xu, Xianpei Han, Jialong Tang, Annan Li, Le Sun, M. Liao, and Shaoyi Chen. 2021. Text2event: Controllable sequence-to-structure generation for end-to-end event extraction. ArXiv, abs/2106.09232. find our card
  39. Yaojie Lu, Qing Liu, Dai Dai, Xinyan Xiao, Hongyu Lin, Xianpei Han, Le Sun, and Hua Wu. 2022. Unified structure generation for universal information extraction. find
  40. Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. find our card
  41. Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generalization via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3470–3487, Dublin, Irelan… find
  42. OpenAI. 2023. Gpt-4 technical report. abs/2303.08774. find our card
  43. ArXiv, openbiocorpora. 2015. openbiocorpora anatem. find
  44. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Low… find
  45. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017a. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada.… find
  46. Xiaoman Pan, Boliang Zhang, Jonathan May, Joel Nothman, Kevin Knight, and Heng Ji. 2017b. Cross-lingual name tagging and linking for 282 languages. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1946–1958, Vancouver, Canada.… find
  47. Mihir Parmar, Swaroop Mishra, Mirali Purohit, Man Luo, Murad Mohammad, and Chitta Baral. 2022. In-BoXBART: Get instructions into biomedical multi-task learning. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 112–128, Seattle, United States. Association for Computatio… find
  48. N Poolsawad, C Kambhampati, and JGF Cleland. 2014. Balancing class for performance of classification with a clinical dataset. In proceedings of the World Congress on Engineering, volume 1, pages 1–6. find
  49. Sebastian Riedel, Limin Yao, and Andrew McCallum. 2010. Modeling relations and their mentions without labeled text. In ECML/PKDD. find
  50. Dan Roth and Wen tau Yih. 2004. A linear programming formulation for global inference in natural language tasks. In Conference on Computational Natural Language Learning. find our card
  51. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. find
  52. Stefan Schweter and Alan Akbik. 2020. FLERT: document-level features for named entity recognition. CoRR, abs/2011.06993. — [arXiv:2011.06993](https://arxiv.org/abs/2011.06993) find
  53. Zhao-Li Sun, Jiazheng Li, Gabriele Pergola, Byron C. Wallace, Bino John, Nigel Greene, Joseph Kim, and Yulan He. 2022. Phee: A dataset for pharmacovigilance event extraction from text. ArXiv, abs/2210.12560. find
  54. Ryuichi Takanobu, Tianyang Zhang, Jiexi Liu, and Minlie Huang. 2018. A hierarchical framework for relation extraction with reinforcement learning. In AAAI Conference on Artificial Intelligence. find
  55. Minghao Tang, Peng Zhang, Yongquan He, Yongxiu Xu, Chengpeng Chao, and Hongbo Xu. 2022. DoSEA: A domain-specific entity-aware framework for cross-domain named entity recogition. In Proceedings of the 29th International Conference on Computational Linguistics, pages 2147–2156, Gyeongju, Republic of K… find
  56. Simone Tedeschi, Valentino Maiorca, Niccolò Campolungo, Francesco Cecconi, and Roberto Navigli. 2021. WikiNEuRal: Combined neural and knowledge-based silver data creation for multilingual NER. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2521– 2533, Punta Cana, Dom… find
  57. Simone Tedeschi and Roberto Navigli. 2022a. MultiN-ERD: A multilingual, multi-genre and fine-grained dataset for named entity recognition (and disambiguation). In Findings of the Association for Computational Linguistics: NAACL 2022, pages 801– 812, Seattle, United States. Association for Computatio… find
  58. Simone Tedeschi and Roberto Navigli. 2022b. MultiN-ERD: A multilingual, multi-genre and fine-grained dataset for named entity recognition (and disambiguation). In Findings of the Association for Computational Linguistics: NAACL 2022, pages 801– 812, Seattle, United States. Association for Computatio… find
  59. Christos Theodoropoulos and Marie-Francine Moens. 2023. An information extraction study: Take in mind the tokenization! find
  60. Asahi Ushio and Jose Camacho-Collados. 2021. T-NER: An all-round python library for transformer-based named entity recognition. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. Association for Computational Linguis… find
  61. Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, and Jose Camacho-Collados. 2022a. Named entity recognition in twitter: A dataset and analysis on short-term temporal shifts. find
  62. Asahi Ushio, Leonardo Neves, Vitor Silva, Francesco Barbieri, and Jose Camacho-Collados. 2022b. Named entity recognition in twitter: A dataset and analysis on short-term temporal shifts. find
  63. C. Walker and Linguistic Data Consortium. 2005. find
  64. ACE 2005 Multilingual Training Corpus. LDC corpora. Linguistic Data Consortium. find
  65. Chenguang Wang, Xiao Liu, Zui Chen, Haoyun Hong, Jie Tang, and Dawn Song. 2023. Deepstruct: Pre-training of language models for structure prediction. find
  66. Liwen Wang, Rumei Li, Yang Yan, Yuanmeng Yan, Sirui Wang, Wei Yu Wu, and Weiran Xu. 2022a. Instructionner: A multi-task instruction-based generative framework for few-shot ner. ArXiv, abs/2203.03903. — [arXiv:2203.03903](https://arxiv.org/abs/2203.03903) find
  67. Xiao Wang, Shihan Dou, Limao Xiong, Yicheng Zou, Qi Zhang, Tao Gui, Liang Qiao, Zhanzhan Cheng, and Xuanjing Huang. 2022b. MINER: Improving out-of-vocabulary named entity recognition from an information theoretic perspective. In Proceedings of the 60th Annual Meeting of the Association for Computati… find
  68. Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021a. Automated concatenation of embeddings for structured prediction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on… find
  69. Yijun Wang, Changzhi Sun, Yuanbin Wu, Hao Zhou, Lei Li, and Junchi Yan. 2021b. UniRE: A unified label space for entity relation extraction. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Proce… find
  70. Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal… find
  71. Chenhao Xie, Jiaqing Liang, Jingping Liu, Chengsong Huang, Wenhao Huang, and Yanghua Xiao. 2021. Revisiting the negative data of distantly supervised relation extraction. CoRR, abs/2105.10158. find
  72. Hang Yan, Junqi Dai, Tuo Ji, Xipeng Qiu, and Zheng Zhang. 2021. A unified generative framework for aspect-based sentiment analysis. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (V… find
  73. Deming Ye, Yankai Lin, and Maosong Sun. 2021. Pack together: Entity and relation extraction with levitated marker. CoRR, abs/2109.06067. — [arXiv:2109.06067](https://arxiv.org/abs/2109.06067) find
  74. Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. 2023. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420. find
  75. Dongxu Zhang and Dong Wang. 2015. Relation classification via recurrent neural network. find
  76. Sheng Zhang, Hao Cheng, Jianfeng Gao, and Hoifung Poon. 2023. Optimizing bi-encoder for named entity recognition via contrastive learning. find
  77. Zexuan Zhong and Danqi Chen. 2020. A frustratingly easy approach for joint entity and relation extraction. CoRR, abs/2010.12812. find our card