Elamur Research⌕
☰
home / papers / 2024 / AutoRE: Document-Level Relation Extraction with Large Language Models

AutoRE: Document-Level Relation Extraction with Large Language Models

Paper · 2024 · Lilong Xue, Dan Zhang, Yuxiao Dong et al.

Summary

Document-level relation extraction (DocRE) requires recovering every relational triplet spread across a whole text, often involving many relation types per document. The authors show that off-the-shelf LLMs (ChatGPT, Mistral-7B, Vicuna-7B, ChatGLM3-6B) handle this poorly. They also argue that the common practice of listing candidate relations inside the prompt breaks down when the inventory is large. Their system, AutoRE, follows a pipeline paradigm called RHF (Relation-Head-Facts). The model first predicts which relations a document contains, then finds subject entities for each relation, and finally generates complete triplets for each relation–head pair. The team rewrote descriptions for all 96 Re-DocRED relation types and built an instruction-tuning set from Re-DocRED. Each of the three stages gets its own QLoRA adapter, and no relation list is supplied at inference. AutoRE achieves state-of-the-art strict micro F1 on Re-DocRED with several base models, beating the earlier end-to-end TAG system. The work shows that small open LLMs can become strong DocRE extractors through a modular, cheaply extendable design.

Key points

  • Evaluation uses Re-DocRED (96 relation types). The test split has 499 documents and 17,448 triplets, and the dev split has 498 documents and 17,236 triplets. Scoring is strict micro F1, where a prediction counts only if the relation, head and tail all match (any valid alias pair is accepted).
  • With prompting alone, ChatGPT scores very low on every paradigm tried (D-F, D-RS-F, D-R-F, D-R-H-F), with F1 between roughly 1.8 and 7.5. This motivated the authors to fine-tune rather than rely on prompting.
  • Compared with TAG, the Mistral-7B version of AutoRE gains about 7.44% on dev and 5.12% on test. Vicuna-7B performs best, gaining 10.03% on dev and 9.03% on test. ChatGLM3-6B also beats the previous state of the art but trails the other two, possibly because its pre-training data is heavily Chinese.
  • Ablations on Mistral-7B show three findings:
    • RHF is the strongest paradigm even with a single adapter.
    • Relation descriptions improve results.
    • Three separate QLoRA modules outperform one shared module. The authors attribute this to subtask imbalance: the relation, head and triplet steps make up about 2.8%, 24.23% and 72.97% of the training data, respectively.
  • Code is released at github.com/THUDM/AutoRE, along with an online demo. The authors acknowledge two limitations: the model cannot handle unseen relations, and 96 relations is far fewer than real-world needs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Information extraction · 167 works

Cite

Cite as Lilong Xue, Dan Zhang, Yuxiao Dong and Jie Tang. 2024. AutoRE: Document-Level Relation Extraction with Large Language Models. arXiv:2403.14888.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2403.14888 · arXiv 2403.14888

@misc{xue2024autore,
  title = {{AutoRE: Document-Level Relation Extraction with Large Language Models}},
  author = {Lilong Xue, Dan Zhang, Yuxiao Dong and Jie Tang},
  year = {2024},
  doi = {10.48550/arxiv.2403.14888},
  eprint = {2403.14888},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2403.14888}
}

References

29 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. 2023. Prompting is programming: A query language for large language models. Proceedings of the ACM on Programming Languages, 7(PLDI):1946–1969. find
  3. Zheng Chen and Changyu Guo. 2022. A pattern-first pipeline approach for entity and relation extraction. Neurocomputing, 494:182–191. find
  4. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  5. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. ArXiv, abs/2305.14314.
  6. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335…
  7. Chufan Gao, Xulin Fan, Jimeng Sun, and Xuan Wang. 2023. Promptre: Weakly-supervised document-level relation extraction via prompting-based data programming. arXiv preprint arXiv:2310.09265. find
  8. Ridong Han, Tao Peng, Chaohao Yang, Benyou Wang, Lu Liu, and Xiang Wan. 2023. Is information extraction solved by chatgpt? an analysis of performance, evaluation criteria, robustness and errors. ArXiv, abs/2305.14450. — [arXiv:2305.14450](https://arxiv.org/abs/2305.14450) find
  9. J. Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685.
  10. Albert Qiaochu Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L’elio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée La… find
  11. Ming Jiang, Jennifer D’Souza, Sören Auer, and J Stephen Downie. 2020. Targeting precision: A hybrid scientific relation extraction pipeline for improved scholarly knowledge organization. Proceedings of the Association for Information Science and Technology, 57(1):e303. find
  12. Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023a. Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness. ArXiv, abs/2304.11633. find
  13. Junpeng Li, Zixia Jia, and Zilong Zheng. 2023b. Semi-automatic data enhancement for document-level relation extraction with distant supervision from large language models. In Conference on Empirical Methods in Natural Language Processing. find
  14. Xiaoya Li, Fan Yin, Zijun Sun, Xiayu Li, Arianna Yuan, Duo Chai, Mingxin Zhou, and Jiwei Li. 2019. Entity-relation extraction as multi-turn question answering. ArXiv, abs/1905.05529.
  15. Qingyu Tan, Lu Xu, Lidong Bing, Hwee Tou Ng, and Sharifah Mahani Aljunied. 2022. Revisiting docred - addressing the false negative problem in relation extraction. In Conference on Empirical Methods in Natural Language Processing.
  16. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  17. Somin Wadhwa, Silvio Amir, and Byron C. Wallace. 2023. Revisiting relation extraction in the era of large language models. Proceedings of the conference. Association for Computational Linguistics. Meeting, 2023:15566–15589. find
  18. Shuhe Wang, Xiaofei Sun, Xiaoya Li, Rongbin Ouyang, Fei Wu, Tianwei Zhang, Jiwei Li, and Guoyin Wang. 2023a. Gpt-ner: Named entity recognition via large language models. arXiv preprint arXiv:2304.10428.
  19. Xiao Wang, Wei Zhou, Can Zu, Han Xia, Tianze Chen, Yuan Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, Jihua Kang, J. Yang, Siyuan Li, and Chunsai Du. 2023b. Instructuie: Multi-task instruction tuning for unified information extraction. ArXiv, abs/2304.08085.
  20. Xiang Wei, Xingyu Cui, Ning Cheng, Xiaobin Wang, Xin Zhang, Shen Huang, Pengjun Xie, Jinan Xu, Yufeng Chen, Meishan Zhang, Yong Jiang, and Wenjuan Han. 2023. Zero-shot information extraction via chatting with chatgpt. ArXiv, abs/2302.10205.
  21. Xinglin Xiao, Yijie Wang, Nan Xu, Yuqi Wang, Hanxuan Yang, Minzheng Wang, Yin Luo, Lei Wang, Wenji Mao, and Daniel Zeng. 2023. Yayi-uie: A chat-enhanced instruction tuning framework for universal information extraction. ArXiv, abs/2312.15548. — [arXiv:2312.15548](https://arxiv.org/abs/2312.15548) find
  22. Derong Xu, Wei Chen, Wenjun Peng, Chao Zhang, Tong Xu, Xiangyu Zhao, Xian Wu, Yefeng Zheng, and Enhong Chen. 2023. Large language models for generative information extraction: A survey. arXiv preprint arXiv:2312.17617.
  23. Dan Zhang, Ziniu Hu, Sining Zhoubian, Zhengxiao Du, Kaiyu Yang, Zihan Wang, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024a. Sciglm: Training scientific language models with self-reflective instruction annotation and tuning. arXiv preprint arXiv:2401.07950. find
  24. Dan Zhang, Sining Zhoubian, Yisong Yue, Yuxiao Dong, and Jie Tang. 2024b. Rest-mcts*: Llm self-training via process reward guided tree search. arXiv preprint arXiv:2406.03816.
  25. Ruoyu Zhang, Yanzeng Li, and Lei Zou. 2023. A novel table-to-graph generation approach for document-level joint entity and relation extraction. In Annual Meeting of the Association for Computational Linguistics. find
  26. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Z. Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jianyun Nie, and Ji rong Wen. 2023. A survey of large l…
  27. Hanwen Zheng, Sijia Wang, and Lifu Huang. 2023. A survey of document-level information extraction. ArXiv, abs/2309.13249. — [arXiv:2309.13249](https://arxiv.org/abs/2309.13249) find
  28. Huixue Zhou, Mingchen Li, Yongkang Xiao, Han Yang, and Rui Zhang. 2023. Llm instruction-example adaptive prompting (leap) framework for clinical relation extraction. medRxiv, pages 2023–12. find
  29. Ming Zhou, Dan Zhang, Yuandong Wang, Yangli-ao Geng, Yuxiao Dong, and Jie Tang. 2024. Lgb: Language model and graph neural network-driven social bot detection. arXiv preprint arXiv:2406.08762. — [arXiv:2406.08762](https://arxiv.org/abs/2406.08762) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xue-2024-autore-document-level-relation-extraction/