Elamur Research⌕
☰
home / papers / 2026 / HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy

HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy

Paper · 2026 · Guoqi Ma, Liang Zhang, Hongyao Tu et al.

Summary

The paper addresses cross-document relation extraction (RE), where the head and tail entities of a relation appear in different documents. Earlier systems pair a small language model with a classifier. The authors first test fine-tuned LLMs on CodRED and find that they often do no better than strong SLM baselines. They trace this mainly to the large label space: 277 relations make similar labels hard to tell apart and make prompts too long. Their model, HCRE, builds a hierarchical relation tree. Leaves are the original relations, and inner nodes are broader concepts. The LLM picks a relation by moving down the tree one level at a time, so it sees only a few options at each step. Mistakes at upper levels can carry over to lower ones. To limit this, a prediction-then-verification step has the model choose a best and a runner-up node, then re-check them using option sets built from their children. On CodRED, HCRE beats existing baselines. This suggests that shrinking the option set is key to making LLMs work for RE with many labels.

Key points

  • The relation tree cuts the options per decision from 277 to about 5.5 on average.
  • The preliminary study fine-tuned LLaMA-3.1-8B-Instruct, Gemma-2-9B-Instruct and Qwen-2.5-7B-Instruct. Their accuracy rose when fewer relation options were given.
  • The authors argue that maximum F1 overstates performance and that P@K depends on dataset size. For example, ECRIM scores 61.64 maximum F1 but 39.25 micro F1 and 42.98 binary F1. They therefore use micro F1 and binary F1 as the main metrics.
  • Results change little with tree depth L ∈ {4, 5, 6}, and L = 5 works best. Ablations show that both the verification strategy and the tree contribute to performance.
  • Code is released at github.com/XMUDeepLIT/HCRE. One stated limitation is that the model processes only one text path per inference, so it cannot use dependencies across paths.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alibaba GroupHangzhou · China168 works · 15,386 citations🎓Xiamen UniversityXiamen · China21 works · 2,258 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Information extraction · 167 works

Cite

Cite as Guoqi Ma, Liang Zhang, Hongyao Tu, Hao Fu, Hui Li, Yujie Lin, Longyue Wang, Weihua Luo and Jinsong Su. 2026. HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1985

@inproceedings{ma2026hcre,
  title = {{HCRE: LLM-based Hierarchical Classification for Cross-Document Relation Extraction with a Prediction-then-Verification Strategy}},
  author = {Guoqi Ma, Liang Zhang, Hongyao Tu, Hao Fu, Hui Li, Yujie Lin, Longyue Wang, Weihua Luo and Jinsong Su},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1985},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1985}
}

References

44 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rui Cai, Xiaodong Zhang, and Houfeng Wang. 2016. Bidirectional recurrent convolutional neural network for relation classification. In ACL 2016. find
  2. Huiyao Chen, Yu Zhao, Zulong Chen, Mengjia Wang, Liangyue Li, Meishan Zhang, and Min Zhang. 2024. Retrieval-style in-context learning for few-shot hierarchical text classification. find
  3. Team Gemma. 2024. Gemma 2: Improving open language models at a practical size. Preprint, arXiv:2408.00118. find
  4. Shibo Hao, Yi Gu, Haodi Ma, Joshua Hong, Zhen Wang, Daisy Wang, and Zhiting Hu. 2023. Reasoning with language model is planning with world model. In EMNLP 2023. find
  5. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  6. Wei Huang, Chen Liu, Bo Xiao, Yihua Zhao, Zhaoming Pan, Zhimin Zhang, Xinyun Yang, and Guiquan Liu. 2022. Exploring label hierarchy in a generative way for hierarchical text classification. In COLING 2022. find
  7. Monika Jain, Raghava Mutharaju, Kuldeep Singh, and Ramakanth Kavuluru. 2024a. Knowledge-driven cross-document relation extraction. In Findings of ACL 2024. find
  8. Vidit Jain, Mukund Rungta, Yuchen Zhuang, Yue Yu, Zeyu Wang, Mu Gao, Jeffrey Skolnick, and Chao Zhang. 2024b. HiGen: Hierarchy-aware sequence generation for hierarchical text classification. In EACL 2024. find
  9. Peng Li, Tianxiang Sun, Qiong Tang, Hang Yan, Yuanbin Wu, Xuanjing Huang, and Xipeng Qiu. 2023. CodeIE: Large code generation models are better few-shot information extractors. In ACL 2023.
  10. Shuquan Lian, Juncheng Liu, Yazhe Chen, Yuhong Chen, and Hui Li. 2026. Swe-agile: A software agent framework for efficiently managing dynamic reasoning context. Preprint, arXiv:2604.11716. find
  11. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. Preprint, arXiv:1711.05101. find
  12. Keming Lu, I-Hung Hsu, Wenxuan Zhou, Mingyu Derek Ma, and Muhao Chen. 2023. Multi-hop evidence retrieval for cross-document relation extraction. In Findings of ACL 2023. find
  13. AI Meta. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  14. Byeonghu Na, Suhyeon Jo, Yeongmin Kim, and Il-chul Moon. 2024. Reward-based input construction for cross-document relation extraction. In ACL 2024. find
  15. Lorenzo Paletto, Valerio Basile, and Roberto Esposito. 2024. Label augmentation for zero-shot hierarchical text classification. In ACL 2024. find
  16. Liangying Shao, Liang Zhang, Minlong Peng, Guoqi Ma, Hao Yue, Mingming Sun, and Jinsong Su. 2024. One2Set + large language model: Best partners for keyphrase generation. In EMNLP 2024. find
  17. Junyoung Son, Jinsung Kim, Jungwoo Lim, Yoonna Jang, and Heuiseok Lim. 2023. Explore the way: Exploring reasoning path by bridging entities for effective cross-document relation extraction. In Findings of EMNLP 2023. find
  18. Seyed Amin Tabatabaei, Sarah Fancher, Michael Parsons, and Arian Askari. 2025. Can large language models serve as effective classifiers for hierarchical multi-label classification of scientific documents at industrial scale? In COLING 2025. find
  19. Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, Jihua Kang, Jingsheng Yang, Siyuan Li, and Chunsai Du. 2023a. Instructuie: Multi-task instruction tuning for unified information extraction. Preprint, arXiv:2304.08085.
  20. Yue Wang, Dan Qiao, Juntao Li, Jinxiong Chang, Qishen Zhang, Zhongyi Liu, Guannan Zhang, and Min Zhang. 2023b. Towards better hierarchical text classification with data generation. In Findings of ACL 2023. find
  21. Zilong Wang, Zifeng Wang, Long Le, Steven Zheng, Swaroop Mishra, Vincent Perot, Yuwei Zhang, Anush Mattapalli, Ankur Taly, Jingbo Shang, Chen-Yu Lee, and Tomas Pfister. 2025. Speculative RAG: Enhancing retrieval augmented generation through drafting. In ICLR 2025. find
  22. Haoran Wu, Xiuyi Chen, Zefa Hu, Jing Shi, Shuang Xu, and Bo Xu. 2023. Local-to-global causal reasoning for cross-document relation extraction. IEEE/CAA Journal of Automatica Sinica, 10(7). find
  23. Lilong Xue, Dan Zhang, Yuxiao Dong, and Jie Tang. 2024. AutoRE: Document-level relation extraction with large language models. In ACL 2024.
  24. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, and 40 others. 2024. Qwen2 technical report. arXiv preprint arXi… find
  25. Yuan Yao, Jiaju Du, Yankai Lin, Peng Li, Zhiyuan Liu, Jie Zhou, and Maosong Sun. 2021. CodRED: A cross-document relation extraction dataset for acquiring knowledge in the wild. In EMNLP 2021. find
  26. Robert J Tibshirani and Bradley Efron. 1993. An introduction to the bootstrap. Monographs on statistics and applied probability, 57(1). find
  27. Yuan Yao, Deming Ye, Peng Li, Xu Han, Yankai Lin, Zhenghao Liu, Zhiyuan Liu, Lixin Huang, Jie Zhou, and Maosong Sun. 2019. DocRED: A large-scale document-level relation extraction dataset. In ACL 2019. find
  28. Hongyao Tu, Liang Zhang, Yujie Lin, Xin Lin, Haibo Zhang, Long Zhang, and Jinsong Su. 2025. LLM-OREF: An open relation extraction framework based on large language models. In EMNLP 2025. find
  29. Chao Yu, Yi Shen, and Yue Mao. 2022. Constrained sequence-to-tree generation for hierarchical text classification. In SIGIR 2022. find
  30. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention is all you need. Preprint, arXiv:1706.03762.
  31. Hao Yue, Shaopeng Lai, Chengyi Yang, Liang Zhang, Junfeng Yao, and Jinsong Su. 2024. Towards better graph-based cross-document relation extraction via non-bridge entity enhancement and prediction debiasing. In Findings of ACL 2024. find
  32. Somin Wadhwa, Silvio Amir, and Byron Wallace. 2023. Revisiting relation extraction in the era of large language models. In ACL 2023. find
  33. Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao. 2014. Relation classification via convolutional deep neural network. In COLING 2014.
  34. Fengqi Wang, Fei Li, Hao Fei, Jingye Li, Shengqiong Wu, Fangfang Su, Wenxuan Shi, Donghong Ji, and Bo Cai. 2022. Entity-centered cross-document relation extraction. In EMNLP 2022. find
  35. Fu Zhang, Hongsen Yu, Jingwei Cheng, and Huangming Xu. 2025a. Entity pair-guided relation summarization and retrieval in LLMs for document-level relation extraction. In Findings of NAACL 2025. find
  36. Kaizhong Zhang and Dennis Shasha. 1989. Simple fast algorithms for the editing distance between trees and related problems. SIAM Journal on Computing, 18(6). find
  37. Liang Zhang, Zijun Min, Jinsong Su, Pei Yu, Ante Wang, and Yidong Chen. 2023a. Exploring effective inter-encoder semantic interaction for document-level relation extraction. In IJCAI 2023. find
  38. Liang Zhang, Jinsong Su, Yidong Chen, Zhongjian Miao, Min Zijun, Qingguo Hu, and Xiaodong Shi. 2022. Towards better document-level relation extraction via iterative inference. In EMNLP 2022. find
  39. Liang Zhang, Jinsong Su, Zijun Min, Zhongjian Miao, Qingguo Hu, Biao Fu, Xiaodong Shi, and Yidong Chen. 2023b. Exploring self-distillation based relational reasoning training for document-level relation extraction. In AAAI 2023. find
  40. Liang Zhang, Zhen Yang, Biao Fu, Ziyao Lu, Liangying Shao, Shiyu Liu, Fandong Meng, Jie Zhou, Xiaoli Wang, and Jinsong Su. 2024. Multi-level cross-modal alignment for speech relation extraction. In EMNLP 2024. find
  41. Liang Zhang, Yang Zhang, Ziyao Lu, Fandong Meng, Jie Zhou, and Jinsong Su. 2025b. A self-denoising model for robust few-shot relation extraction. In ACL 2025. find
  42. Liang Zhang, Chulun Zhou, Fandong Meng, Jinsong Su, Yidong Chen, and Jie Zhou. 2023c. HyperNetwork-based decoupling to improve model generalization for few-shot relation extraction. In EMNLP 2023. find
  43. Yunyi Zhang, Ruozhen Yang, Xueqiang Xu, Rui Li, Jinfeng Xiao, Jiaming Shen, and Jiawei Han. 2025c. Teleclass: Taxonomy enrichment and llm-enhanced hierarchical text classification with minimal supervision. Preprint, arXiv:2403.00165. — [arXiv:2403.00165](https://arxiv.org/abs/2403.00165) find
  44. Sizhe Zhou, Yu Meng, Bowen Jin, and Jiawei Han. 2024. Grasping the essentials: Tailoring large language models for zero-shot relation extraction. In EMNLP 2024. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2026-hcre-llm-based-hierarchical-classification/