Elamur Research⌕
☰
home / papers / 2026 / HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering

HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering

Paper · 2026 · Joongmin Shin, Gyuho Shim, Jeongbae Park et al.

Summary

The paper tackles retrieval-augmented question answering over large collections of long, multi-page PDFs. Such documents mix prose with tables and figures, and systems often fail in two ways: they pick the wrong document from thousands of candidates, or they return evidence fragments that stay disconnected. Standard text chunking discards structural cues, while page-image retrieval wastes context on irrelevant content.

HiKEY uses the document's hierarchy as a primary retrieval signal. Offline, a Document Hierarchical Parsing (DHP) module from M3DocDep rebuilds each PDF as a tree of text, table and figure blocks, each tagged with its section path. Document-level and section-level index cards are then built from this tree. Figures and tables without captions borrow context from a preceding text node.

At query time, retrieval runs coarse to fine:

  • Hybrid lexical/dense routing first narrows the corpus to candidate documents.
  • Sections inside those documents are then ranked by fusing lexical, dense-text and dense-visual scores.
  • Finally, a token-budgeted evidence subgraph is packed, favouring sibling and semantically related units.

On M3DocVQA and FRAMES, HiKEY beats chunk- and page-based baselines in both retrieval and end-to-end QA. This matters for enterprise document search under tight context limits.

Key points

  • Reported gains over text and multimodal RAG baselines: retrieval improves by 4.5–12.9 points, and end-to-end QA (EM/ANLS/ROUGE-L/METEOR) improves by 3.7–6.8 points.
  • On the strict All metric for FRAMES, HiKEY scores 46.2 versus 34.8 for MultiDocFusion. It also stays best at every Reader budget from 0.5K to 2K tokens.
  • Ablations:
    • Exposing Title/Header hierarchy as a separate index field raises the score from 84.6 to 88.6.
    • Removing routing drops it to 81.0.
    • Adding the visual-dense signal moves Avg R@10 from 88.2 to 88.6.
  • Two-level card index: Doc_card (title plus all section paths) handles global routing, and Sec_card holds a section's multimodal units. Section ranking uses MaxSim scoring.
  • Limitations noted by the authors:
    • Offline parsing adds overhead.
    • Results depend on OCR and layout quality.
    • Benefit is smaller for weakly structured files.
    • All results come from a single fixed Reader with a 16K budget.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Korea UniversitySeoul · South Korea26 works · 9,490 citations🎓Konkuk UniversitySeoul · South Korea5 works · 396 citations🎓Korea UniversityTokyo · Japan3 works

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 3Research subjects and languages the work is about.Vision & language · 657 worksQuestion answering · 441 worksInformation retrieval & search · 473 works

Cite

Cite as Joongmin Shin, Gyuho Shim, Jeongbae Park, Jaehyung Seo and Heuiseok Lim. 2026. HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.818

@inproceedings{shin2026hikey,
  title = {{HiKEY: Hierarchical Multimodal Retrieval for Open-Domain Document Question Answering}},
  author = {Joongmin Shin, Gyuho Shim, Jeongbae Park, Jaehyung Seo and Heuiseok Lim},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.818},
  url = {https://doi.org/10.18653/v1/2026.acl-long.818}
}

References

45 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Chunsheng Wu, Huajie Tan, Chunyuan Li, Jing Yang, Jie Yu, Xiyao Wang, Bin Qin, Yumeng Wang, Zizhen Yan, Ziyong Feng, Ziwei Liu, Bo Li, and Jiankang Deng. 2025. Llava-onevision-1.5: Fully… — [arXiv:2509.23661](https://arxiv.org/abs/2509.23661) find
  2. Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the ACL workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65–72.
  3. Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marcal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. 2019. Scene text visual question answering. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4291–4301. find
  4. Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, and Mohit Bansal. 2025. M3docvqa: Multi-modal multi-page multi-document understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 6178– 6188. find
  5. André V. Duarte, João DS Marques, Miguel Graça, Miguel Freire, Lei Li, and Arlindo L. Oliveira. 2024. LumberChunker: Long-form narrative document segmentation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 6473–6486, Miami, Florida, USA. Association for Computationa… find
  6. Manuel Faysse, Hugues Sibille, Tony Wu, Bilel Omrani, Gautier Viaud, CELINE HUDELOT, and Pierre Colombo. 2025. Colpali: Efficient document retrieval with vision language models. In The Thirteenth International Conference on Learning Representations. find
  7. Yair Feldman and Ran El-Yaniv. 2019. Multi-hop paragraph retrieval for open-domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pages 2296–2309, Florence, Italy.
  8. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-augmented generation for large language models: A survey. Preprint, arXiv:2312.10997.
  9. J. Ge, Steve Sun, Joseph Owens, Victor Galvez, O. Gol-ogorskaya, Jennifer C Lai, Mark J Pletcher, and Ki Lai. 2023. Development of a liver disease-specific large language model chat interface using retrieval augmented generation. medRxiv. find
  10. Hongyu Gong, Yelong Shen, Dian Yu, Jianshu Chen, and Dong Yu. 2020. Recurrent chunking mechanisms for long-text machine reading comprehension. pages 6751–6761. find
  11. Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V. Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. Preprint, arXiv:2402.07630.
  12. Seongtae Hong, Joong Min Shin, Jaehyung Seo, Taemin Lee, Jeongbae Park, Cho Man Young, Byeongho Choi, and Heuiseok Lim. 2024. Intelligent predictive maintenance RAG framework for power plants: Enhancing QA with StyleDFS and domain specific instruction tuning. In Proceedings of the 2024 Conference on… find
  13. Chelsi Jain, Yiran Wu, Yifan Zeng, Jiale Liu, Shengyu Dai, Zhenwen Shao, Qingyun Wu, and Huazheng Wang. 2025. SimpleDoc: Multi-Modal document understanding with Dual-Cue page retrieval and iterative refinement. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing…
  14. CheonSu Jeong. 2023. A study on the implementation of generative ai services using an enterprise data-based llm application architecture. Adv. Artif. Intell. Mach. Learn., 3:1588–1618. find
  15. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, pages 39–48, New York, NY, USA. Associat…
  16. Satyapriya Krishna, Kalpesh Krishna, Anhad Mohananey, Steven Schwarcz, Adam Stambler, Shyam Upadhyay, and Manaal Faruqui. 2025. Fact, fetch, and reason: A unified evaluation of retrieval-augmented generation. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associa… find
  17. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2021. Retrieval-augmented generation for knowledge-intensive nlp tasks. Preprint, arXiv:2005.11401.
  18. Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281.
  19. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  20. Hao Liu, Zhengren Wang, Xi Chen, Zhiyu Li, Feiyu Xiong, Qinhan Yu, and Wentao Zhang. 2025. HopRAG: Multi-hop reasoning for logic-aware retrieval-augmented generation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 1897–1913, Vienna, Austria. Association for Computation… find
  21. Jiefeng Ma, Jun Du, Pengfei Hu, Zhenrong Zhang, Jianshu Zhang, Huihui Zhu, and Cong Liu. 2023. Hrdoc: dataset and baseline method toward hierarchical reconstruction of document structures. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on… find
  22. Yubo Ma, Jinsong Li, Yuhang Zang, Xiaobao Wu, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Jiaqi Wang, Yixin Cao, and Aixin Sun. 2025. Towards storage-efficient visual document retrieval: An empirical study on reducing patch-level embeddings. In Findings of the Association for Computational Lin… find
  23. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to Information Retrieval. Cambridge University Press. find
  24. Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed S. Nassar, and Peter Staar. 2022. Doclaynet: A large human-annotated dataset for document-layout segmentation. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’22, pages 3743–3751, New York, NY, USA.… find
  25. Renyi Qu, Ruixuan Tu, and Forrest Sheng Bao. 2025. Is semantic chunking worth the computational cost? In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2155–2177, Albuquerque, New Mexico. Association for Computational Linguistics. find
  26. Qwen Team. 2024. Qwen2.5-vl technical report. arXiv preprint.
  27. Johannes Rausch, Octavio Martinez, Fabian Bissig, Ce Zhang, and Stefan Feuerriegel. 2021. Docparser: Hierarchical document structure parsing from renderings. Proceedings of the AAAI Conference on Artificial Intelligence, 35:4328–4338. find
  28. Johannes Rausch, Gentiana Rashiti, Maxim Gusev, Ce Zhang, and Stefan Feuerriegel. 2023. Dsg: An end-to-end document structure generator. find
  29. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389. find
  30. Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D. Manning. 2024. Raptor: Recursive abstractive processing for tree-organized retrieval. Preprint, arXiv:2401.18059.
  31. Joongmin Shin, Chanjun Park, Jeongbae Park, Jaehyung Seo, and Heuiseok Lim. 2025. MultiDocFusion : Hierarchical and multimodal chunking pipeline for enhanced RAG on long industrial documents. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 20996–21015… find
  32. Joongmin Shin, Jeongbae Park, Jaehyung Seo, and Heuiseok Lim. 2026. M3DocDep: Multi-modal, multi-page, multi-document dependency chunking with large vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). find
  33. R. Smith. 2007. An overview of the tesseract ocr engine. In Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), volume 2, pages 629–633. find
  34. Ryota Tanaka, Taichi Iki, Taku Hasegawa, Kyosuke Nishida, Kuniko Saito, and Jun Suzuki. 2025. Vdocrag: Retrieval-augmented generation over visually-rich documents. pages 24827–24837. find
  35. Rubèn Tito, Dimosthenis Karatzas, and Ernest Valveny. 2023. Hierarchical multimodal transformers for multi-page docvqa. Preprint, arXiv:2212.05935. — [arXiv:2212.05935](https://arxiv.org/abs/2212.05935) find
  36. Prashant Verma. 2025. S2 chunking: A hybrid framework for document segmentation through integrated spatial and semantic analysis. Preprint, arXiv:2501.05485. — [arXiv:2501.05485](https://arxiv.org/abs/2501.05485) find
  37. Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, Zhaokai Wang, Zhe Chen, Hongjie Zhang, Ganlin Yang, Haomin Wang, Qi Wei, Jinhui Yin, Wenhao Li, Erfei Cui, Guanzhou Chen, Zichen Ding, Changyao Tian, Zhenyu Wu, Jingjing Xie,… — InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency
  38. Xixi Wu, Yanchao Tan, Nan Hou, Ruiyang Zhang, and Hong Cheng. 2025. MoLoRAG: Bootstrapping document understanding via multi-modal logic-aware retrieval. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 14035–14056, Suzhou, China. Association for Comput…
  39. Hangdi Xing, Changxu Cheng, Feiyu Gao, Zirui Shao, Zhi Yu, Jiajun Bu, Qi Zheng, and Cong Yao. 2024a. Dochienet: A large and diverse dataset for document hierarchy parsing. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  40. Hangdi Xing, Changxu Cheng, et al. 2024b. Dochienet: A large and diverse dataset for document hierarchy parsing. In EMNLP. find
  41. Howard Yen, Tianyu Gao, Jinhyuk Lee, and Danqi Chen. 2023. Moqa: Benchmarking multi-type open-domain question answering. In Proceedings of the Third DialDoc Workshop on Document-grounded Dialogue and Conversational Question Answering, pages 8–29. find
  42. Antonio Jimeno Yepes, Yao You, Jan Milczek, Sebastian Laverde, and Renyu Li. 2024. Financial report chunking for effective retrieval augmented generation. Preprint, arXiv:2402.05131. — [arXiv:2402.05131](https://arxiv.org/abs/2402.05131) find
  43. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 11975–11986.
  44. Jihao Zhao, Zhiyuan Ji, Zhaoxin Fan, Hanyu Wang, Simin Niu, Bo Tang, Feiyu Xiong, and Zhiyu Li. 2025. MoC: Mixtures of text chunking learners for retrieval-augmented generation system. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  45. Xu Zhong, Elaheh ShafieiBavani, and Antonio Jimeno Yepes. 2020. Image-based table recognition: data, model, and evaluation. In European Conference on Computer Vision (ECCV), pages 564–580. Springer. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shin-2026-hikey-hierarchical-multimodal-retrieval-open/