Elamur Research⌕
☰
home / papers / 2023 / ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction

ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction

Paper · 2023 · Jiabang He; Lei Wang; Yi Hu; Ning Liu; Hui Liu; Xing Xu; and Heng Tao Shen

Summary

The paper investigates whether large language models can perform document information extraction (DIE) from visually rich documents (receipts, forms) through in-context learning alone, without fine-tuning. Two obstacles are identified: LLMs cannot see images (modality gap), and they were not trained to reason about layout (task gap). The proposed framework, ICL-D3IE, converts documents into OCR text with bounding boxes, ordered by the XYCut algorithm. It then builds the prompt from three kinds of examples, drawn from training documents similar to the test ones:

  • Hard demonstrations: segments the model tends to mislabel.
  • Layout-aware demonstrations: examples where the LLM describes spatial relations between text boxes.
  • Formatting demonstrations: examples that make answers easy to parse.

Label mapping turns unnatural tag names into plain words, and the hard examples are refined iteratively. On FUNSD, CORD and SROIE, Davinci-003 and ChatGPT with ICL-D3IE match or beat fully fine-tuned models such as LayoutLMv3, especially under distribution shift. This suggests that few-shot prompting is a practical, low-annotation alternative for document understanding.

Key points

  • Benchmarks: FUNSD, CORD and SROIE, tested in both in-distribution (ID) and out-of-distribution (OOD) settings. The approach wins everywhere except the ID setting on CORD.
  • On SROIE, ICL-D3IE with GPT-3 reaches 97.88%, compared with 96.89% for LayoutLMv3 base fine-tuned on the full training data.
  • Ablation results (Davinci-003):
    • Removing hard demonstrations causes the largest loss.
    • Removing layout-aware demonstrations costs about 10 F1 on CORD but little on SROIE.
    • Removing label mapping notably hurts CORD.
  • With 1 to 12 demonstrations, both ICL-D3IE variants stay well ahead of LayoutLMv3 fine-tuned on the same few examples.
  • Code is released at github.com/MAEHCM/ICL-D3IE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Singapore Management UniversitySingapore · Singapore35 works · 2,457 citations🎓University of Electronic Science and Technology of ChinaChengdu · China38 works · 2,088 citations

Also on the byline: Beijing Forestry University.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 worksInformation Systems · 1,730 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksInformation extraction · 175 works

Cite

Cite as Jiabang He; Lei Wang; Yi Hu; Ning Liu; Hui Liu; Xing Xu; and Heng Tao Shen. 2023. ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction. arXiv:2303.05063.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2303.05063 · arXiv 2303.05063

@misc{xu2023icl,
  title = {{ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction}},
  author = {Jiabang He; Lei Wang; Yi Hu; Ning Liu; Hui Liu; Xing Xu; and Heng Tao Shen},
  year = {2023},
  doi = {10.48550/arxiv.2303.05063},
  eprint = {2303.05063},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2303.05063}
}

References

42 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  2. Haoyu Cao, Xin Li, Jiefeng Ma, Deqiang Jiang, Antai Guo, Yiqing Hu, Hao Liu, Yinsong Liu, and Bo Ren. Query-driven generative network for document information extraction in the wild. In Proceedings of the 30th ACM International Conference on Multimedia, pages 4261–4271, 2022. find
  3. Rui CAO, Roy Ka-Wei LEE, Wen-Haw CHONG, and Jing JIANG. Prompting for multimodal hateful meme classification. 2022. find
  4. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588, 2022. find
  5. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. ArXiv preprint, abs/2204.02311, 2022.
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT, pages 4171–4186, 2019.
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL, 2019.
  8. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey for in-context learning. arXiv preprint arXiv:2301.00234, 2022.
  9. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. 2022.
  10. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
  11. Zhangxuan Gu, Changhua Meng, Ke Wang, Jun Lan, Weiqiang Wang, Ming Gu, and Liqing Zhang. Xylayoutlm: Towards layout-aware multimodal networks for visually-rich document understanding. In CVPR, 2022. find
  12. Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven CH Hoi. From images to textual prompts: Zero-shot vqa with frozen large language models. arXiv preprint arXiv:2212.10846, 2022. — [arXiv:2212.10846](https://arxiv.org/abs/2212.10846) find
  13. Teakgyu Hong, DongHyun Kim, Mingi Ji, Wonseok Hwang, Daehyun Nam, and Sungrae Park. Bros: A pre-trained language model focusing on text and layout for better key information extraction from documents. In AAAI, 2022. find
  14. Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. Promptcap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.09699, 2022. — [arXiv:2211.09699](https://arxiv.org/abs/2211.09699) find
  15. Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking. arXiv preprint arXiv:2204.08387, 2022.
  16. Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar. ICDAR2019 Competition on Scanned Receipt OCR and Information Extraction. In ICDAR, pages 1516–1520, 2019. find
  17. Guillaume Jaume, H. K. Ekenel, and J. Thiran. FUNSD: A Dataset for Form Understanding in Noisy Scanned Documents. ICDARW, 2:1–6, 2019. find
  18. Guillaume Jaume, Hazim Kemal Ekenel, and Jean-Philippe Thiran. Funsd: A dataset for form understanding in noisy scanned documents. In ICDARW, 2019. find
  19. Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Conference on Computer Vision (ECCV), 2022.
  20. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. Neural architectures for named entity recognition. In NAACL HLT, 2016.
  21. Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei. Dit: Self-supervised pre-training for document image transformer. arXiv preprint arXiv:2203.02378, 2022. find
  22. Peizhao Li, Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Varun Manjunatha, and Hongfu Liu. Selfdoc: Self-supervised document representation learning. In CVPR, 2021. find
  23. Yulin Li, Yuxi Qian, Yuechen Yu, Xiameng Qin, Chengquan Zhang, Yan Liu, Kun Yao, Junyu Han, Jingtuo Liu, and Errui Ding. Structext: Structured text understanding with multimodal transformers. In ACM Multimedia, 2021. find
  24. Xiaojing Liu, Feiyu Gao, Qiong Zhang, and Huasha Zhao. Graph convolution for multimodal information extraction from visually rich documents. arXiv preprint arXiv:1903.11279, 2019. find
  25. Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209.14610, 2022.
  26. Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, 2021.
  27. John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 11…
  28. Seunghyun Park, Seung Shin, Bado Lee, Junyeop Lee, Jaeheung Surh, Minjoon Seo, and Hwalsuk Lee. Cord: A consolidated receipt dataset for post-ocr parsing. In Document modal pre-training for visually-rich document understanding. In ACL, 2021. [43] Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Y…
  29. Seunghyun Park, S. Shin, Bado Lee, Jihyo Lee, Jaeheung Surh, Minjoon Seo, and Hwal-Suk Lee. CORD: A Consolidated Receipt Dataset for Post-OCR Parsing. In Workshop on Document Intelligence at NeurIPS 2019, 2019. find
  30. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019.
  31. Omar Shaikh, Hongxin Zhang, William Held, Michael Bernstein, and Diyi Yang. On second thought, let’s not think step by step! bias and toxicity in zero-shot reasoning. arXiv preprint arXiv:2212.08061, 2022.
  32. Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, et al. Challenging big-bench tasks and whether chain-of-thought can solve them. ArXiv preprint, abs/2210.09261, 2022.
  33. Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. arXiv preprint arXiv:2106.13884, 2021.
  34. Boshi Wang, Sewon Min, Xiang Deng, Jiaming Shen, You Wu, Luke Zettlemoyer, and Huan Sun. Towards understanding chain-of-thought prompting: An empirical study of what matters. arXiv preprint arXiv:2212.10001, 2022.
  35. Jiapeng Wang, Lianwen Jin, and Kai Ding. Lilt: A simple yet effective language-independent layout transformer for structured document understanding. In ACL, 2022. find
  36. Lei Wang, Jiabang He, Xing Xu, Ning Liu, and Hui Liu. Alignment-enriched tuning for patch-level pre-trained document image models. arXiv preprint arXiv:2211.14777, 2022. — [arXiv:2211.14777](https://arxiv.org/abs/2211.14777) find
  37. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.
  38. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. CoRR, abs/2206.07682, 20…
  39. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS 2022), 2022. — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  40. Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. In LayoutLM: Pre-training of Text and Layout for Document Image Understanding, pages 1192– 1200, 2020.
  41. Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre-training of text and layout for document image understanding. In KDD, 2020.
  42. Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. LayoutLMv2: Multi- find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/he-2023-icl-d3ie-context-learning-diverse/