Elamur Research⌕
☰
home / papers / 2023 / Chain-of-Thought Prompt Distillation for Multimodal Named Entity Recognition and Multimodal Relation Extraction

Chain-of-Thought Prompt Distillation for Multimodal Named Entity Recognition and Multimodal Relation Extraction

Paper · 2023 · Feng Chen and Yujian Feng

Summary

MNER and MRE use images attached to text, such as social posts, reviews and news, to help identify entities and relations. Both tasks require nontrivial reasoning about language and about how images relate to text. Earlier systems pull extra context from Wikipedia or sample databases, but that context is often off-domain, vague or misleading.

Chen and Feng instead use a large language model as the knowledge source. For each sample, they query the LLM with chain-of-thought prompts that explain nouns, the full sentence and the image–text link, with images first converted to captions by BLIP2. The prompts also create augmented variants by changing style, swapping entities or imagining the image.

A compact student model is then trained with conditional prompt distillation. Its predictions from text alone plus a learnable prompt are aligned with predictions from text, image and generated knowledge. As a result, inference needs neither the image nor an LLM call.

The authors report state-of-the-art results on five MNER/MRE datasets, along with better interpretability, data efficiency and cross-domain transfer.

Key points

  • CoT knowledge has two parts. Multi-grain prompts cover noun, sentence and multimodality. Data-augmentation prompts cover style, entity and image, and the authors claim this is the first multimodal augmentation used for MNER/MRE.
  • Training aligns two input views: [TXT] + [IMG] + [Knowledge] (teacher-like) and [TXT] + [Prompt] (student). This distills the LLM's reasoning into the learnable prompt.
  • The method is motivated by the limits of standard distillation. Many LLMs are reachable only through an API or give unstructured outputs, and they perform poorly on NER/RE on their own.
  • Compared systems include retrieval-based KB-NER and MoRe, which uses CLIP ViT-B/32 kNN image retrieval. Case studies show the method correctly tags hard spans such as "North and South Korea" and Twitter handles.
  • The provided excerpt omits the result tables, so dataset names and exact scores are not listed here. The authors note that output quality depends on how capable the LLM is.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Ant GroupHangzhou · China42 works · 1,131 citations

Also on the byline: Nanjing University of Posts and Telecommunications.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 5Research subjects and languages the work is about.Reasoning · 1,426 worksVision & language · 657 worksInformation extraction · 167 worksNamed entity recognition · 200 worksEfficient models & compression · 242 works

Cite

Cite as Feng Chen and Yujian Feng. 2023. Chain-of-Thought Prompt Distillation for Multimodal Named Entity Recognition and Multimodal Relation Extraction. arXiv:2306.14122.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.14122 · arXiv 2306.14122

@misc{chen2023chain,
  title = {{Chain-of-Thought Prompt Distillation for Multimodal Named Entity Recognition and Multimodal Relation Extraction}},
  author = {Feng Chen and Yujian Feng},
  year = {2023},
  doi = {10.48550/arxiv.2306.14122},
  eprint = {2306.14122},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.14122}
}

References

34 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, C… — Language Models are Few-Shot Learners
  2. Jiaao Chen, Zhenghui Wang, Ran Tian, Zichao Yang, and Diyi Yang. 2020. Local additivity based data augmentation for semi-supervised ner. arXiv preprint arXiv:2010.01677. find
  3. Shuguang Chen, Gustavo Aguilar, Leonardo Neves, and Thamar Solorio. 2021. Data augmentation for cross-domain named entity recognition. arXiv preprint arXiv:2109.01758. find
  4. Xiang Chen, Ningyu Zhang, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang, Luo Si, and Huajun Chen. 2022. Hybrid transformer with multi-level fusion for multimodal knowledge graph completion. arXiv preprint arXiv:2205.02357. — [arXiv:2205.02357](https://arxiv.org/abs/2205.02357) find
  5. Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. 2021. Knowledge distillation: A survey. IJCV, 129:1789–1819. find
  6. Ruobing Xie Xu Han Zhiyuan Liu Fen Lin Leyu Lin Maosong Sun Kai Zhang, Yuan Yao. 2021. Open hierarchical relation extraction. In NAACL. find
  7. Yu Su Kai Zhang, Bernal Jiménez Gutiérrez. 2023. Aligning instruction tasks unlocks large language models as zero-shot relation extractors. In Findings of ACL.
  8. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. ArXiv, abs/2205.11916.
  9. Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. 2023a. Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness. arXiv preprint arXiv:2304.11633. find
  10. Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597.
  11. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. find
  12. Xiaonan Li, Hang Yan, Xipeng Qiu, and Xuanjing Huang. 2020. Flat: Chinese ner using flat-lattice transformer. arXiv preprint arXiv:2004.11795.
  13. Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. Visual attention model for name tagging in multimodal social media. In ACL, pages 1990–1999.
  14. Junyu Lu, Dixiang Zhang, and Pingjian Zhang. 2022. Flat multi-modal interaction transformer for named entity recognition. arXiv preprint arXiv:2208.11039. find
  15. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763.
  16. Lin Sun, Kai Zhang, Fule Ji, and Zhenhua Yang. 2019. TOI-CNN: a solution of information extraction on Chinese insurance policy. In NAACL. find
  17. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  18. Xinyu Wang, Jiong Cai, Yong Jiang, Pengjun Xie, Kewei Tu, and Wei Lu. 2022a. Named entity and relation extraction with multi-modal retrieval. In EMNLP. find
  19. Xinyu Wang, Min Gui, Yong Jiang, Zixia Jia, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021a. Ita: Image-text alignments for multi-modal named entity recognition. arXiv preprint arXiv:2112.06482. find
  20. Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021b. Improving named entity recognition by external context retrieving and cooperative learning. arXiv preprint arXiv:2105.03654.
  21. Xuwu Wang, Junfeng Tian, Min Gui, Zhixu Li, Jiabo Ye, Ming Yan, and Yanghua Xiao. 2022b. Prompt-based entity-related visual clue extraction and integration for multimodal named entity recognition. In International Conference on Database Systems for Advanced Applications, pages 297–305. find
  22. Xuwu Wang, Jiabo Ye, Zhixu Li, Junfeng Tian, Yong Jiang, Ming Yan, Ji Zhang, and Yanghua Xiao. 2022c. Cat-mner: Multimodal named entity recognition with knowledge-refined cross-modal attention. In ICME, pages 1–6. find
  23. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Huai hsin Chi, Quoc Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. ArXiv, abs/2201.11903.
  24. Linzhi Wu, Pengjun Xie, Jie Zhou, Meishan Zhang, Ma Chunping, Guangwei Xu, and Min Zhang. 2022. Robust self-augmentation for named entity recognition with meta reweighting. In NAACL, pages 4049– 4060. find
  25. Bo Xu, Shizhou Huang, Chaofeng Sha, and Hongya Wang. 2022. Maf: A general matching and alignment framework for multimodal named entity recognition. In WSDM, pages 1215–1223. find
  26. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601. find
  27. Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. ACL.
  28. Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579–5588. find
  29. Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named entity recognition in tweets. In AAAI, volume 32. find
  30. Fei Zhao, Chunhui Li, Zhen Wu, Shangyu Xing, and Xinyu Dai. 2022. Learning from different text-image pairs: A relation-enhanced graph convolutional network for multimodal ner. In MM, pages 3983–3992. find
  31. Changmeng Zheng, Junhao Feng, Ze Fu, Yiru Cai, Qing Li, and Tao Wang. 2021a. Multimodal relation extraction with efficient graph alignment. MM. find
  32. Changmeng Zheng, Zhiwei Wu, Junhao Feng, Ze Fu, and Yi Cai. 2021b. Mnre: A challenge multimodal dataset for neural relation extraction with visual evidence in social media posts. In ICME, pages 1–6. find
  33. Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022a. Conditional prompt learning for vision-language models. In CVPR, pages 16816– 16825. find
  34. Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. 2022b. Learning to prompt for vision-language models. IJCV, 130(9):2337–2348. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2023-chain-thought-prompt-distillation-multimodal/