Elamur Research⌕
☰
home / papers / 2026 / MPBoCo: Multimodal Prompt-based Boundary-enhanced Continual Framework for Joint Entity and Relation Extraction

MPBoCo: Multimodal Prompt-based Boundary-enhanced Continual Framework for Joint Entity and Relation Extraction

Paper · 2026 · Guanglu Sun, Xinyu Liu, Lili Liang et al.

Summary

Multimodal knowledge graphs must keep pace with social-media streams in which new entity and relation types keep appearing. Static multimodal extractors would need retraining on all past data. Existing continual approaches either store old examples, which raises storage and privacy concerns, or use pipelines that compound errors. The authors introduce a new task, abbreviated CMERJE, in which entities and relations are extracted jointly from multimodal inputs under a continual-learning regime, and they build a matching benchmark dataset. Their framework, MPBoCo, keeps a VLMO backbone frozen and learns, for each task, a set of image-text key-prompt pairs. For every input, the most relevant pairs are retrieved and injected into the backbone. Prompts from finished tasks are frozen so that later tasks do not overwrite them. Because global prompts tend to blur token-level span edges, a dual-branch fusion module adds an auxiliary text branch that preserves local syntactic continuity and supervises boundary decisions. MPBoCo beats the strongest baselines by 5.5 points in the 10-task setting and 7.2 points in the 5-task setting. It also comes close to a joint-training upper bound without replaying any historical data.

Key points

  • Task and resource: CMERJE is presented as the first dataset for continual multimodal joint extraction. It uses a class-incremental setup with disjoint label sets per task, and the targets are quintuples of subject, subject type, object, object type, and relation.
  • Prompt pool design: Prompt keys are initialized from type descriptions (subject–relation–object) encoded by the frozen text encoder. Key regularization and type-aware enhancement are also used, and ablations confirm both help.
  • Results: In the 5-task setting, MPBoCo reaches an average of 42.5 versus 35.3 for the best baseline. In the 10-task setting, its average is about 28.5.
  • Best configuration: Attaching prompts to 8 unimodal and 2 multimodal expert layers works best. Two multimodal prompt layers consistently outperform one.
  • Backbone and limits: Unfreezing the backbone improves learning of new tasks but increases forgetting. The method depends on VLMO's text and vision experts, and fine-grained boundary recognition remains a stated weakness.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China158 works · 28,841 citations

Also on the byline: Harbin University of Science and Technology.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 657 worksInformation extraction · 167 works

Cite

Cite as Guanglu Sun, Xinyu Liu, Lili Liang, Yang Yu, Fei Lang, Suxia Zhu and Ming Liu. 2026. MPBoCo: Multimodal Prompt-based Boundary-enhanced Continual Framework for Joint Entity and Relation Extraction. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1220

@inproceedings{sun2026mpboco,
  title = {{MPBoCo: Multimodal Prompt-based Boundary-enhanced Continual Framework for Joint Entity and Relation Extraction}},
  author = {Guanglu Sun, Xinyu Liu, Lili Liang, Yang Yu, Fei Lang, Suxia Zhu and Ming Liu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1220},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1220}
}

References

38 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xiang Chen, Jingtian Zhang, Xiaohan Wang, Ningyu Zhang, Tongtong Wu, Yuxiang Wang, Yongheng Wang, and Huajun Chen. 2024. Continual multimodal knowledge graph construction. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence (IJCAI), pages 6225– 6233. find
  2. Jian Cheng, Kaifang Long, Shuang Zhang, Tian Zhang, Lianbo Ma, Shi Cheng, and Yinan Guo. 2023. Text-image scene graph fusion for multi-modal named entity recognition. IEEE Transactions on Artificial Intelligence, 5(6):2828–2839. find
  3. Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. 2021. Template-based named entity recognition using bart. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1835–1845. Association for Computational Linguistics.
  4. Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. 2020. Orthogonal gradient descent for continual learning. In Proceedings of the Twenty-Third International Conference on Artificial Intelligence and Statistics (AISTATS), volume 108, pages 3762–3773. find
  5. Xu Han, Yi Dai, Tianyu Gao, Yankai Lin, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2020. Continual relation learning via episodic memory activation and reconsolidation. In Proceedings of the Fifty-Eighth Annual Meeting of the Association for Computational Linguistics (ACL), pages 6429–6440.
  6. Zhiqiang Hu, Xinyu Li, Xinyu Pan, Sijie Wen, and Jinsong Bao. 2025. A question answering system for assembly process of wind turbines based on multimodal knowledge graph and large language model. Journal of engineering design, 36(7-9):1093–1117. find
  7. Shizhou Huang, Bo Xu, Changqun Li, Jiabo Ye, and Xin Lin. 2024. Mner-mi: A multi-image dataset for multimodal named entity recognition in social media. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages… find
  8. Shubin Huang, Yi Cai, Li Yuan, and Jiexin Wang. 2025. A knowledge-enhanced network for joint multimodal entity-relation extraction. Information Processing & Management, 62(3):104033. find
  9. Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. 2023. Maple: Multi-modal prompt learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19113–19122. find
  10. James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, and 1 others. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS)… find
  11. Junlin Lee, Yequan Wang, Jing Li, and Min Zhang. 2024. Multimodal reasoning with multimodal knowledge graph. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10767–10782. find
  12. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582–… find
  13. Yujie Li, Xin Yang, Hao Wang, Xiangkun Wang, and Tianrui Li. 2024. Learning to prompt knowledge transfer for open-world continual learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 13700–13708. find
  14. Wanying Liang, Pasquale De Meo, Yong Tang, and Jia Zhu. 2024. A survey of multi-modal knowledge graphs: Technologies and trends. ACM Computing Surveys, 56(11):1–41. find
  15. Luping Liu, Meiling Wang, Mozhi Zhang, Linbo Qing, and Xiaohai He. 2022. Uamner: uncertainty-aware multimodal named entity recognition in social media posts. Applied Intelligence, 52(4):4109–4125. find
  16. Peipei Liu, Gaosheng Wang, Hong Li, Jie Liu, Yimo Ren, Hongsong Zhu, and Limin Sun. 2024. Multi-granularity cross-modal representation learning for named entity recognition on social media. Information Processing & Management, 61(1):103546. find
  17. Xinyu Liu, Guanglu Sun, Jing Jin, Fei Lang, and Suxia Zhu. 2026a. Citr: Context-driven implicit triple reasoning for joint multimodal entity-relation extraction. Information Processing & Management, 63(2):104388. find
  18. Xinyu Liu, Guanglu Sun, Fei Lang, and Suxia Zhu. 2025. Mpcl: Multimodal prompt learning for continual relation extraction with type-aware inter-modality alignment. Information Fusion, page 104025. find
  19. Xinyu Liu, Guanglu Sun, Lili Liang, Fei Lang, and Suxia Zhu. 2026b. Hitimi: Hierarchical type-driven inter-modality interaction framework for multimodal named entity recognition. Neurocomputing, 664:132131. find
  20. Arun Mallya and Svetlana Lazebnik. 2018. Packnet: Adding multiple tasks to a single network by iterative pruning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7765– 7773. find
  21. Qi Qin, Han Peng, Wenpeng Hu, Dongyan Zhao, and Bing Liu. 2021. Bns: building network structures dynamically for continual learning. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 20608–20620. find
  22. Reza Shokri and Vitaly Shmatikov. 2015. Privacy-preserving deep learning. In Proceedings of the Twenty-Second ACM SIGSAC conference on computer and communications security (ACM SIGSAC), pages 1310–1321. find
  23. James Seale Smith, Leonid Karlinsky, Vyshnavi Gutta, Paola Cascante-Bonilla, Donghyun Kim, Assaf Arbelle, Rameswar Panda, Rogerio Feris, and Zsolt Kira. 2023. Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning. In IEEE/CVF Conference on Computer Vision… find
  24. Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, and Fang-sheng Weng. 2021. Rpbert: a text-image relation propagation-based bert model for multimodal ner. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 13860–13868. find
  25. Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and William Yang Wang. 2019. Sentence embedding alignment for lifelong relation extraction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NA… find
  26. Jie Wang, Yan Yang, Keyu Liu, Zhiping Zhu, and Xiaorong Liu. 2023a. M3s: Scene graph driven multi-granularity multi-task learning for multi-modal ner. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 31:111–120. find
  27. Xinyi Wang, Zitao Wang, and Wei Hu. 2023b. Serial contrastive knowledge distillation for continual few-shot relation extraction. In Findings of the Association for Computational Linguistics (ACL Findings), pages 12693–12706. find
  28. Zhiyuan Wang, Xiaoyang Qu, Jing Xiao, Bokui Chen, and Jianzong Wang. 2024. Incprompt: Task-aware incremental prompting for rehearsal-free class-incremental learning. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7240–7244. find
  29. Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. 2022. Learning to prompt for continual learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 139–149. find
  30. Zhiwei Wu, Changmeng Zheng, Yi Cai, Junying Chen, Ho-fung Leung, and Qing Li. 2020. Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts. In Proceedings of the 28th ACM International Conference on Multimedia, pages 1038–1046. ACM. find
  31. Dianzhi Yu, Xinni Zhang, Yankai Chen, Aiwei Liu, Yifei Zhang, Philip S Yu, and Irwin King. 2024. Recent advances of multimodal continual learning: A comprehensive survey. arXiv. find
  32. Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3342—-3352. ACL.
  33. Li Yuan, Yi Cai, Jin Wang, and Qing Li. 2023. Joint multimodal entity-relation extraction based on edge-enhanced graph alignment network and word-pair relation tagging. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), volume 37, pages 11051–11059. find
  34. Li Yuan, Yi Cai, Jingyu Xu, Qing Li, and Tao Wang. 2025. A fine-grained network for joint multimodal entity-relation extraction. IEEE Transactions on Knowledge and Data Engineering, 37(1):1–14. find
  35. Dong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, and Guodong Zhou. 2021. Multimodal graph fusion for named entity recognition with targeted visual guidance. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14347–14355. find
  36. Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named entity recognition in tweets. In Proceedings of the AAAI conference on artificial intelligence, volume 32, pages 5674–5681. find
  37. Zhiheng Zhang, Daojian Zeng, and Xue Bai. 2024. Improving continual few-shot relation extraction Split T10 find
  38. Split Train Val Test 848 114 118 862 110 245 634 90 338 484 63 418 404 56 472 235 30 509 279 43 538 304 49 582 225 25 614 222 37 647 Train Val Test 1152 163 162 1087 135 321 913 133 443 706 100 556 639 86 647 *Table 6: Statistics of CMERJE under the 10-task continual setting.* *Table 7: Statistics o… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-mpboco-multimodal-prompt-based-boundary/