Elamur Research⌕
☰
home / papers / 2026 / Memory-Guided Hard Data Augmentation for Multimodal Named Entity Recognition

Memory-Guided Hard Data Augmentation for Multimodal Named Entity Recognition

Paper · 2026 · Xinyu Liu, Kai fu, Yinghan Shi et al.

Summary

The paper targets data augmentation for Multimodal Named Entity Recognition (MNER), where images help disambiguate text. Existing augmentation methods apply the same random perturbations to every model. The authors show that this misses many errors: across HVPNeT, MKGformer and AMNet, roughly 30% of each model's hard examples are unique to that architecture.

They propose Memory-Guided Hard Data Augmentation, a five-stage pipeline:

  1. K-fold cross-validation finds the training samples the target model gets wrong.
  2. These failures go into a Memory Tree, where an LLM plus clustering groups individual mistakes into broader error patterns.
  3. Those patterns drive targeted generation of new text–image samples, using LLMs and diffusion models.
  4. The new samples are checked at the label, text and text–image levels.
  5. Hard samples whose augmentations fail these checks are sent back for another round of generation.

On Twitter-2015 and Twitter-2017, the framework consistently improves several MNER backbones and beats earlier augmentation methods. The authors frame this as a methodological advance in augmentation rather than an absolute state of the art. Its value lies in fixing each model's own blind spots instead of expanding data generically.

Key points

  • Model-specific errors: About 30% of hard instances are unique to each backbone (HVPNeT, MKGformer, AMNet). For HVPNeT, errors shared by all three models make up under 43% of its failures.

  • Error taxonomy: Rule-based coarse classification sorts errors into four types: entity hallucination, boundary detection failure, entity omission and type confusion. The LLM then abstracts these into finer patterns, such as image–text cue misalignment.

  • Ablation results (Twitter-2017, HVPNeT backbone): the full framework reaches 87.13 F1. Removing components lowers it as follows:

    Component removedF1
    Data filtering85.30 (largest drop, mainly in precision)
    Iterative augmentation86.42
    Model-aware mining86.52
    Memory Tree86.83

  • Filtering model: Llama-3-8B is reported as the filter LLM, with 83.35 F1 in a filtering-model comparison.

  • Limitations: Experiments cover only loosely aligned Twitter data. Hallucinations from the LLM or diffusion model can still introduce noise into the augmented data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China167 works · 20,987 citations🏢Huawei TechnologiesShenzhen · China75 works · 12,975 citations🎓Donghua UniversityShanghai · China2 works

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 657 worksNamed entity recognition · 200 works

Cite

Cite as Xinyu Liu, Kai fu, Yinghan Shi, Quanyou Chu, Ming Du, Hongya Wang, Xiaojun Meng, Jiansheng Wei, Yanghua Xiao and Bo Xu. 2026. Memory-Guided Hard Data Augmentation for Multimodal Named Entity Recognition. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1075

@inproceedings{liu2026memory,
  title = {{Memory-Guided Hard Data Augmentation for Multimodal Named Entity Recognition}},
  author = {Xinyu Liu, Kai fu, Yinghan Shi, Quanyou Chu, Ming Du, Hongya Wang, Xiaojun Meng, Jiansheng Wei, Yanghua Xiao and Bo Xu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1075},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1075}
}

References

26 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xiang Chen, Ningyu Zhang, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang, Luo Si, and Huajun Chen. 2022a. Hybrid transformer with multi-level fusion for multimodal knowledge graph completion. In Proceedings of the 45th international ACM SIGIR conference on research and development in inf… find
  2. Xiang Chen, Ningyu Zhang, Lei Li, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2022b. Good visual guidance makes a better extractor: Hierarchical visual prefix for multimodal entity and relation extraction. arXiv preprint arXiv:2205.03521. — [arXiv:2205.03521](https://arxiv.org/abs/2205.03521) find
  3. Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. 2019. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 113–123. find
  4. Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. 2020. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pages 702–703. find
  5. Yinglong Dai, Feng Gao, and Daojian Zeng. 2023. An alignment and matching network with hierarchical visual features for multimodal named entity and relation extraction. In International Conference on Neural Information Processing, pages 298–310. Springer. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
  7. Bosheng Ding, Chengwei Qin, Ruochen Zhao, Tianze Luo, Xinze Li, Guizhen Chen, Wenhan Xia, Junjie Hu, Anh Tuan Luu, and Shafiq Joty. 2024. Data augmentation using large language models: Data perspectives, learning paradigms and challenges. arXiv preprint arXiv:2403.02990. find
  8. Xiaoshuai Hao, Yi Zhu, Srikar Appalaraju, Aston Zhang, Wanqian Zhang, Bo Li, and Mu Li. 2023. Mixgen: A new multi-modal data augmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 379–389. find
  9. Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arxiv 2015. arXiv preprint arXiv:1508.01991.
  10. Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. arXiv preprint arXiv:1603.01360.
  11. Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2020. A survey on deep learning for named entity recognition. IEEE transactions on knowledge and data engineering, 34(1):50–70. find
  12. Ziyan Li, Jianfei Yu, Jia Yang, Wenya Wang, Li Yang, and Rui Xia. 2024. Generative multimodal data augmentation for low-resource multimodal named entity recognition. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 7336– 7345. find
  13. Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. Visual attention model for name tagging in multimodal social media. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1990–1999.
  14. Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. arXiv preprint arXiv:1603.01354.
  15. Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695. find
  16. Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, and Fangsheng Weng. 2021. Rpbert: a text-image relation propagation-based bert model for multimodal ner. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 13860–13868. find
  17. Xinyu Wang, Min Gui, Yong Jiang, Zixia Jia, Nguyen Bach, Tao Wang, Zhongqiang Huang, and Kewei Tu. 2022. Ita: Image-text alignments for multi-modal named entity recognition. In Proceedings of the 2022 conference of the North American chapter of the association for computational linguistics: human la… find
  18. Zaitian Wang, Pengfei Wang, Kunpeng Liu, Pengyang Wang, Yanjie Fu, Chang-Tien Lu, Charu C Aggarwal, Jian Pei, and Yuanchun Zhou. 2025. A comprehensive survey on data augmentation. IEEE Transactions on Knowledge and Data Engineering. find
  19. Bo Xu, Shizhou Huang, Ming Du, Hongya Wang, Hui Song, Yanghua Xiao, and Xin Lin. 2023. A unified visual prompt tuning framework with mixture-of-experts for multimodal information extraction. In International Conference on Database Systems for Advanced Applications, pages 544–554. Springer. find
  20. Bo Xu, Shizhou Huang, Chaofeng Sha, and Hongya Wang. 2022. Maf: A general matching and alignment framework for multimodal named entity recognition. In Proceedings of the fifteenth ACM international conference on web search and data mining, pages 1215–1223. find
  21. Bo Xu, Haiqi Jiang, Jie Wei, Hongyu Jing, Ming Du, Hui Song, Hongya Wang, and Yanghua Xiao. 2025. Enhancing multimodal named entity recognition through adaptive mixup image augmentation. In Proceedings of the 31st International Conference on Computational Linguistics, pages 1802–1812. find
  22. Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. Association for Computational Linguistics.
  23. Dong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, and Guodong Zhou. 2021. Multimodal graph fusion for named entity recognition with targeted visual guidance. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14347–14355. find
  24. Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. 2017. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412.
  25. Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named entity recognition in tweets. In Proceedings of the AAAI conference on artificial intelligence, volume 32. find
  26. Xiaoying Zhou, Yijia Zhang, Zhuang Wang, Mingyu Lu, and Xiaoxia Liu. 2024. Mafn: multi-level attention fusion network for multimodal named entity recognition. Multimedia Tools and Applications, 83(15):45047–45058. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-memory-guided-hard-data-augmentation/