Elamur Research⌕
☰
home / papers / 2026 / E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition

E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition

Paper · 2026 · Meng Zhang, Jinzhong Ning, Xiaolong Wu et al.

Summary

Grounded Multimodal Named Entity Recognition (GMNER) requires finding entity mentions in a social media post, assigning their types, and linking each entity to a bounding box in the attached image. Earlier systems chain separate components, such as BERT-style taggers, object detectors and knowledge modules. Errors then propagate between stages, and the parts cannot be trained jointly. The authors present E2E-GMNER, which handles the whole task inside one LoRA-tuned multimodal LLM from the Qwen2.5-VL family. Given an instruction, the model first produces a chain-of-thought rationale covering visual cues and background knowledge, then outputs the entity records as text, including box coordinates. The rationales used for training are distilled from a larger teacher model. To make coordinate generation more stable, Gaussian Risk-Aware Box Perturbation (GRBP) replaces each single gold box with randomly jittered soft targets. On Twitter-GMNER and Twitter-FMNERG, the approach matches or beats strong baselines without external detectors or online knowledge sources. This suggests that a single generative model can take the place of complex multimodal extraction pipelines.

Key points

  • The authors describe E2E-GMNER as the first fully end-to-end GMNER framework. Each output is serialized as entity | type | [x1, y1, x2, y2], following a reasoning sequence.
  • GRBP randomly shifts the centre and scale of gold boxes, assigning lower probability to larger shifts. The best setting was β = γ = 0.03. It clearly raised Acc@0.5 (grounding recall), while GMNER F1 changed only marginally and MeanIoU dropped slightly.
  • Chain-of-thought teacher comparison: larger teachers produced better downstream results. Qwen2.5-VL-72B outperformed GPT-4o, likely because it shares a model family with the backbone. Every teacher tested still performed reasonably well.
  • Evaluation used the Twitter-GMNER and Twitter-FMNERG benchmarks. Code is available at github.com/Finch-coder/E2E-GMNER.
  • Limitations: the method is tailored to GMNER and has not been tested on relation or event extraction. Chain-of-thought supervision also adds teacher-inference cost.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Dalian University of TechnologyDalian · China20 works · 46 citations🎓Dalian Maritime UniversityDalian · China3 works · 20 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 811 worksNamed entity recognition · 214 works

Cite

Cite as Meng Zhang, Jinzhong Ning, Xiaolong Wu, Hongfei Lin and Yijia Zhang. 2026. E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1127

@inproceedings{zhang2026e2e,
  title = {{E2E-GMNER: End-to-End Generative Grounded Multimodal Named Entity Recognition}},
  author = {Meng Zhang, Jinzhong Ning, Xiaolong Wu, Hongfei Lin and Yijia Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1127},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1127}
}

References

19 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, and 1 others. 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923. — Qwen2.5-VL Technical Report
  2. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  3. Fan Li, Jianxing Yu, Jielong Tang, Wenqing Chen, Hanjiang Lai, Yanghui Rao, and Jian Yin. 2025. Answering complex geographic questions by adaptive reasoning with visual context and external commonsense knowledge. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguist… find
  4. Jinyuan Li, Han Li, Di Sun, Jiahao Wang, Wenkun Zhang, Zan Wang, and Gang Pan. 2024a. Llms as bridges: Reformulating grounded multimodal named entity recognition. In Findings of the Association for Computational Linguistics ACL 2024, pages 1302– 1318. find
  5. Ziyan Li, Jianfei Yu, Jia Yang, Wenya Wang, Li Yang, and Rui Xia. 2024b. Generative multimodal data augmentation for low-resource multimodal named entity recognition. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 7336– 7345. find
  6. Xinkui Lin, Yuhui Zhang, Yongxiu Xu, Kun Huang, Hongzhang Mu, Yubin Wang, Gaopeng Gou, Li Qian, Li Peng, Wei Liu, and 1 others. 2025. Makar: a multi-agent framework based knowledge-augmented reasoning for grounded multimodal named entity recognition. In Proceedings of the 2025 Conference on Empirica… find
  7. Jintao Liu, Chenglong Liu, and Kaiwen Wei. 2024. Multi-view prompt for fine-grained multimodal named entity recognition and grounding. In ECAI 2024, pages 2693–2700. IOS Press. find
  8. Ye Liu, Hui Li, Alberto Garcia-Duran, Mathias Niepert, Daniel Onoro-Rubio, and David S Rosenblum. 2019. Mmkg: multi-modal knowledge graphs. In European Semantic Web Conference, pages 459–474. Springer. find
  9. Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. Visual attention model for name tagging in multimodal social media. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1990–1999.
  10. Hyunjong Ok, Taeho Kil, Sukmin Seo, and Jaeho Lee. 2024. Scanner: Knowledge-enhanced approach for robust multi-modal named entity recognition of unseen entities. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn… find
  11. Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. 2023. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306.14824.
  12. Jielong Tang, Zhenxing Wang, Ziyang Gong, Jianxing Yu, Xiangwei Zhu, and Jian Yin. 2025a. Multi-grained query-guided set prediction network for grounded multimodal named entity recognition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25246–25254. find
  13. Jielong Tang, Yang Yang, Jianxing Yu, Zhen-Xing Wang, Haoyuan Liang, Liang Yao, and Jian Yin. 2025b. Unco: Uncertainty-driven collaborative framework of large and small models for grounded multimodal ner. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, page… find
  14. Jieming Wang, Ziyan Li, Jianfei Yu, Li Yang, and Rui Xia. 2023. Fine-grained multimodal named entity recognition and grounding with a generative framework. In Proceedings of the 31st ACM International Conference on Multimedia, pages 3934–3943. find
  15. Xinyu Wang, Min Gui, Yong Jiang, Zixia Jia, Nguyen Bach, Tao Wang, Zhongqiang Huang, and Kewei Tu. 2022. Ita: Image-text alignments for multi-modal named entity recognition. In Proceedings of the 2022 conference of the North American chapter of the association for computational linguistics: human la… find
  16. Ziqi Wang, Chen Zhu, Zhi Zheng, Xinhang Li, Tong Xu, Yongyi He, Qi Liu, Ying Yu, and Enhong Chen. 2024. Granular entity mapper: Advancing fine-grained multimodal named entity recognition and grounding. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 3211–3226. find
  17. Jianfei YU, Jing JIANG, Li YANG, and Rui XIA. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer.(2020). In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3342–3352.
  18. Jianfei Yu, Ziyan Li, Jieming Wang, and Rui Xia. 2023. Grounded multimodal named entity recognition on social media. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9141–9154. find
  19. Dong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, and Guodong Zhou. 2021. Multimodal graph fusion for named entity recognition with targeted visual guidance. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 14347–14355. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-e2e-gmner-end-end-generative/