Elamur Research⌕
☰
home / papers / 2026 / ThinkLinker: From Low-Rank Interaction to Knowledge-Aware Verification for Multimodal Entity Linking

ThinkLinker: From Low-Rank Interaction to Knowledge-Aware Verification for Multimodal Entity Linking

Paper · 2026 · Yingyao Ma, Yuanyuan Zhou, Congyu Zhang et al.

Summary

ThinkLinker targets Multimodal Entity Linking (MEL), which maps a mention, given its sentence and image, to the correct knowledge-base entity. The authors identify two weaknesses in prior systems. They use separate, hand-designed modules for each kind of feature interaction, and they rely on one-way retrieval with no knowledge-based checking, which fails when context is sparse.

The framework has two stages. Stage one uses CLIP encoders for text and image and compresses local features with a Hybrid Pooling Gate. It then models joint dependencies across modalities and granularities with a Low-rank Fusion Tower, a factorized approximation of high-order tensor fusion that keeps parameter growth small. This produces a ranked list of candidates. Stage two feeds those candidates back to an LLM for multi-turn, dialogue-style verification. The LLM output enriches the mention's context, which is adaptively fused with the original representation and used to fine-tune the first-stage model.

On the WikiMEL and WikiDiverse benchmarks, the authors report that ThinkLinker beats all state-of-the-art baselines. This suggests that verification guided by candidates helps disambiguation when context is weak.

Key points

  • Stage one components are the Hybrid Pooling Gate (gated max, mean and attention pooling) and the Low-rank Fusion Tower (low-rank factors following Liu et al., 2018).
  • Rank k = 4 was chosen as the trade-off between accuracy and size. Larger ranks increased parameters sharply but changed H@1 by only about 1–2 points.
  • Most of the benefit from the knowledge-space/semantic-space (KS-SS) dialogue comes from the first round. The final setting uses 3 rounds for WikiMEL and 2 for WikiDiverse, since extra rounds risk noise or hallucination.
  • On WikiDiverse, even Qwen-2.5-1.5B as the verifier gave about 2% absolute gain over stage one, and larger LLMs added little. Figure labels suggest DeepSeek-V3.1 is the default backbone.
  • Limitations include LLM hallucination, extra latency and compute cost, and a verification step that uses text only. Code is released on GitHub (zhouyuanyu/ThinkLinker).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Southeast UniversityNanjing · China28 works · 354 citations🔬InsermParis · France6 works · 462 citations🎓Université de RennesRennes · France3 works

Also on the byline: Laboratoire Traitement du Signal et de l'Image.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Vision & language · 657 worksNamed entity recognition · 200 worksEfficient models & compression · 242 works

Cite

Cite as Yingyao Ma, Yuanyuan Zhou, Congyu Zhang, Yi Yuan, Jiasong Wu, Lotfi Senhadji and Huazhong Shu. 2026. ThinkLinker: From Low-Rank Interaction to Knowledge-Aware Verification for Multimodal Entity Linking. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1248

@inproceedings{ma2026thinklinker,
  title = {{ThinkLinker: From Low-Rank Interaction to Knowledge-Aware Verification for Multimodal Entity Linking}},
  author = {Yingyao Ma, Yuanyuan Zhou, Congyu Zhang, Yi Yuan, Jiasong Wu, Lotfi Senhadji and Huazhong Shu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1248},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1248}
}

References

33 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Omar Adjali, Romaric Besançon, Olivier Ferret, Hervé Le Borgne, and Brigitte Grau. 2020. Multimodal entity linking for tweets. In Advances in Information Retrieval, pages 463–478. find
  2. Yixin Cao, Lei Hou, Juanzi Li, and Zhiyuan Liu. 2018. Neural collective entity linking. In Proceedings of the 27th International Conference on Computational Linguistics, pages 675–686. find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  4. Yotam Eshel, Noam Cohen, Kira Radinsky, Shaul Markovitch, Ikuya Yamada, and Omer Levy. 2017. Named entity disambiguation for noisy text. In Proceedings of the 21st Conference on Computational Natural Language Learning, pages 58–68. find
  5. Matthew Francis-Landau, Greg Durrett, and Dan Klein. 2016. Capturing semantic similarity for entity linking with convolutional neural networks. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 125…
  6. Faegheh Hasibi, Krisztian Balog, and Svein Erik Bratsberg. 2016. Exploiting entity linking in queries for entity retrieval. In Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval, pages 209–218. find
  7. Pengfei Luo, Tong Xu, Che Liu, Suojuan Zhang, Linli Xu, Minglei Li, and Enhong Chen. 2024. Bridging gaps in content and knowledge for multimodal entity linking. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 9311–9320. find
  8. Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pa… find
  9. Pengfei Luo, Tong Xu, Shiwei Wu, Chen Zhu, Linli Xu, and Enhong Chen. 2023. Multi-grained multimodal interaction network for entity linking. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 1583–1594. find
  10. Zhiwei Hu, Victor Gutiérrez-Basulto, Ru Li, and Jeff Z. Pan. 2025a. Multi-level matching network for multimodal entity linking. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Volume 1, pages 508–519. find
  11. Zhiwei Hu, Víctor Gutiérrez-Basulto, Zhiliang Xiang, Ru Li, and Jeff Z. Pan. 2025b. Multi-level mixture of experts for multimodal entity linking. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Volume 2, pages 979– 990. find
  12. Fengmei Jin, Wen Hua, Thomas Zhou, Jiajie Xu, Matteo Francia, Maria E. Orlowska, and Xiaofang Zhou. 2022. Trajectory-based spatiotemporal entity linking. IEEE Transactions on Knowledge and Data Engineering, 34(9):4499–4513. find
  13. Juyeon Kim, Geon Lee, Taeuk Kim, and Kijung Shin. 2025. Kgmel: Knowledge graph-enhanced multimodal entity linking. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 3015–3019. find
  14. Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021. ViLT: Vision-and-language transformer without convolution or region supervision. In Proceedings of the 38th International Conference on Machine Learning, pages 5583–5594.
  15. Qi Liu, Yongyi He, Tong Xu, Defu Lian, Che Liu, Zhi Zheng, and Enhong Chen. 2024. Unimel: A unified framework for multimodal entity linking with large language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, pages 1909–1919. find
  16. Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, AmirAli Bagher Zadeh, and Louis-Philippe Morency. 2018. Efficient low-rank multimodal fusion with modality-specific factors. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2247–…
  17. Jinghui Lu, Yang Xu, Bin Liang, Ruifeng Xu, Lin Gui, and Yulan He. 2025. Vp-mel: Visual prompts guided multimodal entity linking. In Findings of the Association for Computational Linguistics: ACL 2025, pages 17122–17137. find
  18. Yingyao Ma, Yifan Xue, Jiasong Wu, Lotfi Senhadji, Huazhong Shu, and Jian Yang. 2025. Multimodal entity linking with dynamic modality selection and interactive prompt learning. IEEE Transactions on Knowledge and Data Engineering, 37(9):5467–5480. find
  19. Seungwhan Moon, Leonardo Neves, and Vitor Carvalho. 2018. Multimodal named entity disambiguation for noisy social media posts. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2000–2008. find
  20. Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In Proceedings of the 38th Internatio…
  21. Chenwei Ran, Wei Shen, Jianbo Gao, Yuhan Li, Jianyong Wang, and Yantao Jia. 2023. Learning entity linking features for emerging entities. IEEE Transactions on Knowledge and Data Engineering, 35(7):7088–7102. find
  22. Wei Shen, Yuwei Yin, Yang Yang, Jiawei Han, Jianyong Wang, and Xiaojie Yuan. 2022. Toward tweet entity linking with heterogeneous information networks. IEEE Transactions on Knowledge and Data Engineering, 34(12):6003–6017. find
  23. Senbao Shi, Zhenran Xu, Baotian Hu, and Min Zhang. 2024. Generative multimodal entity linking. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, pages 7654–7665. find
  24. Peng Wang, Jiangheng Wu, and Xiaohang Chen. 2022a. Multimodal entity linking with gated hierarchical fusion and contrastive training. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 938–948. find
  25. Sijia Wang, Alexander Hanbo Li, Henghui Zhu, Sheng Zhang, Pramuditha Perera, Chung-Wei Hang, Jie Ma, William Yang Wang, Zhiguo Wang, Vittorio Castelli, Bing Xiang, and Patrick Ng. 2023. Benchmarking diverse-modal entity linking with generative models. In Findings of the Association for Computational… find
  26. Xuwu Wang, Junfeng Tian, Min Gui, Zhixu Li, Rui Wang, Ming Yan, Lihan Chen, and Yanghua Xiao. 2022b. WikiDiverse: A multimodal entity linking dataset with diversified contextual topics and entity types. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages… find
  27. Junshuang Wu, Richong Zhang, Yongyi Mao, Hongyu Guo, Masoumeh Soflaei, and Jinpeng Huai. 2020a. Dynamic graph convolutional networks for entity linking. In Proceedings of The Web Conference 2020, pages 1149–1159. find
  28. Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020b. Scalable zero-shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 6397– 6407.
  29. Shangyu Xing, Fei Zhao, Zhen Wu, Chunhui Li, Jianbing Zhang, and Xinyu Dai. 2023. Drin: Dynamic relation interactive network for multimodal entity linking. In Proceedings of the 31st ACM International Conference on Multimedia, pages 3599–3608. find
  30. Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016. Joint learning of the embedding of words and entities for named entity disambiguation. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pages 250–259.
  31. Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic parsing via staged query graph generation: Question answering with knowledge base. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natur…
  32. Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Memory fusion network for multi-view sequential learning. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial In… find
  33. Li Zhang, Zhixu Li, and Qiang Yang. 2021. Attention-based multimodal entity linking with high-quality images. In Proceedings of the 26th International Conference on Database Systems for Advanced Applications, pages 533–548. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2026-thinklinker-low-rank-interaction-knowledge/