Elamur Research⌕
☰
home / papers / 2026 / MERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linking

MERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linking

Paper · 2026 · Sathyanarayanan Ramamoorthy, Vishwa Shah, Simran Khanuja et al.

Summary

Entity linking maps an ambiguous mention to its referent in a knowledge base. It becomes difficult when the surrounding text is short, as in news headlines. Until now, multilingual linking research has used text alone. This paper asks whether accompanying images can resolve such ambiguity across languages. The authors define a multilingual multimodal entity linking (MMEL) task under an in-KB assumption and release MERLIN, an evaluation-only testbed. It pairs BBC headlines from the M3LS corpus with their images and links entity mentions to Wikidata. Five languages are covered: Hindi, Japanese, Indonesian, Vietnamese and Tamil. They were chosen for family and script diversity, speaker numbers, and the availability of crowdworkers. Screened Prolific annotators labelled the data in INCEpTION, and majority voting settled the final links. Baselines built on mGENRE, LLaMa-2 and Aya-23 show that the task remains hard even for the strongest systems. Adding visual input improves accuracy, especially for textually ambiguous mentions and for models with weak multilingual skills. This makes MERLIN a first benchmark for image-aided disambiguation in lower-resource languages.

Key points

  • MERLIN contains 5,000 headlines (1,000 per language) with 7,287 mentions and 2,480 unique Wikidata entities. A further 1,243 mentions could not be linked and are excluded from evaluation.
  • Each 100-title split was labelled by three annotators. Annotators first had to reach an F1 of at least 60% on an English screening task. Agreement was measured with Cohen's Kappa.
  • Entities are grouped into PER, Country, LOC, ORG, Event and MISC by traversing the Wikidata P31/P279 hierarchy.
  • Evaluated systems include mGENRE, a GEMEL-style approach with LLaMa-2-7B, and Aya-23 8B.
  • Machine-translating inputs into English raises scores in every language, with the largest gains for Tamil and Hindi, which points to an English bias in current models.
  • Stated limitations:
    • The data comes from a single genre (BBC news).
    • Annotators may make errors.
    • The baselines predict Wikipedia titles rather than Wikidata QIDs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,226 citations🏛️Defence Science and Technology AgencySingapore · Singapore2 works · 2 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 3Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 882 worksVision & language · 811 worksNamed entity recognition · 214 works

Cite

Cite as Sathyanarayanan Ramamoorthy, Vishwa Shah, Simran Khanuja, Zaid Sheikh, Shan Jie, Ann Chia, Shearman Chua and Graham Neubig. 2026. MERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linking. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.633

@article{ramamoorthy2026merlin,
  title = {{MERLIN: A Testbed for Multilingual Multimodal Entity Recognition and Linking}},
  author = {Sathyanarayanan Ramamoorthy, Vishwa Shah, Simran Khanuja, Zaid Sheikh, Shan Jie, Ann Chia, Shearman Chua and Graham Neubig},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.633},
  url = {https://doi.org/10.1162/tacl.a.633}
}

References

35 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7833–7845, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.630 ↗
  2. Omar Adjali, Romaric Besançon, Olivier Ferret, Hervé Le Borgne, and Brigitte Grau. 2020. Building a multimodal entity linking dataset from tweets. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4285–4292. Marseille, France. European Language Resources Association… find
  3. Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan Gomez, Phil Blunsom, Marzieh Fadaee, Ahmet Üstün,… — Aya 23: Open Weight Releases to Further Multilingual Progress
  4. Edoardo Barba, Luigi Procopio, and Roberto Navigli. 2022. ExtEnD: Extractive entity disambiguation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2478–2488, Dublin, Ireland. Association for Computational Linguistics. https:/… find
  5. Roi Blanco, Giuseppe Ottaviano, and Edgar Meij. 2015. Fast and space-efficient entity linking for queries. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, WSDM ’15, pages 179–188, New York, NY, USA. Association for Computing Machinery. https://doi.org/10 .114… find
  6. Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Sem… find
  7. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2019. Question answering by reasoning across documents with graph convolutional networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and…
  8. Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. arXiv preprint arXiv:2010.00904v3. find
  9. Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, and Fabio Petroni. 2022. Multilingual autoregressive entity linking. Transactions of the Association for Computational Linguistics, 10:274–290. https://doi.org… ↗
  10. Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022. Systematic inequalities in language technology performance across the world’s languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486–5505, Dublin, Irelan…
  11. Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020. Entities as experts: Sparse memory access with entity supervision. In Proceedings of the 2020 Conference on Jan A. Botha, Zifei Shan, and Daniel Gillick. 2020. Entity Linking in 100 Languages.
  12. Empirical Methods in Natural Language Processing (EMNLP), pages 4937–4951, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.400 ↗
  13. Tuan Lai, Heng Ji, and ChengXiang Zhai. 2022. Improving candidate retrieval with entity profile generation for Wikidata entity linking. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3696–3711, Dublin, Ireland. Association for Computational Linguistics. https://doi.org… ↗
  14. Xingyu Fu, Weijia Shi, Xiaodong Yu, Zian Zhao, and Dan Roth. 2020. Design challenges in low-resource cross-lingual entity linking. arXiv preprint arXiv:2005.00692. find
  15. Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-shot entity linking by reading entity descriptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3449–3460, Florence, Italy. Associatio… find
  16. Jingru Gan, Jinchang Luo, Haiwei Wang, Shuhui Wang, Wei He, and Qingming Huang. 2021. Multimodal entity linking: A new dataset and a baseline. In Proceedings of the 29th ACM International Conference on Multimedia, MM ’21, pages 993–1001, New York, NY, USA. Association for Computing Machinery. https:… find
  17. Mary L. McHugh. 2012. Interrater reliability: The kappa statistic. Biochemia Medica, 22(3): 276–282. https://doi.org/10.11613/BM.2012.031, PubMed: 23092060 ↗
  18. Stephen Guo, Ming-Wei Chang, and Emre Kiciman. 2013. To link or not to link? A study on end-to-end tweet entity linking. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1020–1030, Atlanta, Georgi… find
  19. Paul McNamee, James Mayfield, Dawn Lawrie, Douglas Oard, and David Doermann. 2011. Cross-language entity linking. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 255–263, Chiang Mai, Thailand. Asian Federation of Natural Language Processing. find
  20. Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, Xu Han, Yankai Lin, Jiao Xue, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. Large multilingual models pivot zero-shot multimodal learning across languages. arXiv preprint arXiv:2308.120…
  21. Seungwhan Moon, Leonardo Neves, and Vitor R. Carvalho. 2018. Multimodal named entity disambiguation for noisy social media posts. In Annual Meeting of the Association for Computational Linguistics. https://doi.org/10.18653/v1/P18-1186 ↗
  22. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. https://doi.org/10.18653/v1/2020 .acl-main.56… ↗
  23. Jan-Christoph Klie, Michael Bugert, Beto Boullosa, Richard Eckart de Castilho, and Iryna Gurevych. 2018. The INCEpTION platform: Machine-assisted and knowledge-oriented interactive annotation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pa…
  24. Mikhail Plekhanov, Nora Kassner, Kashyap Popat, Louis Martin, Simone Merello, Borislav Kozlovskii, Frédéric A. Dreyer, and Nicola Cancedda. 2023. Multilingual end to end entity linking. arXiv preprint arXiv:2306.08896v1. — [arXiv:2306.08896](https://arxiv.org/abs/2306.08896) find
  25. Senbao Shi, Zhenran Xu, Baotian Hu, and Min Zhang. 2024. Generative multimodal entity linking. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 7654–7665, Torino, Italia. ELRA and ICCL. find
  26. Avirup Sil and Radu Florian. 2016. One for all: Towards language independent named entity linking. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2255–2264, Berlin, Germany. Association for Computational Linguistics. https://… find
  27. Xuwu Wang, Junfeng Tian, Min Gui, Zhixu Li, Rui Wang, Ming Yan, Lihan Chen, and Yanghua Xiao. 2022. WikiDiverse: A multimodal entity linking dataset with diversified contextual topics and entity types. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume… find
  28. Shezheng Song, Shasha Li, Shan Zhao, Xiaopeng Li, Chengyu Wang, Jie Yu, Jun Ma, Tianwei Yan, Bin Ji, and Xiaoguang Mao. 2024. DWE+: Dual-way matching enhanced framework for multimodal entity linking. arXiv preprint arXiv:2404.04818v1. — [arXiv:2404.04818](https://arxiv.org/abs/2404.04818) find
  29. Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cristian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Bri… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  30. Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero-shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6397–6407, Online. Association for Computa…
  31. Shangyu Xing, Fei Zhao, Zhen Wu, Chunhui Li, Jianbing Zhang, and Xinyu Dai. 2023. DRIN: Dynamic relation interactive network for multimodal entity linking. Proceedings of the 31st ACM International Conference on Multimedia. https://doi.org/10.1145/3581783.3612575 ↗
  32. Zhenran Xu, Yulin Chen, Senbao Shi, and Baotian Hu. 2022. Enhancing entity linking with contextualized entity embeddings. In Natural Language Processing and Chinese Computing: 11th CCF International Conference, NLPCC 2022, Guilin, China, September 24–25, 2022, Proceedings, Part II, pages 228–239, Be… — [doi:10.1007/978-3-031-17189-5](https://doi.org/10.1007/978-3-031-17189-5) find
  33. Yash Verma, Anubhav Jangra, Raghvendra Verma, and Sriparna Saha. 2023. Large scale multi-lingual multi-modal summarization dataset. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3620–3632, Dubrovnik, Croatia. Association for Com… find
  34. Barry Yao, Sijia Wang, Yu Chen, Qifan Wang, Minqian Liu, Zhiyang Xu, Licheng Yu, and Lifu Huang. 2024. Ameli: Enhancing multimodal entity linking with fine-grained attributes. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Lo… find
  35. Dongsheng Wang, Xiaoqin Feng, Zeming Liu, and Chuan Wang. 2024. 2M-NER: Contrastive learning for multilingual and multimodal ner Li Zhang, Zhixu Li, and Qiang Yang. 2021. Attention-based multimodal entity linking with high-quality images. In Database Systems for Advanced Applications: 26th Internati… — [doi:10.1007/978-3-030-73197-7_35](https://doi.org/10.1007/978-3-030-73197-7_35) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ramamoorthy-2026-merlin-testbed-multilingual-multimodal-entity/