Elamur Research⌕
☰
home / papers / 2025 / BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learning

BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learning

Paper · 2025 · Ercong Nie, Bo Shao, Mingyang Wang et al.

Summary

Knowledge editing (KE) updates facts stored in large language models without retraining. Most gradient-free work is monolingual, and it is unclear whether an edit made in English carries over to questions asked in other languages. The authors build BMIKE-53, a benchmark for cross-lingual in-context knowledge editing (IKE) in 53 languages. It merges zsRE, CounterFact and WikiFactDiff (WFD) into one English JSON format, which GPT-4o then translates into 52 further languages. Each item pairs an edited fact with four query types: reliability, generality, locality and portability. The authors test zero-shot, one-shot and two 8-shot prompting setups. Larger models do better, and so do demonstrations matched to the query type, especially for locality and portability queries. Script matters more than language family: languages written in non-Latin scripts lag behind. Much of this gap comes from language confusion, where the model replies in English instead of the target language. The resource matters because it lets researchers measure parameter-free fact updating across many languages, including closed models where weights cannot be changed.

Key points

  • Test set sizes: zsRE has 743 items, CounterFact 1,031 and WFD 784.
    • The language selection follows MLAMA and BMLAMA.
    • Portability queries for zsRE and CounterFact come from Yao et al. (2023).
    • For WFD, the authors derived portability queries by one-hop reasoning over a knowledge graph built from the dataset.
  • Prompting setups: zero-shot, one-shot, 8-shot mixed demonstrations covering several query types, and 8-shot metric-specific demonstrations aimed at a single query type.
  • Language properties (Lang2Vec analysis):
    • Syntactic, phonological and geographic closeness to English correlates positively with scores, mainly for locality and portability queries.
    • Genetic relatedness (language family) shows no meaningful effect.
  • Script effect: Grouping languages by script (Latin vs. non-Latin) and family (Indo-European vs. not) shows that non-Latin languages perform worse whatever their family. This is relevant for Cyrillic-script languages.
  • Language confusion: Error analysis with Llama3.1-8B on Chinese queries shows correct answers given in English, which count as failures. Code and data are available at github.com/ercong21/MultiKnow.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations🎓Ludwig-Maximilians-Universität MünchenMunich · Germany107 works · 56,478 citations🎓Technical University of MunichMunich · Germany55 works · 48,474 citations🎓Munich Center for Machine LearningMunich · Germany57 works · 1,094 citations🔬Helmholtz Center for Information SecuritySaarbrücken · Germany14 works · 1,172 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Ercong Nie, Bo Shao, Mingyang Wang, Zifeng Ding, Helmut Schmid and Hinrich Schuetze. 2025. BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learning. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.acl-long.798

@inproceedings{nie2025bmike,
  title = {{BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learning}},
  author = {Ercong Nie, Bo Shao, Mingyang Wang, Zifeng Ding, Helmut Schmid and Hinrich Schuetze},
  booktitle = {Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2025},
  doi = {10.18653/v1/2025.acl-long.798},
  url = {https://doi.org/10.18653/v1/2025.acl-long.798}
}

References

28 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Himanshu Beniwal, Kowsik D, and Mayank Singh. 2024. Cross-lingual editing in multilingual language models. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2078–2128, St. Julian’s, Malta. Association for Computational Linguistics. find
  2. Boxi Cao, Hongyu Lin, Xianpei Han, Le Sun, Lingyong Yan, Meng Liao, Tong Xue, and Jin Xu. 2021. Knowledgeable or educated guess? revisiting language models as knowledge bases. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint…
  3. Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2024. Evaluating the Ripple Effects of Knowledge Editing in Language Models. Transactions of the Association for Computational Linguistics, 12:283–298. find
  4. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493– 8502, Dublin, Ireland. Association for Computatio… find
  5. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6491– 6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  6. Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. 2022. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257– 273.
  7. Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  8. Nora Kassner, Philipp Dufter, and Hinrich Schütze. 2021. Multilingual lama: Investigating knowledge in multilingual pretrained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3250–3258.
  9. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2021. Fast model editing at scale. In International Conference on Learning Representations. find
  10. Hichem Ammar Khodja, Frédéric Bechet, Quentin Brabant, Alexis Nasr, and Gwénolé Lecorvé. 2024. Wikifactdiff: A large, realistic, and temporally adaptable dataset for atomic factual knowledge update in causal language models. In Proceedings of the 2024 Joint International Conference on Computational… find
  11. Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. 2022. Memory-based model editing at scale. In International Conference on Machine Learning, pages 15817–15831. PMLR. find
  12. Viet Lai, Nghia Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Nguyen. 2023. ChatGPT beyond English: Towards a comprehensive evaluation of large language models in multilingual learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages…
  13. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333–342, Vancouver, Canada. Association for Computational Linguistics.
  14. Patrick Littell, David R. Mortensen, Ke Lin, Katherine Kairis, Carlisle Turner, and Lori Levin. 2017. URIEL and lang2vec: Representing languages as typological, geographical, and phylogenetic vectors. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational…
  15. Kelly Marchisio, Wei-Yin Ko, Alexandre Berard, Théo Dehaze, and Sebastian Ruder. 2024. Understanding and mitigating language confusion in LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6653– 6677, Miami, Florida, USA. Association for Computatio…
  16. Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems. find
  17. Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations. find
  18. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Ab…
  19. Ercong Nie, Shuzhou Yuan, Bolei Ma, Helmut Schmid, Michael Färber, Frauke Kreuter, and Hinrich Schütze. 2024. Decomposed prompting: Unveiling multilingual linguistic structure knowledge in english-centric large language models. Preprint, arXiv:2402.18397. find
  20. Jirui Qi, Raquel Fernández, and Arianna Bisazza. 2023. Cross-lingual consistency of factual knowledge in multilingual language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10650–10666.
  21. Denny Vrandečić and Markus Krötzsch. 2014. Wikidata: a free collaborative knowledgebase. Communications of the ACM, 57(10):78–85.
  22. Jiaan Wang, Yunlong Liang, Zengkui Sun, Yuxuan Cao, and Jiarong Xu. 2023. Cross-lingual knowledge editing in large language models. arXiv preprint arXiv:2309.08952.
  23. Peng Wang, Ningyu Zhang, Bozhong Tian, Zekun Xi, Yunzhi Yao, Ziwen Xu, Mengru Wang, Shengyu Mao, Xiaohan Wang, Siyuan Cheng, Kangwei Liu, Yuansheng Ni, Guozhou Zheng, and Huajun Chen. 2024a. EasyEdit: An easy-to-use knowledge editing framework for large language models. In Proceedings of the 62nd An… find
  24. Weixuan Wang, Barry Haddow, and Alexandra Birch. 2024b. Retrieval-augmented multilingual knowledge editing. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 335–354, Bangkok, Thailand. Association for Computational Linguistics.
  25. Zihao Wei, Jingcheng Deng, Liang Pang, Hanxing Ding, Huawei Shen, and Xueqi Cheng. 2024. Mlake: Multilingual knowledge editing benchmark for large language models. arXiv preprint arXiv:2404.04990.
  26. Yang Xu, Yutai Hou, Wanxiang Che, and Min Zhang. 2023. Language anisotropic cross-lingual model editing. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5554–5569, Toronto, Canada. Association for Computational Linguistics. find
  27. Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023. Editing large language models: Problems, methods, and opportunities. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10222–10240. system: Y… find
  28. Hanning Zhang, Shizhe Diao, Yong Lin, Yi R Fung, Qing Lian, Xingyao Wang, Yangyi Chen, Heng Ji, and Tong Zhang. 2023. R-tuning: Teaching large language models to refuse unknown questions. arXiv preprint arXiv:2311.09677. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nie-2025-bmike-53-investigating-cross-lingual/