Elamur Research⌕
☰
home / papers / 2025 / MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models

MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models

Paper · 2025 · Zihao Wei, Jingcheng Deng, Liang Pang et al.

Summary

Most knowledge editing research updates and evaluates a model in a single language. This paper asks whether such edits hold up across languages and in multi-hop reasoning. The authors built MLaKE, a benchmark covering English, Chinese, Japanese, French and German. They extracted fact chains from Wikidata using a hand-picked set of common relations, kept only single-hop facts present in all five languages, and then extended them into two-hop chains. ChatGPT proposed plausible counterfactual edit targets, expanded answer aliases, and wrote free-form and multiple-choice questions. Language experts checked the generated text.

Tests of parameter-modifying editors, including MEMIT and StableKE on Vicuna1.5 and Qwen1.5, show that these methods generalize poorly:

  • Edits made in English succeed far more often than edits in other languages.
  • Multi-hop questions are harder still.
  • Transfer works better between related languages than across language families.

Neuron-level analysis suggests that models store knowledge for different languages in largely separate parameters. This finding matters for anyone who needs to update facts in multilingual or lower-resource language deployments.

Key points

  • MLaKE contains 5,360 single-hop and 4,072 multi-hop questions, with more than 13K samples overall. The data is released at github.com/Hi-archers/MLaKE.
  • Fact chains are built from 43 common relations. The most frequent relations are country of citizenship (P27), country (P17) and continent (P30).
  • Single-hop items are aligned across all five languages. Multi-hop items are not aligned and are used to probe generalization and transfer.
  • The authors compared the 100 (MEMIT) or 200 (StableKE) most-changed neurons across languages. The overlap between languages always stayed below 20%, and it was higher among Indo-European languages.
  • The study covers only parameter-editing approaches. In-context editing was not evaluated.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China194 works · 23,906 citations🎓University of Chinese Academy of SciencesBeijing · China147 works · 8,480 citations🔬Institute of Computing TechnologyBeijing · China51 works · 3,411 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Zihao Wei, Jingcheng Deng, Liang Pang, Hanxing Ding, Huawei Shen and Xueqi Cheng. 2025. MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{wei2025mlake,
  title = {{MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models}},
  author = {Zihao Wei, Jingcheng Deng, Liang Pang, Hanxing Ding, Huawei Shen and Xueqi Cheng},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.301/}
}

References

37 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, P… — [arXiv:2407.20224](https://arxiv.org/abs/2407.20224) find
  2. Canyu Chen and Kai Shu. 2023. Combating misinformation in the age of llms: Opportunities and challenges. CoRR, abs/2311.05656. find
  3. Canyu Chen and Kai Shu. 2024. Can llm-generated misinformation be detected? In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open-Review.net. find
  4. Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2023. Evaluating the ripple effects of knowledge editing in language models. CoRR, abs/2307.12976. find
  5. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493– 8502, Dublin, Ireland. Association for Computatio… find
  6. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6491– 6506, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. find
  7. Hanxing Ding, Liang Pang, Zihao Wei, Huawei Shen, and Xueqi Cheng. 2024. Retrieve only when it needs: Adaptive retrieval augmentation for hallucination mitigation in large language models. CoRR, abs/2402.10612. — [arXiv:2402.10612](https://arxiv.org/abs/2402.10612) find
  8. Qingxiu Dong, Damai Dai, Yifan Song, Jingjing Xu, Zhifang Sui, and Lei Li. 2022. Calibrating factual knowledge in pretrained language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 5937–5947, Abu Dhabi, United Arab Emirates. Association for Computational Ling…
  9. Zev Handel. 2008. What is sino-tibetan? snapshot of a field and a language family in flux. Language and Linguistics Compass, 2(3):422–441. find
  10. Peter Hase, Mohit Bansal, Been Kim, and Asma Ghandeharioun. 2023. Does localization inform editing? surprising differences in causality-based localization vs. knowledge editing in language models. CoRR, abs/2301.04213. find
  11. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023a. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Proc…
  12. Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2023b. Transformer-patcher: One mistake worth one neuron. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net.
  13. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  14. Lin Jin-lian. 2004. The chinese characters and vocabulary in japanese. Shandong Foreign Languages Teaching Journal. find
  15. Brian D Joseph. 2005. The indo-european family—the linguistic evidence. History of the Greek Language from the beginnings up to later antiquity, pages 128– 134. find
  16. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), Vancouver, Canada, August 3-4, 2017, pages 333–342. Association for Computational L…
  17. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. CMMLU: measuring massive multitask language understanding in chinese. CoRR, abs/2306.09212.
  18. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In NeurIPS. find
  19. Kevin Meng, Arnab Sen Sharma, Alex J. Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net. find
  20. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D. Manning. 2022a. Fast model editing at scale. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net. find
  21. Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. 2022b. Memory-based model editing at scale. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 158… find
  22. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural…
  23. Anton Sinitsin, Vsevolod Plokhotnyuk, Dmitry V. Pyrkin, Sergei Popov, and Artem Babenko. 2020. Editable neural networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net. find
  24. Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. find
  25. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  26. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023a. Is chatgpt a good NLG evaluator? A preliminary study. CoRR, abs/2303.04048.
  27. Jiaan Wang, Yunlong Liang, Zengkui Sun, Yuxuan Cao, and Jiarong Xu. 2023b. Cross-lingual knowledge editing in large language models. CoRR, abs/2309.08952.
  28. Peng Wang, Ningyu Zhang, Xin Xie, Yunzhi Yao, Bozhong Tian, Mengru Wang, Zekun Xi, Siyuan Cheng, Kangwei Liu, Guozhou Zheng, and Huajun Chen. 2023c. Easyedit: An easy-to-use knowledge editing framework for large language models. CoRR, abs/2308.07269. find
  29. Weixuan Wang, Barry Haddow, and Alexandra Birch. 2023d. Retrieval-augmented multilingual knowledge editing. CoRR, abs/2312.13040.
  30. Yifan Wei, Xiaoyan Yu, Huanhuan Ma, Fangyu Lei, Yixuan Weng, Ran Song, and Kang Liu. 2023. Assessing knowledge editing in language models via relation perspective. CoRR, abs/2311.09053. — [arXiv:2311.09053](https://arxiv.org/abs/2311.09053) find
  31. Zihao Wei, Liang Pang, Hanxing Ding, Jingcheng Deng, Huawei Shen, and Xueqi Cheng. 2024. Stable knowledge editing in large language models. arXiv preprint arXiv:2402.13048. — [arXiv:2402.13048](https://arxiv.org/abs/2402.13048) find
  32. Shicheng Xu, Liang Pang, Mo Yu, Fandong Meng, Huawei Shen, Xueqi Cheng, and Jie Zhou. 2024. Unsupervised information refinement training of large language models for retrieval-augmented generation. CoRR, abs/2402.18150. find
  33. Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023. Editing large language models: Problems, methods, and opportunities. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10222–10240, Singapore… find
  34. Yue Zhang, Ming Zhang, Haipeng Yuan, Shichun Liu, Yongyao Shi, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Llmeval: A preliminary study on how to evaluate large language models. In Thirty-Eighth AAAI Conference on Artificial Intelligence, AAAI 2024, Thirty-Sixth Conference on Innovative Application… find
  35. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of l…
  36. Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. 2023. Mquake: Assessing knowledge editing in language models via multi-hop questions. CoRR, abs/2305.14795. find
  37. Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix X. Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. CoRR, abs/2012.00363. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wei-2025-mlake-multilingual-knowledge-editing-benchmark/