Elamur Research⌕
☰
home / papers / 2025 / Multilingual Knowledge Editing with Language-Agnostic Factual Neurons

Multilingual Knowledge Editing with Language-Agnostic Factual Neurons

Paper · 2025 · Xue Zhang, Yunlong Liang, Fandong Meng et al.

Summary

The paper tackles multilingual knowledge editing (MKE), which means changing a fact in an LLM so the correction holds in several languages at once. Existing locate-then-edit methods such as ROME, MEMIT and PMET, as well as LoRA fine-tuning, tend to lose accuracy when edits in different languages are applied together, because the edits interfere with each other. The authors first examine FFN activations in Llama-3.1-8B and Qwen2-7B. They use English PARAREL facts and Chinese machine translations of them, and they find a shared subset of neurons that responds to the same fact in both languages. They call these language-agnostic factual neurons (LAFNs). Their method, LU-LAFNs, uses paraphrases to locate these neurons for a given edit and then optimizes update values for them. Instead of rewriting model weights, it caches the updates under the aligned multilingual subject and applies them only when a query mentions that subject. On Bi-ZsRE and MzsRE, LU-LAFNs outperforms prior MKE methods. Bilingual editing no longer hurts results and even slightly improves them, which suggests that modeling cross-lingual links between facts is useful.

Key points

  • The neuron analysis drew 3,000 sentences per relation from PARAREL's 34 relations (about 100k sentences) and translated them into Chinese with Google Translate. Shared neurons cluster in the middle layers and the final layer, peaking at layer 12 in Llama-3.1-8B and layer 18 in Qwen2-7B.
  • On Bi-ZsRE with Llama-3.1-8B, joint en+zh editing scored 73.92 (en) and 67.29 (zh), slightly above monolingual editing. Baselines declined under joint editing, for example M-MEMIT dropped by 9.55 points on en.
  • The best configuration edits layers 11, 12, 13 and 31 with β = 0.1, which updates about 68% of neurons per layer and reaches an average of 74.91. Editing only the last layer performs poorly (33.43).
  • Updating just 5% of LAFNs per layer (β = 0.7) still scores 70.25, beating the strongest baseline, M-PMET, at 67.40.
  • In ablations, locating without paraphrases, editing all neurons, or editing random neurons each scored slightly lower on Llama-3.1-8B, Qwen2-7B and bloomz-7b1-mt. Code is available at github.com/XZhang00/LU-LAFNs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢TencentShenzhen · China134 works · 19,743 citations🎓Beijing Jiaotong UniversityBeijing · China22 works · 1,754 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 works

Cite

Cite as Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu and Jie Zhou. 2025. Multilingual Knowledge Editing with Language-Agnostic Factual Neurons. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{zhang2025multilingual,
  title = {{Multilingual Knowledge Editing with Language-Agnostic Factual Neurons}},
  author = {Xue Zhang, Yunlong Liang, Fandong Meng, Songming Zhang, Yufeng Chen, Jinan Xu and Jie Zhou},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.385/}
}

References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sunit Bhattacharya and Ondřej Bojar. 2023. Unveiling multilinguality in transformer models: Exploring language specificity in feed-forward networks. In Proceedings of the 6th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pages 120–126, Singapore. Association for Computat… find
  2. Yuheng Chen, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. 2023. Journey to the center of the knowledge neurons: Discoveries of language-independent knowledge neurons and degenerate knowledge neurons. Preprint, arXiv:2308.13198. — [arXiv:2308.13198](https://arxiv.org/abs/2308.13198) find
  3. In our approach, it is necessary to provide the Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, aligned multilingual knowledge to be edited and Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, et al. 2024. The llama 3 herd of models. Preprint, their corresponding multilingual subjects, which arXiv:2…
  4. Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. 2024. If the corresponding subjects are not annotated, ex-Model editing can hurt general abilities of large lan-isting LLMs can be utilized to identify the aligned guage models. Preprint, arXiv:2401.04700. m… — [arXiv:2401.04700](https://arxiv.org/abs/2401.04700) find
  5. Moreover, Weizhu Chen. 2021. Lora: Low-rank adaptation of the current method for determining whether a sub-large language models. Preprint, arXiv:2106.09685. ject exists in the cache adopts the exact-match ap-Aditi Khandelwal, Harman Singh, Hengrui Gu, Tian-proach, which is too strict. We will optim…
  6. Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein,… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  7. S. Kullback and R. A. Leibler. 1951. On Information and Sufficiency. The Annals of Mathematical Statistics, 22(1):79 – 86. find
  8. Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. 2024. Pmet: Precise model editing in a transformer. Preprint, arXiv:2308.08742. find
  9. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems, volume 35, pages 17359–17372. Curran Associates, Inc. find
  10. Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations. find
  11. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.… — Crosslingual Generalization through Multitask Finetuning
  12. Telmo Pires, Eva Schlinger, and Dan Garrette. 2019. How multilingual is multilingual BERT? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4996–5001, Florence, Italy. Association for Computational Linguistics.
  13. Nianwen Si, Hao Zhang, and Weiqiang Zhang. 2024. Mpn: Leveraging multilingual patch neuron for cross-lingual model editing. Preprint, arXiv:2401.03190. — [arXiv:2401.03190](https://arxiv.org/abs/2401.03190) find
  14. Zengkui Sun, Yijin Liu, Jiaan Wang, Fandong Meng, Jinan Xu, Yufeng Chen, and Jie Zhou. 2024. Outdated issue aware decoding for factual knowledge editing. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9282–9293, Bangkok, Thailand. Association for Computational Linguist… find
  15. Tianyi Tang, Wenyang Luo, Haoyang Huang, Dongdong Zhang, Xiaolei Wang, Xin Zhao, Furu Wei, and Ji-Rong Wen. 2024. Language-specific neurons: The key to multilingual capabilities in large language models. Preprint, arXiv:2402.16438.
  16. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… — Cross-Lingual Knowledge Editing in Large Language Models
  17. Weixuan Wang, Barry Haddow, and Alexandra Birch. 2023b. Retrieval-augmented multilingual knowledge editing. Preprint, arXiv:2312.13040.
  18. Yang Xu, Yutai Hou, Wanxiang Che, and Min Zhang. 2023. Language anisotropic cross-lingual model editing. In Findings of the Association for Computational Linguistics: ACL 2023, pages 5554–5569, Toronto, Canada. Association for Computational Linguistics. find
  19. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai D… find
  20. Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023. Editing large language models: Problems, methods, and opportunities. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 10222–10240, Singapore… find
  21. Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. How do large language models handle multilingualism? Preprint, arXiv:2402.18815.
  22. Zexuan Zhong, Zhengxuan Wu, Christopher D. Manning, Christopher Potts, and Danqi Chen. 2023. find
  23. Mquake: Assessing knowledge editing in language models via multi-hop questions. Preprint, arXiv:2305.14795. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2025-multilingual-knowledge-editing-language-agnostic/