Elamur Research⌕
☰
home / papers / 2026 / Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs

Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs

Paper · 2026 · Wenhao Yu, Zhicong Lu, Bo Lv et al.

Summary

Editing many facts in an LLM at once tends to work less well as the batch grows. The authors trace this to semantic representation entanglement: edits that share concepts or sentence templates, such as many "capital of X" facts, land in overlapping regions of hidden space and interfere with each other. The effect is stronger across languages. A controlled pilot study on LLaMA-3-8B shows MEMIT and AlphaEdit losing accuracy on such entangled batches.

Their method, Orthogonal Representation Editing (ORE), keeps the backbone frozen and builds on ReFT. It works in three steps:

  • It collects hidden states from unrelated but structurally similar prompts drawn from ZsRE and CounterFact.
  • It runs PCA on these states to estimate a general semantic subspace.
  • It removes each edit vector's projection onto that subspace, so updates carry only fact-specific directions.

A gated non-linear representation head then learns where to intervene and how much to inject. The authors report that ORE beats existing editors in batch and cross-lingual settings. This matters for multilingual models, where shared semantics otherwise spread edits into non-target languages.

Key points

  • Pilot data: Gemini 3 generated 200 "capital" samples (100 French, 100 Chinese) with identical syntax. A control set of 100 random ZsRE items was translated into French and Chinese.
  • MEMIT editing success on LLaMA-3-8B fell from 86% on random samples to 71% (French), 64% (Chinese) and 56% (mixed Fr–Zh). AlphaEdit showed a similar drop.
  • The edit vector is the difference between hidden states for the target and the predicted declarative statement. It is then orthogonalized against the top-k PCA basis U_gs.
  • The gated head combines an MLP bottleneck (DOWN_PROJ/UP_PROJ with SiLU), RMSNorm and a dynamic position gate, replacing ReFT's linear, manually positioned interventions.
  • Limitations: experiments cover only 7B–8B models, standard editing settings and QA-style accuracy. Code is available at github.com/YVVH/ORE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢TencentShenzhen · China163 works · 21,032 citations🎓University of Chinese Academy of SciencesBeijing · China188 works · 8,744 citations🎓Tianjin UniversityTianjin · China52 works · 1,128 citations🎓Chongqing UniversityChongqing · China14 works · 318 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Wenhao Yu, Zhicong Lu, Bo Lv, Fangyin Ma, Kaiwen Wei, Shihao Yang and Nayu Liu. 2026. Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1892

@inproceedings{yu2026orthogonal,
  title = {{Orthogonal Representation Editing: Decoupling Semantic Entanglement in Batch Knowledge Editing of LLMs}},
  author = {Wenhao Yu, Zhicong Lu, Bo Lv, Fangyin Ma, Kaiwen Wei, Shihao Yang and Nayu Liu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1892},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1892}
}

References

24 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2023. Transformer-patcher: One mistake worth one neuron. arXiv preprint arXiv:2301.09785.
  2. Hervé Abdi and Lynne J Williams. 2010. Principal component analysis. Wiley interdisciplinary reviews: computational statistics, 2(4):433–459. find
  3. Himanshu Beniwal, D Kowsik, and Mayank Singh. 2024. Cross-lingual editing in multilingual language models. In Findings of the Association for Computational Linguistics: EACL 2024, pages 2078–2128. find
  4. Houcheng Jiang, Junfeng Fang, Ningyu Zhang, Guojun Ma, Mingyang Wan, Xiang Wang, Xiangnan He, and Tat-seng Chua. 2025a. Anyedit: Edit any knowledge encoded in language models. arXiv preprint arXiv:2502.05628. — [arXiv:2502.05628](https://arxiv.org/abs/2502.05628) find
  5. Baolong Bi, Shenghua Liu, Lingrui Mei, Yiwei Wang, Junfeng Fang, Pengliang Ji, and Xueqi Cheng. 2025. Decoding by contrasting knowledge: Enhancing large language model confidence on edited facts. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo… find
  6. Houcheng Jiang, Junfeng Fang, Tianyu Zhang, Baolong Bi, An Zhang, Ruipeng Wang, Tao Liang, and Xiang Wang. 2025b. Neuron-level sequential editing for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16678…
  7. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. arXiv preprint arXiv:1706.04115. null-space constraints. In Findings of the Association for Computational Linguistics: ACL 2025, pages 8796–8810, Vienna, Austria. Association for…
  8. Qi Li and Xiaowen Chu. 2025. AdaEdit: Advancing continuous knowledge editing for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4127–4149, Vienna, Austria. Association for Computational Linguistics. find
  9. Changyue Wang, Weihang Su, Qingyao Ai, Yujia Zhou, and Yiqun Liu. 2025. Decoupling reasoning and knowledge injection for in-context knowledge editing. arXiv preprint arXiv:2506.00536. find
  10. Tianci Liu, Ruirui Li, Yunzhe Qi, Hui Liu, Xianfeng Tang, Tianqi Zheng, Qingyu Yin, Monica Xiao Cheng, Jun Huan, Haoyu Wang, and Jing Gao. 2025. Unlocking efficient, scalable, and continual knowledge editing with basis-level representation fine-tuning. In The Thirteenth International Conference on L… find
  11. Jun-Yu Ma, Hong Wang, Hao-Xiang Xu, Zhen-Hua Ling, and Jia-Chen Gu. 2025. Perturbation-restrained sequential model editing. In The Thirteenth International Conference on Learning Representations. find
  12. Ben Mann, Nick Ryder, Melanie Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, and 1 others. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 1(3):3.
  13. Jiaan Wang, Yunlong Liang, Zengkui Sun, Yuxuan Cao, Jiarong Xu, and Fandong Meng. 2024a. Cross-lingual knowledge editing in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11676–11686.
  14. Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. 2024b. Knowledge editing for large language models: A survey. ACM Computing Surveys, 57(3):1–37.
  15. Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2024. ReFT: Representation fine-tuning for language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  16. Kevin Meng, David Bau, Alex J Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in GPT. In Advances in Neural Information Processing Systems. find
  17. Haoyu Xu, Pengxiang Lan, Enneng Yang, Guibing Guo, Jianzhe Zhao, Linying Jiang, and Xingwei Wang. 2025. Knowledge decoupling via orthogonal projection for lifelong editing of large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1:… find
  18. Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations. find
  19. Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023. Editing large language models: Problems, methods, and opportunities. arXiv preprint arXiv:2305.13172. find
  20. Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. 2022. Memory-based model editing at scale. In International Conference on Machine Learning, pages 15817–15831. PMLR. find
  21. Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363. find
  22. David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. 2019. Experience replay for continual learning. Advances in neural information processing systems, 32. find
  23. Marco Scialanga, Thibault Laugel, Vincent Grari, and Marcin Detyniecki. 2025. SAKE: Steering activations for knowledge editing. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15966–15978, Vienna, Austria. Association for Comp… find
  24. Wei Sun, Tingyu Qu, Mingxiao Li, Jesse Davis, and Marie-Francine Moens. 2025. Mitigating negative interference in multilingual knowledge editing through find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2026-orthogonal-representation-editing-decoupling-semantic/