Elamur Research⌕
☰
home / papers / 2026 / OrthoEdit: Principled and Stable Knowledge Editing via Orthogonal Subspace Projection

OrthoEdit: Principled and Stable Knowledge Editing via Orthogonal Subspace Projection

Paper · 2026 · Shanbao Qiao, Xuebing Liu, Akshat Gupta et al.

Summary

Sequential knowledge editing lets a large language model's stored facts be corrected or updated batch after batch without retraining. Over long edit sequences, however, new updates tend to disturb both the model's original knowledge and facts inserted in earlier rounds. AlphaEdit shields only pre-existing knowledge through null-space projection and protects earlier edits with a soft objective term. O-Edit relies on loss-level gradient regularization.

OrthoEdit replaces both approaches with exact projection. Each FFN weight update is forced into a subspace orthogonal to the original knowledge keys and to every previously edited key. The method works in three stages:

  • It progressively refines a shared null space that excludes directions used by past edits.
  • It extracts a compact principal subspace of the current target keys, which saves capacity for future edits.
  • It projects the update onto the orthogonal intersection of the two.

The authors prove that the constraints hold formally. Experiments on several models show better editing accuracy and preserved general abilities over thousands of batched edits, at a cost close to MEMIT and AlphaEdit. This makes long-running model maintenance more practical.

Key points

  • On GLUE-style general-capability tests with up to 5,000 edits (batch size 100):
    • PRUNE collapses by 2,000 edits and RECT by 3,000.
    • AlphaEdit breaks down at 5,000.
    • OrthoEdit stays near its original scores, with small drops only on SST and MMLU.
    • A 6,000-edit run with batch size 300 shows the same pattern.
  • Ablations on LLaMA3-8B and Qwen2.5-7B:
    • Progressive null space projection (PNS) alone already beats the AlphaEdit baseline.
    • Adding principal space extraction (PRI) and orthogonalization improves results further.
  • The SVD threshold τ2 controls a trade-off. Higher values improve locality, while lower values raise edit success but increase interference. An automatically selected τ2 gives balanced results.
  • Following MEMIT, knowledge to preserve is approximated by keys from 100,000 Wikipedia facts. The original-knowledge null space is initialized the same way as in AlphaEdit.
  • Efficiency:
    • Peak memory is similar to MEMIT and AlphaEdit.
    • Per-sample time falls between the two (e.g., 8.32 s vs. 7.84 s for MEMIT and 8.46 s for AlphaEdit).
    • Code is available at github.com/JoveReCode/OrthoEdit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States175 works · 227,440 citations🎓Jeonbuk National UniversityJeonju · South Korea2 works · 205 citations🎓Ulsan National Institute of Science and TechnologyUlsan · South Korea2 works

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Shanbao Qiao, Xuebing Liu, Akshat Gupta and Seung-Hoon Na. 2026. OrthoEdit: Principled and Stable Knowledge Editing via Orthogonal Subspace Projection. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl.a.700

@article{qiao2026orthoedit,
  title = {{OrthoEdit: Principled and Stable Knowledge Editing via Orthogonal Subspace Projection}},
  author = {Shanbao Qiao, Xuebing Liu, Akshat Gupta and Seung-Hoon Na},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.1162/tacl.a.700},
  url = {https://doi.org/10.1162/tacl.a.700}
}

References

59 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. TAC, 7(8):1. find
  2. Baolong Bi, Shenghua Liu, Yiwei Wang, Lingrui Mei, Hongcheng Gao, Junfeng Fang, and Xueqi Cheng. 2024. Struedit: Structured outputs enable the fast and accurate knowledge editing for large language models. arXiv preprint arXiv:2409.10132. — [arXiv:2409.10132](https://arxiv.org/abs/2409.10132) find
  3. Yuchen Cai and Ding Cao. 2024. O-edit: Orthogonal subspace editing for language model sequential editing. arXiv preprint arXiv:2410.11469. — [arXiv:2410.11469](https://arxiv.org/abs/2410.11469) find
  4. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. find
  5. Roi Cohen, Eden Biran, Ori Yoran, Amir Globerson, and Mor Geva. 2024. Evaluating the ripple effects of knowledge editing in language models. Transactions of the Association for Computational Linguistics, 12:283–298. find
  6. William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
  7. Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Xiang Wang, Xiangnan He, and Tat-seng Chua. 2024. Alphaedit: Null-space constrained knowledge editing for language models. arXiv preprint arXiv:2410.02355. find
  8. Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. Detecting hallucinations in large language models using semantic entropy. 630(8017):625–630. find
  9. Isabel O. Gallegos, Ryan A. Rossi, Joe Barrow, Md Mehrab Tanjim, Sungchul Kim, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, and Nesreen K. Ahmed. 2023. Bias and fairness in large language models: A survey.
  10. Hengrui Gu, Kaixiong Zhou, Xiaotian Han, Ninghao Liu, Ruobing Wang, and Xin Wang. 2024a. PokeMQA: Programmable knowledge editing for multi-hop question answering. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8069–8083, Bang… find
  11. Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. 2024b. Model editing can hurt general abilities of large language models. CoRR. find
  12. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  13. Akshat Gupta, Anurag Rao, and Gopala Anumanchipalli. 2024. Model editing at scale leads to gradual and catastrophic forgetting. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15202–15232, Bangkok, Thailand. Association for Computational Linguistics. find
  14. Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023. Aging with grace: Lifelong model editing with discrete key-value adaptors. In Advances in Neural Information Processing Systems. find
  15. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).
  16. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  17. Xiusheng Huang, Jiaxiang Liu, Yequan Wang, and Kang Liu. Reasons and solutions for the decline in model performance after editing. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  18. Xiusheng Huang, Yequan Wang, Jun Zhao, and Kang Liu. 2024. Commonsense knowledge editing based on free-text in LLMs. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 14870–14880, Miami, Florida, USA. Association for Computational Linguistics. find
  19. Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2023. Transformer-patcher: One mistake worth one neuron. arXiv preprint arXiv:2301.09785.
  20. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12).
  21. Houcheng Jiang, Junfeng Fang, Ningyu Zhang, Guojun Ma, Mingyang Wan, Xiang Wang, Xiangnan He, and Tat-seng Chua. 2025. Anyedit: Edit any knowledge encoded in language models. arXiv preprint arXiv:2502.05628. — [arXiv:2502.05628](https://arxiv.org/abs/2502.05628) find
  22. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 333–342.
  23. Qi Li and Xiaowen Chu. 2024. Can we continually edit language models? on the knowledge attenuation in sequential model editing. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5438–5455, Bangkok, Thailand. Association for Computational Linguistics. find
  24. Xiaopeng Li, Shasha Li, Shezheng Song, Jing Yang, Jun Ma, and Jie Yu. 2023. Pmet: Precise model editing in a transformer. arXiv preprint arXiv:2308.08742. find
  25. Jiateng Liu, Pengfei Yu, Yuji Zhang, Sha Li, Zixuan Zhang, Ruhi Sarikaya, Kevin Small, and Heng Ji. 2024. EVEDIT: Event-based knowledge editing for deterministic knowledge propagation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 4907–4926, Miami,… find
  26. Yifan Lu, Yigeng Zhou, Jing Li, Yequan Wang, Xuebo Liu, Daojing He, Fangming Liu, and Min Zhang. 2024. Knowledge editing with dynamic knowledge graphs for multi-hop question answering. arXiv preprint arXiv:2412.13782. — [arXiv:2412.13782](https://arxiv.org/abs/2412.13782) find
  27. Jun-Yu Ma, Hong Wang, Hao-Xiang Xu, Zhen-Hua Ling, and Jia-Chen Gu. 2024. Perturbation-restrained sequential model editing. CoRR, abs/2405.16821. — [arXiv:2405.16821](https://arxiv.org/abs/2405.16821) find
  28. Laurens van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86):2579–2605. find
  29. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35. find
  30. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022b. Mass editing memory in a transformer. arXiv preprint arXiv:2210.07229. find
  31. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2022a. Fast model editing at scale. In International Conference on Learning Representations. find
  32. Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. 2022b. Memory-based model editing at scale. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 15817–15831. PMLR. find
  33. Seyed Mahed Mousavi, Simone Alghisi, and Giuseppe Riccardi. 2024. Dyknow: dynamically verifying time-sensitive factual knowledge in llms. arXiv preprint arXiv:2404.08700. find
  34. Yasumasa Onoe, Michael Zhang, Eunsol Choi, and Greg Durrett. 2022. Entity cloze by date: What LMs know about unseen entities. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 693–702, Seattle, United States. Association for Computational Linguistics.
  35. OpenAI. 2023. Gpt-4 technical report. ArXiv, abs/2303.08774.
  36. Siyuan Qi, Bangcheng Yang, Kailin Jiang, Xiaobo Wang, Jiaqi Li, Yifan Zhong, Yaodong Yang, and Zilong Zheng. 2024. In-context editing: Learning knowledge from self-induced distributions. find
  37. Shanbao Qiao, Xuebing Liu, and Seung-Hoon Na. 2024a. COMEM: In-context retrieval-augmented mass-editing memory in large language models. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2333–2347, Mexico City, Mexico. Association for Computational Linguistics. find
  38. Shanbao Qiao, Xuebing Liu, and Seung-Hoon Na. 2024b. DistillMIKE: Editing distillation of massive in-context knowledge editing in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 7639–7654, Bangkok, Thailand. Association for Computational Linguisti… find
  39. Shanbao Qiao, Xuebing Liu, and Seung-Hoon Na. 2025. Wasserstein distance constraint and parameter sparsification for batched and iterative knowledge editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25019–25028. find
  40. Ankit Singh Rawat, Chen Zhu, Daliang Li, Felix Yu, Manzil Zaheer, Sanjiv Kumar, and Srinadh Bhojanapalli. 2021. Modifying memories in transformer models. In International Conference on Machine Learning (ICML), volume 2020. find
  41. Domenic Rosati, Robie Gonzales, Jinkun Chen, Xuemin Yu, Yahya Kayani, Frank Rudzicz, and Hassan Sajjad. 2024. Long-form evaluation of model editing. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volu… find
  42. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–…
  43. Chenmien Tan, Ge Zhang, and Jie Fu. 2024. Massive editing for large language model via meta learning. In The Twelfth International Conference on Learning Representations. find
  44. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
  45. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353– 355…
  46. Renzhi Wang and Piji Li. 2024. Memoe: Enhancing model editing with mixture of experts adaptors. arXiv preprint arXiv:2405.19086. — [arXiv:2405.19086](https://arxiv.org/abs/2405.19086) find
  47. Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625–641. find
  48. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa…
  49. Ziwen Xu, Shuxun Wang, Kewei Xu, Haoming Xu, Mengru Wang, Xinle Deng, Yunzhi Yao, Gou, Xi Chen, and Huajun Chen. 2024b. Instructedit: instruction-based knowledge editing for large language models. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI ’24… find
  50. Guozhou Zheng, Huajun Chen, and Ningyu Zhang. 2025. Easyedit2: An easy-to-use steering framework for editing large language models. find
  51. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei… find
  52. Wanli Yang, Fei Sun, Jiajun Tan, Xinyu Ma, Qi Cao, Dawei Yin, Huawei Shen, and Xueqi Cheng. 2025. The mirage of model editing: Revisiting evaluation in the wild.
  53. Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023. Editing large language models: Problems, methods, and opportunities. CoRR, abs/2305.13172. find
  54. Lang Yu, Qin Chen, Jie Zhou, and Liang He. 2024. Melo: Enhancing model editing with neuron-indexed dynamic lora. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19449–19457. find
  55. Mengqi Zhang, Xiaotian Ye, Qiang Liu, Pengjie Ren, Shu Wu, and Zhumin Chen. 2024a. Knowledge graph enhanced large language model editing. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22647–22662, Miami, Florida, USA. Association for Computational L… find
  56. Mengqi Zhang, Xiaotian Ye, Qiang Liu, Shu Wu, Pengjie Ren, and Zhumin Chen. 2025. Uncovering overfitting in large language model editing. In The Thirteenth International Conference on Learning Representations. find
  57. Ningyu Zhang, Bozhong Tian, Siyuan Cheng, Xiaozhuan Liang, Yi Hu, Kouying Xue, Yanjie Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei… find
  58. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223.
  59. Zexuan Zhong, Zhengxuan Wu, Christopher D Manning, Christopher Potts, and Danqi Chen. 2023. MQuAKE: Assessing knowledge editing in language models via multi-hop questions. arXiv preprint arXiv:2305.14795. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/qiao-2026-orthoedit-principled-stable-knowledge-editing/