Elamur Research⌕
☰
home / papers / 2026 / SpecEdit: A Spectral Approach for Multi-Round Knowledge Editing

SpecEdit: A Spectral Approach for Multi-Round Knowledge Editing

Paper · 2026 · Junxian Liu, Zikun Deng, Xin Wu et al.

Summary

Repeated batch editing of a large language model tends to wear down quality over time. Newly inserted facts are written less reliably, paraphrased queries generalize worse, and neighbouring knowledge gets disturbed. The authors study this in locate-then-edit methods such as MEMIT and AlphaEdit. In these methods, the closed-form update adds the second-order statistics of the model's original knowledge (K0) to those of earlier edits (Kp) and then inverts the sum.

A spectral analysis on GPT2-XL with CounterFact shows that the two sources behave very differently. Intrinsic knowledge spreads its energy over many directions, while edit history concentrates it in a few dominant ones, so simply summing them skews the resulting update.

SpecEdit addresses this in three stages:

  • it isolates the editing-critical subspace (spectral decoupling);
  • it restores information lost for both sources (compensation);
  • it recombines the refined parts into a new closed-form solution (spectral fusion).

The module attaches to existing solvers without altering their optimization. It gives consistent improvements across several LLMs and editing methods, offering an interpretable path toward more stable lifelong editing.

Key points

  • The spectrum of Cp = KpKpᵀ shows three abrupt drops, and the position of the second drop scales with the number of edited facts (near i ≈ 500 for 500 edits). This point is used to split Cp into an editing-critical part (Cp1) and a weak tail (Cp2), and C0 is decomposed by projecting it onto the Cp1 subspace.
  • SpecEdit works as a plug-and-play preprocessing and enhancement step for MEMIT and AlphaEdit, evaluated on GPT2-XL, GPT-J and LLaMA3.
  • In the ablations on GPT2-XL (dim(Kp) = 6400), compensation alone lowers efficacy and generalization while helping specificity and fluency. Adding decoupling turns it into a net gain. On GPT-J, the combined gains begin to saturate, which the authors attribute to limited memory capacity.
  • Stated limitations: the split point and hyperparameters may need manual tuning per model, layer or dataset, and the compensation scheme is deliberately coarse-grained.
  • Code: github.com/MKCatch/SpecEdit

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Zhejiang UniversityHangzhou · China172 works · 12,988 citations🎓South China University of TechnologyGuangzhou · China22 works · 1,554 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Junxian Liu, Zikun Deng, Xin Wu, Dazhen Deng and Yi Cai. 2026. SpecEdit: A Spectral Approach for Multi-Round Knowledge Editing. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.922

@inproceedings{liu2026specedit,
  title = {{SpecEdit: A Spectral Approach for Multi-Round Knowledge Editing}},
  author = {Junxian Liu, Zikun Deng, Xin Wu, Dazhen Deng and Yi Cai},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.922},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.922}
}

References

30 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 6491–6506. find
  2. Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. 2022. Knowledge neurons in pretrained transformers. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493– 8502. find
  3. Chandler Davis and W. M. Kahan. 1970. The rotation of eigenvectors by a perturbation. iii. SIAM Journal on Numerical Analysis, 7(1):1–46. find
  4. Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Jie Shi, Xiang Wang, Xiangnan He, and Tat-Seng Chua. 2025. AlphaEdit: Null-space constrained knowledge editing for language models. In International Conference on Learning Representations. find
  5. Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 5484–5495. find
  6. Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. 2024. Model editing harms general abilities of large language models: Regularization to the rescue. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 16801–16819.
  7. Akshat Gupta, Phudish Prateepamornkul, Maochuan Lu, Ahmed Alaa, Thomas Hartvigsen, and Gopala Anumanchipalli. 2025. Lifelong knowledge editing requires better regularization. In Findings of the Association for Computational Linguistics: EMNLP. find
  8. Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023. Aging with GRACE: Lifelong model editing with discrete key-value adaptors. In Proceedings of Annual Conference on Neural Information Processing Systems. find
  9. Chenhui Hu, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. 2024. WilKE: Wise-layer knowledge editor for lifelong knowledge editing. In Findings of the Association for Computational Linguistics, pages 3476–3503. find
  10. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Trans. Inf. Syst., 43(2):42:1– 42:55…
  11. Zeyu Huang, Yikang Shen, Xiaofeng Zhang, Jie Zhou, Wenge Rong, and Zhang Xiong. 2023. Transformer-Patcher: One mistake worth one neuron. In Proceedings of the International Conference on Learning Representations.
  12. Ziwei Ji, Tiezheng Yu, Yan Xu, Nayeon Lee, Etsuko Ishii, and Pascale Fung. 2023. Towards mitigating LLM hallucination via self reflection. In Findings of the Association for Computational Linguistics: EMNLP, pages 1827–1843.
  13. Houcheng Jiang, Junfeng Fang, Tianyu Zhang, Baolong Bi, An Zhang, Ruipeng Wang, Tao Liang, and Xiang Wang. 2025. Neuron-level sequential editing for large language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16678– 1670…
  14. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In Proceedings of the Conference on Computational Natural Language Learning (CoNLL 2017), pages 333–342.
  15. Jun-Yu Ma, Hong Wang, Hao-Xiang Xu, Zhen-Hua Ling, and Jia-Chen Gu. 2025. Perturbation-restrained sequential model editing. In Proceedings of the Thirteenth International Conference on Learning Representations. find
  16. Aman Madaan, Niket Tandon, Peter Clark, and Yiming Yang. 2022. MemPrompt: Memory-assisted prompt editing with user feedback. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
  17. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. In Advances in Neural Information Processing Systems, volume 35, pages 17359–17372. Curran Associates, Inc. find
  18. Kevin Meng, Arnab Sen Sharma, Alex J Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. In The Eleventh International Conference on Learning Representations. find
  19. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2022a. Fast model editing at scale. In International Conference on Learning Representations. find
  20. Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D. Manning, and Chelsea Finn. 2022b. Memory-based model editing at scale. In Proceedings of International Conference on Machine Learning, volume 162, pages 15817–15831. find
  21. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of Annual Conference on Neural Information Processing Systems, pages 5998– 6008.
  22. Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. 2024a. WISE: Rethinking the knowledge memory for lifelong model editing of large language models. In Proceedings of the Annual Conference on Neural Information Processing Systems. find
  23. Song Wang, Yaochen Zhu, Haochen Liu, Zaiyi Zheng, Chen Chen, and Jundong Li. 2024b. Knowledge editing for large language models: A survey. ACM Comput. Surv., 57(3).
  24. Xin Wu, Yuqi Bu, Yi Cai, and Tao Wang. 2024. Updating large language models’ memories with time constraints. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13693– 13702. find
  25. Haoyu Xu, Pengxiang Lan, Enneng Yang, Guibing Guo, Jianzhe Zhao, Linying Jiang, and Xingwei Wang. 2025. Knowledge decoupling via orthogonal projection for lifelong editing of large language models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  26. Yunzhi Yao, Peng Wang, Bozhong Tian, Siyuan Cheng, Zhoubo Li, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023. Editing large language models: Problems, methods, and opportunities. pages 10222– 10240. find
  27. Ningyu Zhang, Yunzhi Yao, Bozhong Tian, Peng Wang, Shumin Deng, Mengru Wang, Zekun Xi, Shengyu Mao, Jintian Zhang, Yuansheng Ni, Siyuan Cheng, Ziwen Xu, Xin Xu, Jia-Chen Gu, Yong Jiang, Pengjun Xie, Fei Huang, Lei Liang, Zhiqiang Zhang, and 3 others. 2024. A comprehensive study of knowledge editing… find
  28. Ce Zheng, Lei Li, Qingxiu Dong, Yuxuan Fan, Zhiyong Wu, Jingjing Xu, and Baobao Chang. 2023. Can we edit factual knowledge by in-context learning? In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 4862–4876. find
  29. Bihan Zhou, Haopeng Ren, Li Yuan, Yi Cai, Liuwen Cao, and Zikun Deng. 2025a. Ruleedit: Towards rule-level knowledge generalization to mitigate over-editing in large language models. In Findings of the Association for Computational Linguistics: ACL 2025, pages 3159–3175. find
  30. Wei Zhou, Wei Wei, Guibang Cao, and Fei Wang. 2025b. Editing memories through few targeted neurons. In Proceedings of the AAAI Conference on Artificial Intelligence. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-specedit-spectral-approach-multi-round/