Elamur Research⌕
☰
home / papers / 2026 / Representation-Guided Parameter-Efficient LLM Unlearning

Representation-Guided Parameter-Efficient LLM Unlearning

Paper · 2026 · Zeguan Xiao, Lang Mo, Yun Chen et al.

Summary

Removing memorized private or harmful content from LLMs without retraining is hard. LoRA-based unlearning methods such as FILA and VILA pick which weights to modify using Fisher-information importance scores. Because of superposition, a single parameter often serves many concepts, so these scores cannot cleanly separate knowledge that should be forgotten from knowledge that should be kept. The authors propose ReGLU, which works with the geometry of layer representations instead of parameter importance. It has two parts. RILA initializes the LoRA matrices from the leading eigenvectors of a subspace where forget-set activations vary strongly and retain-set activations vary little. ROL is a regularizer that keeps the LoRA up-projection orthogonal to the dominant directions of retain-set representations. Experiments on the TOFU and WMDP benchmarks with Llama-2-7B, Phi-1.5 and Zephyr-7B-beta show better forgetting with less loss of utility than state-of-the-art baselines. Initialization needs only forward passes, which makes it cheaper than gradient-based alternatives. The work suggests that controlling subspaces is more reliable than estimating per-parameter importance for selective forgetting.

Key points

  • Two components:
    • RILA is an initialization that maximizes forget-versus-retain "differential energy" through eigen-decomposition of activation covariances.
    • ROL is an orthogonality loss with respect to the principal subspace of the retain set.
  • Orthogonality to the retain subspace (Llama-2-7B, TOFU Forget 5%): ReGLU scores 0.87, compared with FILA at 0.70 and VILA at 0.82. Removing ROL drops ReGLU to 0.82.
  • Initialization cost on TOFU Forget 10% with a 7B model: RILA takes 0.51 GPU hours, VILA 0.81 and FILA 23.32. With randomized SVD, RILA drops to 0.07–0.12 GPU hours on 7B and 0.19–0.29 GPU hours on Llama2-70B.
  • Limitations:
    • Results depend on how representative the small data subsets used for covariance estimation are.
    • Performance varies with the chosen rank r of the LoRA adapters and with λ, the weight given to the regularizer.
    • Evaluation is mostly limited to TOFU and WMDP.
  • Code: github.com/sustech-nlp/ReGLU (Findings of ACL 2026).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Southern University of Science and TechnologyShenzhen · China22 works · 1,337 citations🎓Shanghai University of Finance and EconomicsShanghai · China15 works · 825 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Zeguan Xiao, Lang Mo, Yun Chen, Lei Yang, Jiehui Zhao, Lili Yang and Guanhua Chen. 2026. Representation-Guided Parameter-Efficient LLM Unlearning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.717

@inproceedings{xiao2026representation,
  title = {{Representation-Guided Parameter-Efficient LLM Unlearning}},
  author = {Zeguan Xiao, Lang Mo, Yun Chen, Lei Yang, Jiehui Zhao, Lili Yang and Guanhua Chen},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.717},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.717}
}

References

27 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency, pages 610–623. find
  2. Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. 2022. What does it mean for a language model to preserve privacy? In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’22, page 2280–2292, New York, NY, USA. Associati… find
  3. Yinzhi Cao and Junfeng Yang. 2015. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463– 480. IEEE. find
  4. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2022. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646. find
  5. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, et al. 2024. The wmdp benchmark: measuring and reducing malicious use with unlearning. In Proceedings of the 41st International Conference on Machin…
  6. Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee. 2023. Textbooks are all you need ii: phi-1.5 technical report. Preprint, arXiv:2309.05463.
  7. Bo Liu, Qiang Liu, and Peter Stone. 2022. Continual learning and private unlearning. In Conference on Lifelong Learning Agents, pages 243–254. PMLR. find
  8. Minrui Luo, Fuhang Kuang, Yu Wang, Zirui Liu, and Tianxing He. 2025. Sc-lora: Balancing efficient fine-tuning and knowledge preservation via subspace-constrained lora. arXiv preprint arXiv:2505.23724. — [arXiv:2505.23724](https://arxiv.org/abs/2505.23724) find
  9. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary Chase Lipton, and J Zico Kolter. 2024. TOFU: A task of fictitious unlearning for LLMs. In First Conference on Language Modeling.
  10. Sungmin Cha, Sungjun Cho, Dasol Hwang, and Moontae Lee. 2025. Towards robust and parameter-efficient knowledge unlearning for llms. In International Conference on Learning Representations. find
  11. Fanxu Meng, Zhaohui Wang, and Muhan Zhang. 2024. Pissa: Principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems, 37:121038– 121072.
  12. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. 2022. Toy models of superposition. arXiv preprint arXiv:2209.10652. find
  13. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture models. Preprint, arXiv:1609.07843.
  14. Chongyu Fan, Jiancheng Liu, Licong Lin, Jinghan Jia, Ruiqi Zhang, Song Mei, and Sijia Liu. 2024. Simplicity prevails: Rethinking negative preference optimization for llm unlearning. arXiv preprint arXiv:2410.07163. — [arXiv:2410.07163](https://arxiv.org/abs/2410.07163) find
  15. Ronald A Fisher. 1922. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character, 222(594-604):309–368. find
  16. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR).
  17. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations.
  18. Yejin Kim, Eunwon Kim, Buru Chang, and Junsuk Choe. 2025. Improving fisher information estimation and efficiency for lora-based llm unlearning. arXiv preprint arXiv:2508.21300. — [arXiv:2508.21300](https://arxiv.org/abs/2508.21300) find
  19. Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. 2023. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:2311.17035.
  20. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  21. Joel A. Tropp. 2012. User-friendly tail bounds for sums of random matrices. Foundations of Computational Mathematics, 12(4):389–434. find
  22. Lewis Tunstall, Edward Emanuel Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro Von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M Rush, and Thomas Wolf. 2024. Zephyr: Direct distillation of LM alignment. In First… find
  23. Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. 2025. MiLoRA: Harnessing minor singular components for parameter-efficient LLM finetuning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Tech… find
  24. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Transactions on Ma…
  25. Zeguan Xiao, Siqing Li, Yong Wang, Xuetao Wei, Jian Yang, Yun Chen, and Guanhua Chen. 2026. Modeling llm unlearning as an asymmetric two-task learning problem. arXiv preprint arXiv:2604.14808.
  26. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning. In First Conference on Language Modeling. projection onto subspace S; (2) derive the expected projection energy in terms of covariance matrices; and (3) apply Ky…
  27. Step 1: Projection operator representation. Let ∆W = BA. At initialization, B init = Q r ⊤ and A init = Q ⊤ r W 0 , so ∆W = Q r Q r W 0 . For any input x, let h = W 0 x be the corresponding output representation. Since Q r is an orthonormal basis for the subspace S, we have Q r Q ⊤ r = Π S . Thus: ∆… — Representation Engineering: A Top-Down Approach to AI Transparency

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2026-representation-guided-parameter-efficient-llm/