Elamur Research⌕
☰
home / papers / 2026 / HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learning

HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learning

Paper · 2026 · Yangfan Wang, Tianyang Sun, Chen Tang et al.

Summary

Lifelong model editing (LME) repeatedly corrects outdated or wrong facts in a deployed LLM while trying not to damage unrelated or earlier-edited knowledge. The authors argue that current editors make a mistake by modifying the same fixed, dense block of layers for every fact, even though different facts seem to reside in different layers. This leads to over-modification and catastrophic forgetting. HiEdit casts editing as a two-level hierarchical reinforcement learning problem. A high-level hypernetwork acts as a manager: it reads gradient signals and produces an importance distribution over layers, choosing where to edit. A low-level hypernetwork acts as a worker and computes the actual parameter updates for the chosen layers. An intrinsic reward compares sparse updates against full-layer updates, which pushes the system toward minimal changes. On ZsRE and CounterFact with several LLMs, HiEdit beats RLEdit by 8.48% on average while touching only half the layers per edit. It stays robust over very long edit streams, which matters for keeping deployed models current without retraining.

Key points

  • Stream length: experiments include up to 20,000 sequential ZsRE edits on Llama-3-8B. AlphaEdit degrades markedly by about 8,000 edits. HiEdit keeps General Retention (mean F1 over six GLUE tasks) close to the unedited model.
  • Layer selection: layers are chosen with a TopK router, and K is a fixed hyperparameter. Dynamic schemes such as Top-p are left for future work.
  • Comparison with random layer choice (Table 3, ZsRE, Llama-3-8B):
    • HiEdit reaches Specificity 48.42 and Retention 75.16.
    • The Random 1 and Random 2 baselines reach Specificity 30.32 and 33.60, and Retention 69.72 and 68.03.
  • Learned layer preferences:
    • Across 2,000 samples, layer 13 is selected in over 88% of cases.
    • Music facts tend to use layers 20 and 24.
    • Disease facts tend to use layers 14 and 15.
  • Code is available at github.com/yangfanww/hiedit. Evaluation covers only structured general-domain knowledge (ZsRE, CounterFact).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China158 works · 28,841 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Yangfan Wang, Tianyang Sun, Chen Tang, Jie Liu, Wei Cai and Jingchi Jiang. 2026. HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1855

@inproceedings{wang2026hiedit,
  title = {{HiEdit: Lifelong Model Editing with Hierarchical Reinforcement Learning}},
  author = {Yangfan Wang, Tianyang Sun, Chen Tang, Jie Liu, Wei Cai and Jingchi Jiang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1855},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1855}
}

References

37 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Pierre-Luc Bacon, Jean Harb, and Doina Precup. 2017. The option-critic architecture. In AAAI 2017. find
  2. Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432. find
  3. Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. find
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners.
  5. Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, and Tat-Seng Chua. 2025. Alphaedit: Null-space constrained knowledge editing for language models. find
  6. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  7. Jia-Chen Gu, Hao-Xiang Xu, Jun-Yu Ma, Pan Lu, Zhen-Hua Ling, Kai-Wei Chang, and Nanyun Peng. 2024. Model editing harms general abilities of large language models: Regularization to the rescue. In EMNLP 2024.
  8. Jean Harb, Pierre-Luc Bacon, Martin Klissarov, and Doina Precup. 2018. When waiting is not an option: Learning options with a deliberation cost. In AAAI 2018. find
  9. Tom Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. 2023. Aging with grace: Lifelong model editing with discrete key-value adaptors. find
  10. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In ICLR 2021.
  11. Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models.
  12. Jingchi Jiang, Rujia Shen, Chao Zhao, Yi Guan, Xuehui Yu, and Xuelian Fu. 2025. Causal discovery based on hierarchical reinforcement learning. Expert Systems with Applications 2025. find
  13. Angeliki Lazaridou, Adhi Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Gimenez, Cyprien de Masson d’Autume, Tomas Kocisky, Sebastian Ruder, and 1 others. 2021. Mind the gap: Assessing temporal generalization in neural language models. find
  14. Omer Levy, Minjoon Seo, Eunsol Choi, and Luke Zettlemoyer. 2017. Zero-shot relation extraction via reading comprehension. In CoNLL 2017.
  15. Zherui Li, Houcheng Jiang, Hao Chen, Baolong Bi, Zhenhong Zhou, Fei Sun, Junfeng Fang, and Xiang Wang. 2025. Reinforced lifelong editing for language models. find
  16. Nicola De Cao, Wilker Aziz, and Ivan Titov. 2021. Editing factual knowledge in language models. In EMNLP 2021. find
  17. Jun-Yu Ma, Hong Wang, Hao-Xiang Xu, Zhen-Hua Ling, and Jia-Chen Gu. 2024. Perturbation-restrained sequential model editing. find
  18. William B Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In IWP 2005.
  19. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associations in gpt. find
  20. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2023. Mass-editing memory in a transformer. find
  21. Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2022. Fast model editing at scale. find
  22. Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. 2018. Data-efficient hierarchical reinforcement learning. find
  23. Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In EMNLP 2019.
  24. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP 2013.
  25. Richard S Sutton, Doina Precup, and Satinder Singh. 1999. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence 1999. find
  26. Chenmien Tan, Ge Zhang, and Jie Fu. 2024. Massive editing for large language models via meta learning. find
  27. Chen Tang, Bo Lv, Zifan Zheng, Bohao Yang, Kun Zhao, Ning Liao, Xiaoxing Wang, Feiyu Xiong, Zhiyu Li, Nayu Liu, and 1 others. 2025. Graphmoe: Amplifying cognitive depth of mixture-of-experts network via introducing self-rethinking mechanism. arXiv preprint arXiv:2501.07890. — [arXiv:2501.07890](https://arxiv.org/abs/2501.07890) find
  28. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  29. Chen Tessler, Shahar Givony, Tom Zahavy, Daniel Mankowitz, and Shie Mannor. 2017. A deep hierarchical approach to lifelong learning in minecraft. In AAAI 2017. find
  30. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Workshop BlackboxNLP of EMNLP 2018.
  31. Haoyu Wang, Tianci Liu, Ruirui Li, Monica Xiao Cheng, Tuo Zhao, and Jing Gao. 2024. Roselora: Row and column-wise sparse low-rank adaptation of pre-trained language model for knowledge editing and fine-tuning. In EMNLP 2024. find
  32. Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. 2019. Neural network acceptability judgments. find
  33. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In NAACL 2018.
  34. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, and 1 others. 2020. Transformers: State-of-the-art natural language processing. In EMNLP 2020.
  35. Yunzhi Yao, Ningyu Zhang, Zekun Xi, Mengru Wang, Ziwen Xu, Shumin Deng, and Huajun Chen. 2024. Knowledge circuits in pretrained transformers. find
  36. Taolin Zhang, Qizhou Chen, Dongyang Li, Chengyu Wang, Xiaofeng He, Longtao Huang, Jun Huang, and 1 others. 2024. Dafnet: Dynamic auxiliary fusion for sequential model editing in large language models. In Findings of ACL 2024. find
  37. Chen Zhu, Ankit Singh Rawat, Manzil Zaheer, Srinadh Bhojanapalli, Daliang Li, Felix Yu, and Sanjiv Kumar. 2020. Modifying memories in transformer models. arXiv preprint arXiv:2012.00363. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-hiedit-lifelong-model-editing-hierarchical/