Elamur Research⌕
☰
home / papers / 2026 / When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs

When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs

Paper · 2026 · Zhongxiang Sun, Yi Zhan, Chenglei Shen et al.

Summary

The paper examines an overlooked risk of personalized LLMs. When a model is conditioned on a user's chat history or profile, it may answer factual questions in line with what the user said before instead of with the truth. The authors call this failure "personalization-induced hallucination." They attribute it to representational entanglement: the hidden-state shift caused by personalization is not orthogonal to factual directions, so it pulls activations away from correct knowledge.

To counter this, they propose FPPS (Factuality-Preserving Personalized Steering), a lightweight method applied at inference time. It finds layers sensitive to personalization, probes whether factual reasoning is being distorted, and steers representations back toward factuality only when needed. They also release PFQABench, a benchmark that pairs factual and personalized questions within realistic user sessions.

Across several LLM backbones and personalization methods, FPPS raises factual accuracy without hurting personalized answers. This matters because commercial assistants increasingly enable memory features by default. In a simulated teaching setup, personalized models also led to measurably worse knowledge acquisition by learners.

Key points

  • Representation evidence: On PFQABench factual QA, hallucinated answers show lower final-layer cosine similarity between personalized and non-personalized responses than truthful answers do (p < 0.001). This points to a latent distortion rather than random decoding noise.
  • FPPS components: FPPS combines three modules:
    • a Representation Shift Locator, which finds vulnerable layers;
    • a Factuality Entanglement Prober, which detects distortion from internal activations;
    • an Adaptive Knowledge Steering Module, which intervenes only when needed.
  • FPPS variants: The method comes in three versions: FPPS-H uses a strict on/off gate, FPPS-S applies graded steering that can push in either direction, and FPPS-M combines the two through adaptive control.
  • Simulated learning: LLM-simulated "students" learned facts less accurately from personalized teacher models than from standard ones. Adding FPPS-M to the personalized teachers recovered an average of +7.0% learning accuracy.
  • Scope and limits: The study targets prompting-based personalization, the setup used by ChatGPT Memory, Gemini Personal Context and Claude Memory. FPPS needs access to intermediate activations, so it was tested only on open-weight models and cannot be applied directly to closed APIs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Renmin University of ChinaBeijing · China114 works · 16,118 citations🎓University of International Business and EconomicsBeijing · China7 works · 37 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Zhongxiang Sun, Yi Zhan, Chenglei Shen, Weijie Yu, Xiao Zhang, Ming He and Jun Xu. 2026. When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.395

@inproceedings{sun2026when,
  title = {{When Personalization Misleads: Understanding and Mitigating Hallucinations in Personalized LLMs}},
  author = {Zhongxiang Sun, Yi Zhan, Chenglei Shen, Weijie Yu, Xiao Zhang, Ming He and Jun Xu},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.395},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.395}
}

References

36 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Using claude’s chat, search, and memory to build on previous context. find
  2. Jianlv Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. 2024. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity 8049 9 text embeddings through self-knowledge distillation. Preprint, arXiv:2402.03216.
  3. Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R Glass, and Pengcheng He. 2023. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations.
  4. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, and 1 others. 2022. Toy models of superposition. arXiv preprint arXiv:2209.10652. find
  5. Google. 2025. Gemini: Personalization overview. find
  6. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 3214–3252.
  7. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In The Twelfth International Conference on Learning Representations.
  8. Jiahong Liu, Zexuan Qiu, Zhongyang Li, Quanyu Dai, Wenhao Yu, Jieming Zhu, Minda Hu, Menglin Yang, Tat-Seng Chua, and Irwin King. 2025. A survey of personalized large language models: Progress and future directions. arXiv preprint arXiv:2502.11528. — [arXiv:2502.11528](https://arxiv.org/abs/2502.11528) find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  10. Potsawee Manakul, Adian Liusie, and Mark Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. In Proceedings of the 2023 conference on empirical methods in natural language processing, pages 9004– 9017. find
  11. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609– 6625.
  12. Chimaobi Okite, Naihao Deng, Kiran Bodipati, Huaidian Hou, Joyce Chai, and Rada Mihalcea. 2025. Benchmarking and improving llm robustness for personalized generation. arXiv preprint arXiv:2509.19358. find
  13. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and 1 others. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems,…
  14. Jennifer King, Kevin Klyman, Emily Capstick, Tiffany Saade, and Victoria Hsieh. 2025. User privacy and large language models: An analysis of frontier developers’ privacy policies. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 8, pages 1465–1477. find
  15. Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A. Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, Nedim Lipka, and Hamed Zamani. 2024a. Longlamp: A benchmark for personalized long-form text generation. ArXiv, abs/2407.11016. — [arXiv:2407.11016](https://arxiv.org/abs/2407.11016) find
  16. Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, and 1 others. 2024b. Longlamp: A benchmark for personalized long-form text generation. arXiv preprint arXiv:2407.11016. — [arXiv:2407.11016](https://arxiv.org/abs/2407.11016) find
  17. Tomo Lazovich. 2023. Filter bubbles and affective polarization in user-personalized large language model outputs. In Proceedings on, pages 29–37. PMLR. find
  18. OpenAI. 2025. Memory and new controls for chatgpt. find
  19. Yilun Qiu, Xiaoyan Zhao, Yang Zhang, Yimeng Bai, Wenjie Wang, Hong Cheng, Fuli Feng, and Tat-Seng Chua. 2025a. Measuring what makes you unique: Difference-aware user modeling for enhancing llm personalization. arXiv preprint arXiv:2503.02450. find
  20. Yilun Qiu, Xiaoyan Zhao, Yang Zhang, Yimeng Bai, Wenjie Wang, Hong Cheng, Fuli Feng, and Tat-Seng Chua. 2025b. Measuring what makes you unique: Difference-aware user modeling for enhancing llm personalization. ArXiv, abs/2503.02450. find
  21. Reddit. 2025. Geminis new personal context is not just better. find
  22. Chris Richardson, Yao Zhang, Kellen Gillespie, Sudipta Kar, Arshdeep Singh, Zeynab Raeesy, Omar Zia Khan, and Abhinav Sethy. 2023a. Integrating summarization and retrieval for enhanced personalization via large language models. arXiv preprint arXiv:2310.20081. — [arXiv:2310.20081](https://arxiv.org/abs/2310.20081) find
  23. Chris Richardson, Yao Zhang, Kellen Gillespie, Sudipta Kar, Arshdeep Singh, Zeynab Raeesy, Omar Zia Khan, and Abhinav Sethy. 2023b. Integrating summarization and retrieval for enhanced personalization via large language models. ArXiv, abs/2310.20081. — [arXiv:2310.20081](https://arxiv.org/abs/2310.20081) find
  24. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Turner. 2024. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15504–15522. 8050 10 find
  25. Zhongxiang Sun, Zihua Si, Xiaoxue Zang, Kai Zheng, Yang Song, Xiao Zhang, and Jun Xu. 2025. Largepig for hallucination-free query generation: Your large language model is secretly a pointer generator. In Proceedings of the ACM on Web Conference 2025, pages 4766–4779. find
  26. ZhongXiang Sun, Xiaoxue Zang, Kai Zheng, Jun Xu, Xiao Zhang, Weijie Yu, Yang Song, and Han Li. Redeep: Detecting hallucination in retrieval-augmented generation via mechanistic interpretability. In The Thirteenth International Conference on Learning Representations. and Yaodong Yang. Amulet: Realign… find
  27. Zhi Zheng, WenShuo Chao, Zhaopeng Qiu, Hengshu Zhu, and Hui Xiong. 2024. Harnessing large language models for text-rich sequential recommendation. Proceedings of the ACM Web Conference 2024. find
  28. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248.
  29. Anvesh Rao Vijjini, Somnath Basu Roy Chowdhury, and Snigdha Chaturvedi. 2025. Exploring safety-utility trade-offs in personalized language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologie… find
  30. Bin Wu, Zhengyan Shi, Hossein A Rahmani, Varsha Ramineni, and Emine Yilmaz. 2024. Understanding the role of user profile in the personalization of large language models. arXiv preprint arXiv:2406.17803. — [arXiv:2406.17803](https://arxiv.org/abs/2406.17803) find
  31. Di Wu, Hongwei Wang, Wenhao Yu, Yuwei Zhang, Kai-Wei Chang, and Dong Yu. 2025. Longmemeval: Benchmarking chat assistants on long-term interactive memory. In The Thirteenth International Conference on Learning Representations.
  32. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2024. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. find
  33. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 2369–…
  34. Linhai Zhang, Jialong Wu, Deyu Zhou, and Yulan He. 2025. Proper: A progressive learning framework for personalized large language models with group-level adaptation. arXiv preprint arXiv:2503.01303.
  35. You Zhang, Jin Wang, Liang-Chih Yu, Dan Xu, and Xuejie Zhang. 2024. Personalized lora for human-centered text understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19588–19596. find
  36. Zhaowei Zhang, Fengshuo Bai, Qizhi Chen, Chengdong Ma, Mingzhi Wang, Haoran Sun, Zilong Zheng, 8051 11 Contents \#\# Introduction \#\# Related Work find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sun-2026-when-personalization-misleads-understanding-mitigating/