Elamur Research⌕
☰
home / papers / 2026 / Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalization

Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalization

Paper · 2026 · Weixu Zhang, Ye Yuan, Changjiang Han et al.

Summary

LLMs can adapt to a user's tone and topical interests, but current personalization methods (prompting, retrieval of user profiles, fine-tuning) treat this ability as a black box. The authors ask where personalization actually arises inside the transformer. They hypothesize a small set of attention heads, called Preference Heads, that causally carry user-specific stylistic and topical signals. To locate them, they ablate each head and compute a Preference Contribution Score (PCS): the increase in loss on user-aligned reference outputs when that head is removed. The top-K heads are retained. Their training-free method, Differential Preference Steering (DPS), runs the model twice at each decoding step, once intact and once with these heads masked. It then strengthens the gap between personalized and generic logits, scaled by a strength parameter γ. An optional extension clusters users by profile-embedding similarity and routes them to cluster-specific head sets. On the LaMP and LongLaMP benchmarks with several open LLMs, DPS improves personalization fidelity while keeping coherence and cost reasonable. This offers an interpretable, parameter-free handle on personalization.

Key points

  • Preference Head discovery is done offline. Each head's PCS reflects its causal effect on user-aligned generation. Performance peaks at a moderate number of heads, and adding many more mostly adds noise. Jaccard overlap analysis shows that the top-ranked heads stay stable as K grows.
  • Cluster-aware DPS supports hard or soft routing. For classification tasks, hard routing holds a small edge, whereas for generation tasks, soft routing yields more consistent improvements.
  • DPS adds no parameters or prompt changes. It needs two forward passes per decoding step, a cost comparable to CAD and DeCoRe. Unlike CAD, it requires no re-encoded prompt prefill. FLOP estimates for LLaMA-3-8B with 32 generated tokens show only modest overhead.
  • On LaMP-4 (personalized headline generation), human annotators judged 100 sampled instances in pairwise comparisons and preferred DPS over CAD for profile alignment. GPT-5.2 as a judge also favored DPS on a 1–5 scale:
    • relevance 4.45 vs 3.97
    • fluency 4.83 vs 4.51
    • style 3.91 vs 3.62
    • preference alignment 3.93 vs 3.63
    • factuality 4.21 vs 4.08
  • Limitations: DPS requires white-box access to attention heads, so it cannot be used with API-only models. The extra forward pass may also matter in latency-critical settings. Code is released at github.com/weixuzhang/DPS.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Université de MontréalMontreal · Canada68 works · 79,382 citations🏢SalesforceSan Francisco · United States56 works · 17,867 citations🎓McGill UniversityMontreal · Canada87 works · 11,027 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates124 works · 5,676 citations🔬Mila - Quebec Artificial Intelligence InstituteMontreal · Canada54 works · 13,565 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksInterpretability · 130 works

Cite

Cite as Weixu Zhang, Ye Yuan, Changjiang Han, Yuxing Tian, Zipeng Sun, Linfeng Du, Jikun Kang, Hong Kang, Xue Liu and Haolun Wu. 2026. Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalization. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.2205

@inproceedings{zhang2026preference,
  title = {{Preference Heads in Large Language Models: A Mechanistic Framework for Interpretable Personalization}},
  author = {Weixu Zhang, Ye Yuan, Changjiang Han, Yuxing Tian, Zipeng Sun, Linfeng Du, Jikun Kang, Hong Kang, Xue Liu and Haolun Wu},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.2205},
  url = {https://doi.org/10.18653/v1/2026.acl-long.2205}
}

References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: an automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization@ACL 2005, Ann Arbor, Michigan, USA, June 2…
  2. Ruizhe Chen, Xiaotian Zhang, Meng Luo, Wenhao Chai, and Zuozhu Liu. 2025a. PAD: personalized alignment of llms at decoding-time. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  3. Shijie Chen, Bernal Jimenez Gutierrez, and Yu Su. 2025b. Attention in large language models yields efficient zero-shot re-rankers. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  4. Yung-Sung Chuang, Yujia Xie, Hongyin Luo, Yoon Kim, James R. Glass, and Pengcheng He. 2024. Dola: Decoding by contrasting layers improves factuality in large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.n…
  5. Joy Crosbie and Ekaterina Shutova. 2025. Induction heads as an essential mechanism for pattern matching in in-context learning. In Findings of the Association for Computational Linguistics: NAACL 2025, Albuquerque, New Mexico, USA, April 29 - May 4, 2025, pages 5034–5096. Association for Computation… find
  6. Ge Gao, Alexey Taymanov, Eduardo Salinas, Paul Mineiro, and Dipendra Misra. 2024. Aligning LLM agents by learning latent preference from user edits. find
  7. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 - 15, 2024. find
  8. Aryo Pradipta Gema, Chen Jin, Ahmed Abdulaal, Tom Diethe, Philip Teare, Beatrice Alex, Pasquale Minervini, and Amrutha Saseendran. 2024. Decore: Decoding by contrasting retrieval heads to mitigate hallucinations. CoRR, abs/2410.18860. find
  9. Difan Jiao, Yilun Liu, Zhenwei Tang, Daniel Matter, Jürgen Pfeffer, and Ashton Anderson. 2024. SPIN: sparsifying and integrating internal neurons in large language models for text classification. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual me… find
  10. Ishita Kumar, Snigdha Viswanathan, Sushrita Yerra, Alireza Salemi, Ryan A. Rossi, Franck Dernoncourt, Hanieh Deilamsalehy, Xiang Chen, Ruiyi Zhang, Shubham Agarwal, Nedim Lipka, and Hamed Zamani. 2024. Longlamp: A benchmark for personalized long-form text generation. CoRR, abs/2407.11016. — [arXiv:2407.11016](https://arxiv.org/abs/2407.11016) find
  11. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  12. Jiahong Liu, Zexuan Qiu, Zhongyang Li, Quanyu Dai, Jieming Zhu, Minda Hu, Menglin Yang, and Irwin King. 2025. A survey of personalized large language models: Progress and future directions. CoRR, abs/2502.11528. — [arXiv:2502.11528](https://arxiv.org/abs/2502.11528) find
  13. Richard Yuanzhe Pang, Weizhe Yuan, He He, Kyunghyun Cho, Sainbayar Sukhbaatar, and Jason Weston. 2024. Iterative reasoning preference optimization. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, C…
  14. Yilun Qiu, Xiaoyan Zhao, Yang Zhang, Yimeng Bai, Wenjie Wang, Hong Cheng, Fuli Feng, and Tat-Seng Chua. 2025. Measuring what makes you unique: Difference-aware user modeling for enhancing LLM personalization. In Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, Ju… find
  15. Xiaoye Qu, Zengqi Yu, Dongrui Liu, Wei Wei, Daizong Liu, Jianfeng Dong, and Yu Cheng. 2025. Cooperative or competitive? understanding the interaction between attention heads from A game theory perspective. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Vo… find
  16. Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. 2024. Lamp: When large language models meet personalization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, page…
  17. Weijia Shi, Xiaochuang Han, Mike Lewis, Yulia Tsvetkov, Luke Zettlemoyer, and Wen-tau Yih. 2024. Trusting your evidence: Hallucinate less with context-aware decoding. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language…
  18. Aaron David Tucker, Kianté Brantley, Adam Cahall, and Thorsten Joachims. 2024. Coactive learning for large language models using implicit user feedback. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net. find
  19. Mengru Wang, Ziwen Xu, Shengyu Mao, Shumin Deng, Zhaopeng Tu, Huajun Chen, and Ningyu Zhang. 2025. Beyond prompt engineering: Robust behavior control in llms via steering target atoms. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),…
  20. Wenhao Wu, Yizhong Wang, Guangxuan Xiao, Hao Peng, and Yao Fu. 2025. Retrieval head mechanistically explains long-context factuality. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net.
  21. Kayo Yin and Jacob Steinhardt. 2025. Which attention heads matter for in-context learning? In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025. OpenReview.net. find
  22. Jinghao Zhang, Yuting Liu, Wenjie Wang, Qiang Liu, Shu Wu, Liang Wang, and Tat-Seng Chua. 2025a. Personalized text generation with contrastive activation steering. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Au…
  23. Wuwei Zhang, Fangcong Yin, Howard Yen, Danqi Chen, and Xi Ye. 2025b. Query-focused retrieval heads improve long-context reasoning and re-ranking. CoRR, abs/2506.09944. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-preference-heads-large-language-models/