Elamur Research⌕
☰
home / papers / 2026 / Where Paths Split: Localized, Calibrated Control of Moral Reasoning in Large Language Models

Where Paths Split: Localized, Calibrated Control of Moral Reasoning in Large Language Models

Paper · 2026 · Chenchen Yuan, Zheyu Zhang and Gjergji Kasneci

Summary

LLMs tend to shift between moral stances depending on context. Existing steering methods, such as prompting or adding a vector scaled by an unbounded coefficient, are hard to interpret and give no guarantee about how strongly a target view will appear. The authors propose Convergent-Divergent Routing (CDR), an inference-time technique for deontology versus utilitarianism. It first uses ridge-regression probes trained on the ETHICS dataset to find attention heads relevant to each framework, and FFN vectors aligned with each framework's indicator token. It then locates "branch points": layers where both frameworks share attention heads but their FFN units differ. Blocking the non-target route at these points already gives binary control. For graded control, Common Spatial Patterns (borrowed from EEG analysis) yields one utilitarian and one deontological direction per layer. Dual Logit Calibration then applies a closed-form, minimum-norm edit inside that 2D subspace, so the output follows user weights on the simplex. On real-life moral dilemmas, the method outperforms recent baselines and mostly preserves general skills. This makes value steering more predictable and easier to inspect.

Key points

  • Models and evaluation: experiments use LLaMA-2-7B-Chat, Vicuna-7B-v1.5 and Yi-1.5-6B-Chat on real-life dilemmas (Nguyen et al., 2022). Further checks include the DailyDilemmas benchmark, a utilitarianism vs. justice pair, and human evaluation.
  • Preference weights: steering strength is set by bounded weights (α_U, α_D), each non-negative and summing to 1, instead of an open-ended scalar. On Llama, the reported calibration error is an MAE of 1.84 pp, with no deviation above 3.23 pp. PCA and PLS-DA baselines responded poorly to changes in α_U.
  • Localization: predictive heads cluster in middle layers for Llama and Vicuna, and in middle-to-upper layers for Yi-1.5. Framework-specific FFN units make up only about 15–25.7% (utilitarian) and 14.9–20.5% (deontological) of each layer's units.
  • General capabilities: performance on benchmarks such as TriviaQA and GSM8K stays largely intact. Clear drops appear only at very low α_U on Vicuna and Yi-1.5.
  • Scope and limitations: the method requires no fine-tuning, and code and data are released on GitHub (yuanchencn/Moral-Reasoning). It has been tested only on standard multi-head attention models, not GQA, and only on pairs of ethical frameworks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technical University of MunichMunich · Germany62 works · 48,970 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 works

Cite

Cite as Chenchen Yuan, Zheyu Zhang and Gjergji Kasneci. 2026. Where Paths Split: Localized, Calibrated Control of Moral Reasoning in Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1933

@inproceedings{yuan2026where,
  title = {{Where Paths Split: Localized, Calibrated Control of Moral Reasoning in Large Language Models}},
  author = {Chenchen Yuan, Zheyu Zhang and Gjergji Kasneci},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1933},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1933}
}

References

52 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abubakar Abid, Martin J Zhang, Vivek K Bagaria, and James Zou. 2018. Exploring patterns enriched in a dataset with contrastive principal component analysis. Nature communications, 9(1):2134. find
  2. Jadie Adams, Brian Hu, Emily Veenhuis, David Joy, Bharadwaj Ravichandran, Aaron Bray, Anthony Hoogs, and Arslan Basharat. 2025. Steerable pluralism: Pluralistic alignment via few-shot comparative regression. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 8, pages 15–25. find
  3. Utkarsh Agarwal, Kumar Tanmay, Aditi Khandelwal, and Monojit Choudhury. 2024. Ethical reasoning and moral value alignment of llms depend on the language we prompt them in. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-… find
  4. Lisa P Argyle, Ethan C Busby, Nancy Fulda, Joshua R Gubler, Christopher Rytting, and David Wingate. 2023. Out of one, many: Using language models to simulate human samples. Political Analysis, 31(3):337–351.
  5. Matthew Barker and William Rayens. 2003. Partial least squares for discrimination. Journal of Chemometrics: A Journal of the Chemometrics Society, 17(3):166– 173. find
  6. Benjamin Blankertz, Ryota Tomioka, Steven Lemm, Motoaki Kawanabe, and Klaus-Robert Muller. 2007. Optimizing spatial filters for robust eeg single-trial analysis. IEEE Signal processing magazine, 25(1):41–56. find
  7. Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ales Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the N…
  8. Anne-Laure Boulesteix and Korbinian Strimmer. 2007. Partial least squares: a versatile tool for the analysis of high-dimensional genomic data. Briefings in bioinformatics, 8(1):32–44. find
  9. Stephen Boyd and Lieven Vandenberghe. 2004. Convex optimization. Cambridge university press. find
  10. Mohna Chakraborty, Lu Wang, and David Jurgens. 2025. Structured moral reasoning in language models: A value-grounded evaluation framework. arXiv preprint arXiv:2506.14948.
  11. Runjin Chen, Andy Arditi, Henry Sleight, Owain Evans, and Jack Lindsey. 2025. Persona vectors: Monitoring and controlling character traits in language models. arXiv preprint arXiv:2507.21509. — [arXiv:2507.21509](https://arxiv.org/abs/2507.21509) find
  12. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  13. Yu Ying Chiu, Liwei Jiang, and Yejin Choi. 2024. Dailydilemmas: Revealing value preferences of llms with quandaries of daily life. arXiv preprint arXiv:2410.02683. — [arXiv:2410.02683](https://arxiv.org/abs/2410.02683) find
  14. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. CoRR, abs/2110.14168.
  15. Junchen Ding, Penghao Jiang, Zihao Xu, Ziqi Ding, Yichen Zhu, Jiaojiao Jiang, and Yuekang Li. 2025. " pull or not to pull?”: Investigating moral biases in leading large language models across ethical dilemmas. arXiv preprint arXiv:2508.07284. — [arXiv:2508.07284](https://arxiv.org/abs/2508.07284) find
  16. Rohit K Dubey, Damian Dailisan, and Sachit Mahajan. 2025. Addressing moral uncertainty using large language models for ethical decision-making. arXiv preprint arXiv:2503.05724. — [arXiv:2503.05724](https://arxiv.org/abs/2503.05724) find
  17. Maxwell Forbes, Jena D Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653–670. find
  18. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  19. Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoav Goldberg. 2022. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, pages 30–45. Association for Computational Linguistics…
  20. Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, Zhiyuan Liu, and Maosong Sun. 2024. Controllable preference optimization: Toward controllable multi-objective alignment. In Proceedings of the 2024 Conference on Empirical Methods in Natu… find
  21. Wes Gurnee and Max Tegmark. 2024. Language models represent space and time. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. Open-Review.net. find
  22. Leif Hancox-Li and Borhane Blili-Hamelin. 2024. Is ethics about ethics? evaluating the ethics benchmark. arXiv preprint arXiv:2410.13009. — [arXiv:2410.13009](https://arxiv.org/abs/2410.13009) find
  23. Shirley Anugrah Hayati, Minhwa Lee, Dheeraj Rajagopal, and Dongyeop Kang. 2024. How far can we extract diverse perspectives from large language models? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5336–5366. find
  24. Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. Aligning AI with shared human values. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  25. Zhijing Jin, Max Kleiman-Weiner, Giorgio Piatti, Sydney Levine, Jiarui Liu, Fernando Gonzalez Adauto, Francesco Ortu, András Strausz, Mrinmaya Sachan, Rada Mihalcea, Yejin Choi, and Bernhard Schölkopf. 2025. Language model alignment in multilingual trolley problems. In The Thirteenth International C…
  26. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611.
  27. Junsol Kim, James Evans, and Aaron Schein. 2025. Linear representations of political perspective emerge in large language models. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  28. ZJ Koles, MS Lazar, and SZ Zhou. 1990. Spatial patterns underlying population differences in the background eeg. Brain topography, 2(4):275–284. find
  29. Anita Körner and Roland Deutsch. 2023. Deontology and utilitarianism in real life: A set of moral dilemmas based on historic events. Personality and Social Psychology Bulletin, 49(10):1511–1528. find
  30. Olivier Ledoit and Michael Wolf. 2004. A well-conditioned estimator for large-dimensional covariance matrices. Journal of multivariate analysis, 88(2):365–411. find
  31. Johannes Müller-Gerking, Gert Pfurtscheller, and Henrik Flyvbjerg. 1999. Designing optimal spatial filters for single-trial eeg classification in a movement task. Clinical neurophysiology, 110(5):787–798. find
  32. Tuan Dung Nguyen, Georgiana Lyall, Alasdair Tran, Minjeong Shin, Nicholas George Carroll, Colin Klein, and Lexing Xie. 2022. Mapping topics in 100,000 real-life moral dilemmas. In Proceedings of the International AAAI Conference on Web and Social Media, volume 16, pages 699–710. find
  33. OpenAI. 2024. Gpt-4o system card. abs/2410.21276.
  34. CoRR, Yupeng Qi, Ziyu Lyu, Min Yang, Yanlin Wang, Lu Bai, and Lixin Cui. 2025. Midpo: Dual preference optimization for safety and helpfulness in large language models via a mixture of experts framework. arXiv preprint arXiv:2506.02460. find
  35. Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. 2023. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36:71095–7113…
  36. Abhinav Sukumar Rao, Aditi Khandelwal, Kumar Tanmay, Utkarsh Agarwal, and Monojit Choudhury. 2023. Ethical reasoning over moral alignment: A case and framework for in-context ethical policies in llms. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 13370– 13388. find
  37. Nina Rimsky, Nick Gabrieli, Julian Schulz, Meg Tong, Evan Hubinger, and Alexander Matt Turner. 2024. Steering llama 2 via contrastive activation addition. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand,… find
  38. Shibani Santurkar, Esin Durmus, Faisal Ladhak, Cinoo Lee, Percy Liang, and Tatsunori Hashimoto. 2023. Whose opinions do language models reflect? In International Conference on Machine Learning, pages 29971–30004. PMLR.
  39. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell L. Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. 2024. A roadmap to pluralistic alignment. CoRR, abs/2402.05070.
  40. Charles Spearman. 1961. The proof and measurement of association between two things. find
  41. Judith Jarvis Thomson. 1984. The trolley problem. Yale LJ, 94:1395. find
  42. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  43. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J Vazquez, Ulisse Mini, and Monte MacDiarmid. 2023. Steering language models with activation engineering. arXiv preprint arXiv:2308.10248.
  44. Haoxiang Wang, Yong Lin, Wei Xiong, Rui Yang, Shizhe Diao, Shuang Qiu, Han Zhao, and Tong Zhang. 2024a. Arithmetic control of llms for diverse user preferences: Directional preference alignment with multi-objective rewards. In Proceedings of the 62nd Annual Meeting of the Association for Computation…
  45. Kaiwen Wang, Rahul Kidambi, Ryan Sullivan, Alekh Agarwal, Christoph Dann, Andrea Michi, Marco Gelmi, Yunxuan Li, Raghav Gupta, Avinava Dubey, Alexandre Ramé, Johan Ferret, Geoffrey Cideron, Le Hou, Hongkun Yu, Amr Ahmed, Aranyak Mehta, Léonard Hussenot, Olivier Bachem, and Edouard Leurent. 2024b. Co… find
  46. Tiannan Wang, Meiling Tao, Ruoyu Fang, Huilin Wang, Shuai Wang, Yuchen Eleanor Jiang, and Wangchunshu Zhou. 2024c. Ai persona: Towards life-long personalization of llms. arXiv preprint arXiv:2412.13103. — [arXiv:2412.13103](https://arxiv.org/abs/2412.13103) find
  47. Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2024. Reft: Representation finetuning for language models. Advances in Neural Information Processing Systems, 37:63908–63962.
  48. Zhengxuan Wu, Qinan Yu, Aryaman Arora, Christopher D Manning, and Christopher Potts. 2025. Improved representation steering for language models. arXiv preprint arXiv:2505.20809. find
  49. Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, and 11 others. 2024. Yi: Open foundation models by 01.ai. CoRR, abs/2403.046…
  50. Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, and Yaodong Yang. 2024. Panacea: Pareto alignment via preference adaptation for llms. Advances in Neural Information Processing Systems, 37:75522–75558. find
  51. Jingyan Zhou, Minda Hu, Junan Li, Xiaoying Zhang, Xixin Wu, Irwin King, and Helen Meng. 2024. Rethinking machine ethics–can llms perform moral reasoning through the lens of moral theories? In Findings of the Association for Computational Linguistics: NAACL 2024, pages 2227–2242. find
  52. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. In The eleventh international conference on learning representations.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yuan-2026-where-paths-split-localized-calibrated/