Elamur Research⌕
☰
home / papers / 2026 / PAM: Enhancing General Alignment of Large Reasoning Models through Priority-Aware Metacognition

PAM: Enhancing General Alignment of Large Reasoning Models through Priority-Aware Metacognition

Paper · 2026 · Zhihao Xu, Fuzhen Yang, Liang Lin et al.

Summary

Large Reasoning Models perform well on math and coding, but longer chains of thought do not reliably make them safer or more helpful in general-purpose use. Sometimes reasoning even increases harmful output. The authors borrow the idea of metacognition from cognitive science and propose PAM (Priority-Aware Metacognition). Before reasoning, the model decides which human preference matters most for the query, such as harmlessness for a dangerous request. It then uses that judgment to steer its reasoning and its answer.

A prompting-only pilot on R1-Distill-Qwen-7B showed that adding a priority check outperformed both standard CoT and no-thinking modes on safety, helpfulness and instruction following. PAM makes this behaviour trainable in two stages:

  1. A cold-start stage builds warm-up data from Flavell's three components of metacognitive knowledge: tasks, self and strategies.
  2. A preference-optimization stage reinforces the behaviour.

Under identical training pipelines, PAM gains about 10 points on helpfulness and harmlessness benchmarks. The approach is data-centric and fits existing RLHF workflows.

Key points

  • In the training-free pilot, Priority-Aware CoT was best on all three preferences. Standard CoT was weak on safety, while Zero CoT gained safety but lost helpfulness and instruction following.

  • Benchmarks include AlpacaEval2, Advbench, WildJailbreak and IFEval. Backbones are R1-Qwen-7B and R1-LLaMA-8B.

  • Cold-start ablation:

    VariantAlpacaEval2AdvbenchWildJailbreak
    Vanilla28.3287.1162.25
    PRM-Distill31.4399.6178.95
    PRM-Self31.1899.8077.65

    Because self-generated metacognition (PRM-Self) works about as well as distilled data, the gain comes from the mechanism itself rather than from distillation.

  • With R1-LLaMA-8B, a combined reward (answer plus fine-grained metacognition/CoT rewards) reaches 89.25% on WildJailbreak, at a small cost in helpfulness.

  • Among reward models, Skywork worked best. It beat ArmoRM-Llama3-8B-v0.1 and an LLM-as-Judge setup with Qwen3-30B-A3B-Instruct-2507, which scored 23.40 on AlpacaEval2 and 65.00 on WildJailbreak.

  • Limitations: experiments use only small (~7B-scale) models and English-only tasks.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Renmin University of ChinaBeijing · China139 works · 17,752 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,685 worksBias, fairness & safety · 587 works

Cite

Cite as Zhihao Xu, Fuzhen Yang, Liang Lin and Xiting Wang. 2026. PAM: Enhancing General Alignment of Large Reasoning Models through Priority-Aware Metacognition. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.432

@inproceedings{xu2026pam,
  title = {{PAM: Enhancing General Alignment of Large Reasoning Models through Priority-Aware Metacognition}},
  author = {Zhihao Xu, Fuzhen Yang, Liang Lin and Xiting Wang},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.432},
  url = {https://doi.org/10.18653/v1/2026.acl-long.432}
}

References

36 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anastasia Efklides. 2006. Metacognition and affect: What can metacognitive experiences tell us about the learning process? Educational research review, 1(1):3–14. find
  2. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925.
  3. Pranjal Aggarwal and Sean Welleck. 2025. L1: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprint arXiv:2503.04697.
  4. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
  5. Ge Chang, Jinbo Su, Jiacheng Liu, Pengfei Yang, Yuhao Shang, Huiwen Zheng, Hongli Ma, Yan Liang, Yuanchun Li, and Yunxin Liu. 2025. Grail: Learning to interact with large knowledge graphs for retrieval augmented reasoning. arXiv preprint arXiv:2508.05498. — [arXiv:2508.05498](https://arxiv.org/abs/2508.05498) find
  6. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv prep…
  7. Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. 2023. Ultrafeedback: Boosting language models with high-quality feedback. find
  8. Aniket Didolkar, Nicolas Ballas, Sanjeev Arora, and Anirudh Goyal. 2025. Metacognitive reuse: Turning recurring llm reasoning into concise behaviors. arXiv preprint arXiv:2509.13237. find
  9. Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Haonan Dong, Haoran Ye, Wenhao Zhu, Kehan Jiang, and Guojie Song. 2025. Meta-r1: Empowering large reasoning models with metacognition. arXiv preprint arXiv:2508.17291. — [arXiv:2508.17291](https://arxiv.org/abs/2508.17291) find
  10. John H Flavell. 1979. Metacognition and cognitive monitoring: A new area of cognitive–developmental inquiry. American psychologist, 34(10):906. find
  11. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  12. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  13. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874.
  14. Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, et al. 2024. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint arXiv:2405.11143.
  15. Janis E Jacobs and Scott G Paris. 1987. Children’s metacognition about reading: Issues in definition, measurement, and instruction. Educational psychologist, 22(3-4):255–278. find
  16. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07974.
  17. Jiaming Ji, Donghai Hong, Borong Zhang, Boyuan Chen, Josef Dai, Boren Zheng, Tianyi Qiu, Boxun Li, and Yaodong Yang. 2024. Pku-saferlhf: A safety alignment preference dataset for llama family models. arXiv e-prints, pages arXiv–2406. find
  18. Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, et al. 2024. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. Advances in Neural Information Processing Systems,…
  19. Daniel Kahneman. 2011. macmillan. find
  20. Xiaomin Li, Zhou Yu, Zhiwei Zhang, Xupeng Chen, Ziji Zhang, Yingying Zhuang, Narayanan Sadagopan, and Anurag Beniwal. 2025. When thinking fails: The pitfalls of reasoning for instruction-following in llms. arXiv preprint arXiv:2505.11423. — [arXiv:2505.11423](https://arxiv.org/abs/2505.11423) find
  21. Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. 2023. Alpacaeval: An automatic evaluator of instruction-following models. find
  22. Tong Wu, Chong Xiang, Jiachen T Wang, G Edward Suh, and Prateek Mittal. 2025b. Effectively controlling reasoning models through thinking intervention. arXiv preprint arXiv:2503.24370. — [arXiv:2503.24370](https://arxiv.org/abs/2503.24370) find
  23. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  24. Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. 2025. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387.
  25. Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. 2025. Demystifying long chain-of-thought reasoning in llms. arXiv preprint arXiv:2502.03373.
  26. Xiaoyun Zhang, Jingqing Ruan, Xing Ma, Yawen Zhu, Haodong Zhao, Hao Li, Jiansong Chen, Ke Zeng, and Xunliang Cai. 2025. When to continue thinking: Adaptive thinking mode switching for efficient reasoning. arXiv preprint arXiv:2505.15400. find
  27. Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. 2025. s1: Simple test-time scaling. arXiv preprint arXiv:2501.19393.
  28. Weixiang Zhao, Xingyu Sui, Jiahe Guo, Yulin Hu, Yang Deng, Yanyan Zhao, Bing Qin, Wanxiang Che, Tat-Seng Chua, and Ting Liu. 2025. Trade-offs in large reasoning models: An empirical analysis of deliberative and adaptive reasoning over foundational capabilities. arXiv preprint arXiv:2503.17979. — [arXiv:2503.17979](https://arxiv.org/abs/2503.17979) find
  29. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.
  30. Kaiwen Zhou, Chengzhi Liu, Xuandong Zhao, Shreed-har Jangam, Jayanth Srinivasa, Gaowen Liu, Dawn Song, and Xin Eric Wang. 2025. The hidden risks of large reasoning models: A safety assessment of r1. arXiv preprint arXiv:2502.12659.
  31. Gregory Schraw and David Moshman. 1995. Metacognitive theories. Educational psychology review, 7(4):351–371. find
  32. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
  33. Yuqing Wang and Yun Zhao. 2023. Metacognitive prompting improves understanding in large language models. arXiv preprint arXiv:2308.05342. find
  34. Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J Zhang, Makesh Narsimhan Sreedhar, and Oleksii Kuchaiev. 2024. Helpsteer2: Open-source dataset for training top-performing reward models. arXiv preprint arXiv:2406.08673.
  35. Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, et al. 2025. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond. arXiv preprint arXiv:2503.10460.
  36. Mingqi Wu, Zhihao Zhang, Qiaole Dong, Zhiheng Xi, Jun Zhao, Senjie Jin, Xiaoran Fan, Yuhao Zhou, Huijie Lv, Ming Zhang, et al. 2025a. Reasoning or memorization? unreliable results of reinforcement learning due to data contamination. arXiv preprint arXiv:2507.10532. — [arXiv:2507.10532](https://arxiv.org/abs/2507.10532) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xu-2026-pam-enhancing-general-alignment-large/