Elamur Research⌕
☰
home / papers / 2026 / Anchoring Depends on Confidence and Post-Training in Language Models

Anchoring Depends on Confidence and Post-Training in Language Models

Paper · 2026 · Hillary N. Owusu and Naomi H. Feldman

Summary

Large language models can be swayed by irrelevant numbers placed before a quantitative question. This is the anchoring effect known from human psychology. The authors ask whether a model's confidence or its factual correctness predicts how strongly anchors pull it. They test six models: base, instruction-tuned and DeepSeek-R1-distilled versions of Llama-3.1-8B and Qwen-7B. The test set is 50 questions from the Open Anchoring Dataset (OpAQ), each preceded by an unrelated "slot machine" number that is either low or high. Instead of sampling, they score nine fixed candidate answers with length-normalized log-likelihoods. They then use Total Variation Distance to measure how far the anchored distribution moves from the control one. Regression analyses (pooled OLS, mixed-effects and per-model) show that sharper baseline distributions resist anchoring. Distance from the true answer adds no predictive value, so a confidently wrong model is as robust as a correct one. Post-training strengthens this link, and high anchors move models more than low ones. This gives practitioners a signal for flagging vulnerable numerical outputs that does not require ground truth.

Key points

  • Models: Llama-3.1-8B (Base, Instruct, DeepSeek-R1-Distill), Qwen2.5-7B (Base, Instruct) and DeepSeek-R1-Distill-Qwen-7B, which is built on Qwen2.5-Math-7B.
  • Data: 600 observations from 50 OpAQ items × 2 anchor conditions × 6 models. The findings also hold on 99 items (1,188 observations), with main effects at p < .001.
  • Certainty vs. accuracy: Certainty is the maximum candidate probability in the control condition. It lowers TVD in pooled OLS (β = −0.037, p = .001) and in the mixed-effects model (β = −0.052, p < .001). Accuracy, measured as log-transformed relative MAE, is not significant (p = .48 and p = .74).
  • Effect of post-training: In pooled data, base models show a weak certainty slope (β = −0.012, n.s.). Instruct (interaction β = −0.072) and Distill (interaction β = −0.054) slopes are significantly steeper. Per-model fits show that base models do have an internal link, but their confidence scales are not aligned with each other.
  • Asymmetry: High anchors have a stronger effect than low ones, peaking at medium certainty. Instruction-tuned models keep 78–86% directional consistency for high anchors but only 50–64% for low anchors, compared with ≥92% for base models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, College ParkCollege Park · United States132 works · 25,380 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksDecision Sciences · 209 worksGeneral Decision Sciences · 4 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Hillary N. Owusu and Naomi H. Feldman. 2026. Anchoring Depends on Confidence and Post-Training in Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-short.16

@inproceedings{owusu2026anchoring,
  title = {{Anchoring Depends on Confidence and Post-Training in Language Models}},
  author = {Hillary N. Owusu and Naomi H. Feldman},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.16},
  url = {https://doi.org/10.18653/v1/2026.acl-short.16}
}

References

13 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dale J. Barr, R. Levy, Christoph Scheepers, and Harry J. Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of memory and language, 68 3. find
  2. Jessica Echterhoff, Yao Liu, Abeer Alessa, Julian McAuley, and Zexue He. 2024. Cognitive bias in decision-making with llms. Preprint, arXiv:2403.00811. find
  3. Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. 2023. Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt. Nature Computational Science, 3(10):833–838. find
  4. Yiming Huang, Biquan Bie, Zuqiu Na, Weilin Ruan, Songxin Lei, Yutao Yue, and Xinlei He. 2025. An empirical study of the anchoring effect in llms: Existence, mechanism, and potential mitigations. Preprint, arXiv:2505.15392. — [arXiv:2505.15392](https://arxiv.org/abs/2505.15392) find
  5. Karen E. Jacowitz and Daniel Kahneman. 1995. Measures of anchoring in estimation tasks. Personality and Social Psychology Bulletin, 21(11):1161–1166. find
  6. Each candidate string y i corresponds to a unique numeric value v i ∈ R. To measure the extent to which the distributional shift moves probability mass toward the anchor value, we compute the percentage change in the model’s estimate in each anchoring condition relative to the control baseline: Falk… find
  7. E[v \| x ctl ] − E[v \| x low ] E[v \| x ctl ] (4) ∆ high = E[v \| x high ] − E[v \| x ctl ] E[v \| x ctl ] (5) where E[v \| x] is the expected value over the candidate set: Jiaxu Lou and Yifan Sun. 2024. Anchoring bias in large language models: An experimental study. Preprint, arXiv:2412.06593. — [arXiv:2412.06593](https://arxiv.org/abs/2412.06593) find
  8. Jeremy K. Nguyen. 2024. Human bias in ai models? anchoring effects and mitigation strategies in large language models. Journal of Behavioral and Experimental Finance, 43(C):None. find
  9. Lukas Röseler, Lucia Weber, Ena P. B. Stijović, Katharina A. K. Jaekel, J. F. (Janne)ke M. T. (Janneke) G. (Gijsbers) Gijsbers, and Nir Milstein. 2022. The Open Anchoring Quest Dataset: Anchored Estimates from 96 Studies on Anchoring Effects. Journal of Open Psychology Data, 10(1):16. find
  10. Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafael Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher D. Manning. 2023. Just ask for calibration: Strategies for eliciting calibrated confidence scores from language models fine-tuned with human feedback. Preprint, arXiv:2305.14975. find
  11. Amos Tversky and Daniel Kahneman. 1974. Judgment under uncertainty: Heuristics and biases. Science, 185(4157):1124–1131. find
  12. Felipe Valencia-Clavijo. 2025. Anchors in the machine: Behavioral and attributional evidence of anchoring bias in llms. Preprint, arXiv:2511.05766. — [arXiv:2511.05766](https://arxiv.org/abs/2511.05766) find
  13. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. Huggin…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/owusu-2026-anchoring-depends-confidence-post-training/