Elamur Research⌕
☰
home / papers / 2026 / Who Endorsed It? Measuring Authority Bias Across Expertise Levels in Language Models

Who Endorsed It? Measuring Authority Bias Across Expertise Levels in Language Models

Paper · 2026 · Priyanka Mary Mammen, Emil Joswin and Shankar Venkitachalam

Summary

The paper asks whether language models trust an answer more depending on how expert its endorser appears. Earlier work on sycophancy and authority bias usually treated authority as an on/off property. Here, authority is modelled as a graded scale. The authors build multiple-choice prompts from math, legal and medical benchmarks. Each question appears once without a hint and eight more times with a one-line hint attributed to one of four domain-specific personas of rising rank. Half of these hints point to the right option and half to a wrong one. They test 11 open models, both reasoning and non-reasoning, reading answer probabilities directly from the output logits. The effect follows the hierarchy: more senior endorsers produce larger gains when they are right, and larger accuracy losses and greater confidence in wrong answers when they mislead. A steering vector built from writing style alone captures this "expertise" direction, and subtracting it reduces the bias. This matters for deploying LLMs as decision aids in high-stakes fields.

Key points

  • Datasets: AQUA-RAT (algebraic reasoning), LEXam (English questions only), MedMCQA and MedQA.
  • Persona tiers:
    • Science: High Schooler → Undergrad → Grad Student → Professor.
    • Law: Undergraduate Law Student → Senior Legal Counsel.
    • Medicine: First-Year Medical Student → Board-Certified Physician.
  • Models: 4 reasoning models (e.g., Qwen3-4B-Thinking, DeepSeek-R1-Qwen3-8B, Phi-4-Reasoning, Olmo-3.1-32B-Think) and 7 non-reasoning models (e.g., LLaMA-3.1-8B, Gemma-3-12B-IT, Mistral-7B).
  • Metrics: ΔAccuracy, ΔEntropy and Robustness Rate, all measured against the no-hint baseline.
  • Representative example: DeepSeek-R1 on MedQA has a baseline accuracy of 0.543. Correct hints raise it by up to +0.381, and misleading hints lower it by between −0.019 and −0.356, with the size of the shift growing with endorser rank.
  • Steering vector: It is the mean difference in residual-stream activations between professor-style and high-schooler-style answers, using about 30 questions per domain with no persona labels. Subtracting it restores accuracy toward baseline under misleading Board-Certified Physician hints on MedMCQA, while adding it increases trust in low-credibility endorsers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Massachusetts AmherstAmherst Center · United States114 works · 68,731 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Priyanka Mary Mammen, Emil Joswin and Shankar Venkitachalam. 2026. Who Endorsed It? Measuring Authority Bias Across Expertise Levels in Language Models. Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.75

@inproceedings{mammen2026who,
  title = {{Who Endorsed It? Measuring Authority Bias Across Expertise Levels in Language Models}},
  author = {Priyanka Mary Mammen, Emil Joswin and Shankar Venkitachalam},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.75},
  url = {https://doi.org/10.18653/v1/2026.gem-main.75}
}

References

37 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marah Abdin, Sahaj Agarwal, Ahmed Awadallah, Vidhisha Balachandran, Harkirat Behl, Lingjiao Chen, Gustavo de Rosa, Suriya Gunasekar, Mojan Javaheripi, Neel Joshi, and 1 others. 2025. Phi-4-reasoning technical report. arXiv preprint arXiv:2504.21318.
  2. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, and 1 others. 2024. Phi-4 technical report. arXiv preprint arXiv:2412.08905.
  3. Haozhe An, Christabel Acquaye, Colin Wang, Zongxia Li, and Rachel Rudinger. 2024. Do large language models discriminate in hiring decisions on the basis of race, ethnicity, and gender? arXiv preprint arXiv:2406.10486. find
  4. Noel F Ayoub, Karthik Balakrishnan, Marc S Ayoub, Thomas F Barrett, Abel P David, and Stacey T Gray. 2024. Inherent bias in large language models: a random sampling analysis. Mayo Clinic Proceedings: Digital Health, 2(2):186–191. find
  5. Guiming Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or llms as the judge? a study on judgement bias. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8301–8327.
  6. Yanda Chen, Joe Benton, Ansh Radhakrishnan, Jonathan Uesato, Carson Denison, John Schulman, Arushi Somani, Peter Hase, Misha Wagner, Fabien Roger, and 1 others. 2025. Reasoning models don’t always say what they think. arXiv preprint arXiv:2505.05410.
  7. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. 2024. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475. find
  8. Yu Fan, Jingwei Ni, Jakob Merane, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Etienne Salimbeni, Florian Geering, Oliver Dreyer, and 1 others. 2025. Lexam: Benchmarking legal reasoning on 340 law exams. arXiv preprint arXiv:2505.12864. find
  9. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  10. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  11. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  12. Di Jin, Eileen Pan, Nassim Oufattole, Wei-Hung Weng, Hanyi Fang, and Peter Szolovits. 2021. What disease does this patient have? a large-scale open domain question answering dataset from medical exams. Applied Sciences, 11(14):6421.
  13. Ryan Koo, Minhwa Lee, Vipul Raheja, Jong Inn Park, Zae Myung Kim, and Dongyeop Kang. 2024. Benchmarking cognitive biases in large language models as evaluators. In Findings of the Association for Computational Linguistics: ACL 2024, pages 517–545. find
  14. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. 2017. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146. find
  15. Zehao Liu and Xi Lin. 2025. Breaking minds, breaking systems: Jailbreaking large language models via human-like psychological manipulation. arXiv preprint arXiv:2512.18244. — [arXiv:2512.18244](https://arxiv.org/abs/2512.18244) find
  16. Team Olmo, Allyson Ettinger, Amanda Bertsch, Bailey Kuehl, David Graham, David Heineman, Dirk Groeneveld, Faeze Brahman, Finbarr Timbers, Hamish Ivison, and 1 others. 2025. Olmo 3. arXiv preprint arXiv:2512.13961.
  17. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2022. Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Conference on health, inference, and learning, pages 248–260. PMLR.
  18. Junsoo Park, Seungyeon Jwa, Ren Meiying, Daeyoung Kim, and Sanghyuk Choi. 2024. Offsetbias: Leveraging debiased data for tuning evaluators. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 1043–1067.
  19. Pouya Pezeshkpour and Estevam Hruschka. 2023. Large language models sensitivity to the order of options in multiple-choice questions, 2023. URL https://arxiv.org/abs/2308.11483. ↗
  20. Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. Verbosity bias in preference labeling by large language models. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following. find
  21. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R Johnston, and 1 others. 2023. Towards understanding sycophancy in language models. arXiv preprint arXiv:2310.13548.
  22. Lin Shi, Chiyu Ma, Wenhua Liang, Xingjian Diao, Weicheng Ma, and Soroush Vosoughi. 2024. Judging the judges: A systematic study of position bias in llm-as-a-judge. arXiv preprint arXiv:2406.07791.
  23. Andreas Stephan, Dawei Zhu, Matthias Aßenmacher, Xiaoyu Shen, and Benjamin Roth. 2025. From calculation to adjudication: Examining llm judges on mathematical reasoning tasks. In Proceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM 2 ), pages 759–773.
  24. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  25. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024a. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  26. Qwen Team and 1 others. 2024b. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2(3). find
  27. Alexander Matt Turner, Lisa Thiergart, Gavin Leech, David Udell, Juan J. Vazquez, Ulisse Mini, and Monte MacDiarmid. 2024. Steering language models with activation engineering. Preprint, arXiv:2308.10248.
  28. Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and 1 others. 2024. Large language models are not fair evaluators. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pag… find
  29. Qian Wang, Zhanzhi Lou, Zhenheng Tang, Nuo Chen, Xuandong Zhao, Wenxuan Zhang, Dawn Song, and Bingsheng He. 2025. Assessing judging bias in large reasoning models: An empirical study. arXiv preprint arXiv:2504.09946. — [arXiv:2504.09946](https://arxiv.org/abs/2504.09946) find
  30. Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V Le. 2023. Simple synthetic data reduces sycophancy in large language models. arXiv preprint arXiv:2308.03958.
  31. Minghao Wu and Alham Fikri Aji. 2023. Style over substance: Evaluation biases for large language models.” arxiv. arXiv preprint arXiv:2307.03025.
  32. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  33. Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, and 1 others. 2024. Justice or prejudice? quantifying biases in llm-as-ajudge. arXiv preprint arXiv:2410.02736.
  34. Zheng Zhang, Peilin Zhao, Deheng Ye, and Hao Wang. 2025. Enhancing jailbreak attacks on llms via persona prompts. arXiv preprint arXiv:2507.22171. layers of the network, approximately in the range [L/3, 2L/3], consistent with prior work on activation steering. A systematic layer-wise ablation remain… — [arXiv:2507.22171](https://arxiv.org/abs/2507.22171) find
  35. Lili Zhao, Yang Wang, Qi Liu, Mengyun Wang, Wei Chen, Zhichao Sheng, and Shijin Wang. 2025. Evaluating large language models through role-guide and self-reflection: A comparative study. In The Thirteenth International Conference on Learning Representations. find
  36. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.
  37. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, and 1 others. 2023. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mammen-2026-who-endorsed-it-measuring-authority/