Elamur Research⌕
☰
home / papers / 2026 / SycoBench-600: Measuring Sycophancy and Correction Selectivity in LLM Assistants

SycoBench-600: Measuring Sycophancy and Correction Selectivity in LLM Assistants

Paper · 2026 · Debu Sinha

Summary

LLM assistants tuned for helpfulness often give in to users who are wrong, yet they should still accept genuine corrections. Existing sycophancy benchmarks mostly measure how easily models yield to pressure. They do not separate accepting a valid fix from caving to a false claim. SycoBench-600 addresses this with a controlled multiple-choice setup. It contains 600 hand-written English items built on 272 normalized stems, spanning 8 domains and three difficulty tiers. Each item is tested under three misleading pressures (doubt, appeal to authority, an explicit wrong suggestion) and a matched condition in which the user proposes the correct answer. Every perturbation is run with three fixed prompt paraphrases. The author introduces correction selectivity, defined as the update rate minus the wrong-flip rate, and evaluates seven proprietary and open-weight assistants. The central finding is that resisting pressure and accepting correction are separate abilities. Models fall into three groups: robust and selective, robust but stubborn, or quick to update in either direction. This matters for building assistants that are hard to mislead yet still correctable. Raw logs and regeneration scripts are released.

Key points

  • Dataset and protocol: 75 items per domain, with a 20/40/40 split across easy, medium and hard. Models must answer with a single letter (A–D) at temperature 0. The parser takes the last standalone capital letter in the response.
  • Evaluation scope: Final results use a 555-question intersection. Forty-five flawed items were repaired and rerun, but Mistral-7B was retired from its provider and could not be rerun on them, so those items were dropped for all models. Confidence intervals come from a cluster bootstrap over question IDs.
  • Best overall: Gemini-2.5-Flash had the highest baseline accuracy (95.3%) and pressure-robust accuracy (81.7%).
  • Other model results:
    • GPT-4o reached the highest selectivity (71.6).
    • Mistral-7B was the most sycophantic, flipping to wrong answers 50.5% of the time.
    • Llama-4-Maverick flipped least (5.9%), but 31% of its baseline-wrong runs were unparseable and excluded from correction metrics.
    • GPT-4o-mini was the most stubborn (64.7% no-change under correct suggestions).
    • Claude-3.5-Haiku almost always accepted corrections (0.2% stubbornness) but also caved often (36.4% sycophancy).
  • Pressure types: The three pressures have different effects. Authority prompts were the strongest for most models, while explicit wrong suggestions hurt Mistral-7B and Llama-4-Maverick most.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Debu Sinha. 2026. SycoBench-600: Measuring Sycophancy and Correction Selectivity in LLM Assistants. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1759

@inproceedings{sinha2026sycobench,
  title = {{SycoBench-600: Measuring Sycophancy and Correction Selectivity in LLM Assistants}},
  author = {Debu Sinha},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1759},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1759}
}

References

23 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Solomon E. Asch. 1951. Effects of group pressure upon the modification and distortion of judgments. In Groups, Leadership and Men. find
  2. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Nat McAleese, and 1 others. 2022. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073.
  3. Jan Batzner, Volker Stocker, Stefan Schmid, and Gjergji Kasneci. 2024. GermanPartiesQA: Benchmarking commercial large language models and AI companions for political alignment and sycophancy. find
  4. Emily M. Bender and Batya Friedman. 2018. Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics. find
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems.
  6. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155.
  7. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, and 1 others. 2023. A survey on evaluation of large language models.
  8. Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, and 1 others. 2022. Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251.
  9. Wei Chen, Zhen Huang, Liang Xie, Binbin Lin, Houqiang Li, Le Lu, Xinmei Tian, Deng Cai, Yonggang Zhang, Wenxiao Wang, Xu Shen, and Jieping Ye. 2024. From yes-men to truth-tellers: Addressing sycophancy in large language models with pinpoint tuning. In Proceedings of the 41st International Conference… find
  10. Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems. find
  11. Jesse Dodge, Suchin Gururangan, Dallas Card, Roy Schwartz, and Noah A. Smith. 2019. Show your work: Improved reporting of experimental results. In Proceedings of EMNLP-IJCNLP. find
  12. Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. 2018. The hitchhiker’s guide to testing statistical significance in natural language processing. In Proceedings of ACL. find
  13. Aaron Fanous, Jacob Goldberg, Ank A. Agarwal, Joanna Lin, Anson Zhou, Roxana Daneshjou, and Sanmi Koyejo. 2025. SycEval: Evaluating LLM sycophancy. find
  14. Ivo Petrov, Jasper Dekoninck, and Martin Vechev. 2025. BrokenMath: A benchmark for sycophancy in theorem proving with LLMs. find
  15. Joelle Pineau, Philippe Vincent-Lamarre, Koustuv Sinha, Vincent Lariviere, Alina Beygelzimer, Florence d’Alche Buc, Emily Fox, and Hugo Larochelle. 2021. Improving reproducibility in machine learning research. Journal of Machine Learning Research. find
  16. Jerry Wei, Da Huang, Yifeng Lu, Denny Zhou, and Quoc V. Le. 2023. Simple synthetic data reduces sycophancy in large language models.
  17. Botai Yuan, Yutian Zhou, Yingjie Wang, Fushuo Huo, Yongcheng Jing, Li Shen, Ying Wei, Zhiqi Shen, Ziwei Liu, Tianwei Zhang, Jie Yang, and Dacheng Tao. 2025. EchoBench: Benchmarking sycophancy in medical large vision-language models. find
  18. Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. 2023. Instruction-following evaluation for large language models.
  19. Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, and Kate Crawford. 2021. Datasheets for datasets. Communications of the ACM. find
  20. Jiseung Hong, Grace Byun, Seungone Kim, and Kai Shu. 2025. Measuring sycophancy of language models in multi-turn dialogues. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 2239–2259, Suzhou, China. Association for Computational Linguistics. find
  21. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics…
  22. Lars Malmqvist. 2024. Sycophancy in large language models: Causes and mitigations.
  23. Stanley Milgram. 1963. Behavioral study of obedience. Journal of Abnormal and Social Psychology, 67(4):371–378. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sinha-2026-sycobench-600-measuring-sycophancy-correction/