Elamur Research⌕
☰
home / papers / 2024 / AI Sandbagging: Language Models can Strategically Underperform on Evaluations

AI Sandbagging: Language Models can Strategically Underperform on Evaluations

Paper · 2024 · Teun van der Weij, Felix Hofstatter, Ollie Jaffe et al.

Summary

The paper examines whether today's language models can deliberately score lower on capability tests than they really could. The authors call this behavior sandbagging. They define it as intentional underperformance on an evaluation. It can be driven either by the developer (developer sandbagging) or by the system itself (AI system sandbagging).

They ran two kinds of experiments:

  • Prompting: frontier models such as GPT-4 and Claude 3 Opus were asked to fail dangerous-capability questions (WMDP) while still answering general-knowledge questions (MMLU) well.
  • Fine-tuning: smaller open models were trained on synthetic data so that a capability stays hidden unless a password appears in the prompt.

Both approaches produced selective underperformance. The password-locking learned from synthetic data transferred to the real WMDP benchmark. Models could also be steered toward a chosen accuracy level. Imitating the answer pattern of a weaker model worked only moderately well.

This matters because labs and regulators increasingly base deployment decisions on such evaluations. The results show that black-box behavioral testing alone may be unreliable.

Key points

  • The setup separates dangerous-capability benchmarks (WMDP) from general ones (MMLU). The aim is a low score on the first while keeping performance on the second.
  • Mistral 7b and Llama 3 8b were password-locked using synthetic data, and the hidden capability generalized to held-out WMDP questions. A BERT-embedding similarity check found that how closely the synthetic data resembled real data made little difference.
  • Both frontier and small models could be prompted or fine-tuned to hit target accuracy levels.
  • Emulating "high-school level" ability on MMLU subsets mostly failed zero-shot. A chain-of-thought scratchpad helped somewhat, but GPT-4 tended to overshoot on college questions and Claude 3 tended to undershoot on high-school questions.
  • Code is available at github.com/TeunvdWeij/sandbagging, and the paper includes the MAPS canary string.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Imperial College LondonLondon · United Kingdom56 works · 13,671 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksSocial Sciences · 2,009 worksSafety Research · 665 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Teun van der Weij, Felix Hofstatter, Ollie Jaffe, Samuel F. Brown and Francis Rhys Ward. 2024. AI Sandbagging: Language Models can Strategically Underperform on Evaluations. arXiv:2406.07358.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.07358 · arXiv 2406.07358

@misc{weij2024ai,
  title = {{AI Sandbagging: Language Models can Strategically Underperform on Evaluations}},
  author = {Teun {van der Weij}, Felix Hofstatter, Ollie Jaffe, Samuel F. Brown and Francis Rhys Ward},
  year = {2024},
  doi = {10.48550/arxiv.2406.07358},
  eprint = {2406.07358},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.07358}
}

References

86 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI Safety Institute (2024). Inspect An open-source framework for large language model evaluations. https://ukgovernmentbeis.github.io/inspect_ai/. [Accessed 18-05-2024]. ↗
  2. Ajeya Cotra (2022). Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover. https://www.alignmentforum.org/posts/ pRkFkzwKZ2zfa3R6H/without-specific-countermeasures-the-easiest-path-to. [Accessed: 19-05-2024]. ↗
  3. Alex Albert (2024). Fun story from our internal testing on Claude 3 Opus. https://twitter.com/alexalbert__/status/1764722513014329620. [Accessed: 19-05-2024]. ↗
  4. Anthropic (2023). Anthropic’s Responsible Scaling Policy. find
  5. Anthropic (2024a). Introducing the next generation of Claude. https://www.anthropic.com/ news/claude-3-family. [Accessed 18-05-2024]. ↗
  6. Anthropic (2024b). Simple probes can catch sleeper agents. https://www.anthropic.com/ research/probes-catch-sleeper-agents. [Accessed: 27-04-2024]. ↗
  7. Anthropic (2024). The Claude 3 Model Family: Opus, Sonnet, Haiku. find
  8. Antrhopic (2024). Challenges in evaluating AI systems. https://www.anthropic.com/news/ evaluating-ai-systems. [Accessed 22-05-2024]. ↗
  9. Apollo Research (2023). Understanding strategic deception and deceptive alignment. https://www.apolloresearch.ai/blog/understanding-strategic-deception-and-deceptive-alignment. [Accessed 18-05-2024]. ↗
  10. Ashton, H. (2023). Definitions of intent suitable for algorithms. Artificial Intelligence and Law, 31(3):515–546. find
  11. Bowman, S. R. (2023). Eight Things to Know about Large Language Models. find
  12. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… — Language Models are Few-Shot Learners
  13. BSA (2023). AI Act. https://digital-strategy.ec.europa.eu/en/policies/ regulatory-framework-ai. [Accessed 30-04-2024]. ↗
  14. Carlini, N., Jagielski, M., Choquette-Choo, C. A., Paleka, D., Pearce, W., Anderson, H., Terzis, A., Thomas, K., and Tramèr, F. (2023). Poisoning Web-Scale Training Datasets is Practical. find
  15. Carlsmith, J. (2023). Scheming AIs: Will AIs fake alignment during training in order to get power? arXiv preprint arXiv:2311.08379. — [arXiv:2311.08379](https://arxiv.org/abs/2311.08379) find
  16. Casper, S., Ezell, C., Siegmann, C., Kolt, N., Curtis, T. L., Bucknall, B., Haupt, A., Wei, K., Scheurer, J., Hobbhahn, M., et al. (2024). Black-Box Access is Insufficient for Rigorous AI Audits. arXiv preprint arXiv:2401.14446. — [arXiv:2401.14446](https://arxiv.org/abs/2401.14446) find
  17. Epoch AI (2023). Key Trends and Figures in Machine Learning. https://epochai.org/ trends. [Accessed: 19-05-2024]. ↗
  18. European Commission (2024a). AI Act. https://digital-strategy.ec.europa.eu/en/ policies/regulatory-framework-ai. [Accessed 30-04-2024]. ↗
  19. European Commission (2024b). European AI Office. https://digital-strategy.ec.europa.eu/en/policies/ai-office. [Accessed 30-04-2024]. ↗
  20. European Parliament (2024). Corrigendum. https://www.europarl.europa.eu/doceo/ document/TA-9-2024-0138-FNL-COR01_EN.pdf. [Accessed: 13-05-2024, pages 294– 295]. ↗
  21. Gekhman, Z., Yona, G., Aharoni, R., Eyal, M., Feder, A., Reichart, R., and Herzig, J. (2024). Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? arXiv preprint arXiv:2405.05904. find
  22. Google (2024). Gemini: A Family of Highly Capable Multimodal Models.
  23. Google DeepMind (2024). Frontier Safety Framework. find
  24. Greenblatt, R., Roger, F., Krasheninnikov, D., and Krueger, D. (2024). Stress-Testing Capability Elicitation With Password-Locked Models. arXiv preprint arXiv:2405.19550. — [arXiv:2405.19550](https://arxiv.org/abs/2405.19550) find
  25. Greenblatt, R., Shlegeris, B., Sachan, K., and Roger, F. (2023). AI control: Improving safety despite intentional subversion. arXiv preprint arXiv:2312.06942. — [arXiv:2312.06942](https://arxiv.org/abs/2312.06942) find
  26. Gurnee, W. and Tegmark, M. (2023). Language models represent space and time. arXiv preprint arXiv:2310.02207. find
  27. Halpern, J. and Kleiman-Weiner, M. (2018). Towards formal definitions of blameworthiness, intention, and moral responsibility. In Proceedings of the AAAI conference on artificial intelligence, volume 32. find
  28. Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. (2020). Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  29. Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. (2022). Training compute-optimal large language models. arXiv preprint arXiv:2203.15556.
  30. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2021). LoRA: Low-Rank Adaptation of Large Language Models.
  31. Huang, W., Zheng, X., Ma, X., Qin, H., Lv, C., Chen, H., Luo, J., Qi, X., Liu, X., and Magno, M. (2024). An empirical study of llama3 quantization: From llms to mllms. find
  32. Hubinger, E. (2023). When can we trust model evaluations? https://www.alignmentforum.org/posts/dBmfb76zx6wjPsBC7/when-can-we-trust-model-evaluations. [Accessed 30-04-2024]. ↗
  33. Hubinger, E., Denison, C., Mu, J., Lambert, M., Tong, M., MacDiarmid, M., Lanham, T., Ziegler, D. M., Maxwell, T., Cheng, N., Jermyn, A., Askell, A., Radhakrishnan, A., Anil, C., Duvenaud, D., Ganguli, D., Barez, F., Clark, J., Ndousse, K., Sachan, K., Sellitto, M., Sharma, M., DasSarma, N., Grosse,… — Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
  34. Hubinger, E., van Merwijk, C., Mikulik, V., Skalse, J., and Garrabrant, S. (2021). Risks from Learned Optimization in Advanced Machine Learning Systems. find
  35. Jain, S., Kirk, R., Lubana, E. S., Dick, R. P., Tanaka, H., Grefenstette, E., Rockt"aschel, T., and Krueger, D. S. (2023). Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. arXiv preprint arXiv:2311.12786. — [arXiv:2311.12786](https://arxiv.org/abs/2311.12786) find
  36. Jiang, A. Q., Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux, M.-A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. (2023). Mistral 7B. find
  37. Järviniemi, O. and Hubinger, E. (2024). Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistant. arXiv preprint arXiv:2405.01576. — [arXiv:2405.01576](https://arxiv.org/abs/2405.01576) find
  38. Kinniment, M., Sato, L. J. K., Du, H., Goodrich, B., Hasin, M., Chan, L., Miles, L. H., Lin, T. R., Wijk, H., Burget, J., Ho, A., Barnes, E., and Christiano, P. (2024). Evaluating Language-Model Agents on Realistic Autonomous Tasks. find
  39. Leike, J. (2023). Self-exfiltration is a key dangerous capability . https://aligned.substack.com/p/self-exfiltration. [Accessed 15-05-2024]. ↗
  40. Li, N., Pan, A., Gopal, A., Yue, S., Berrios, D., Gatti, A., Li, J. D., Dombrowski, A.-K., Goel, S., Phan, L., et al. (2024). The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning. arXiv preprint arXiv:2403.03218.
  41. Lynch, A., Guo, P., Ewart, A., Casper, S., and Hadfield-Menell, D. (2024). Eight Methods to Evaluate Robust Unlearning in LLMs. arXiv preprint arXiv:2402.16835. — [arXiv:2402.16835](https://arxiv.org/abs/2402.16835) find
  42. Maham, P. and Küspert, S. (2023). Governing General Purpose AI. Policy. find
  43. Mallen, A., Brumley, M., Kharchenko, J., and Belrose, N. (2024). Eliciting Latent Knowledge from Quirky Language Models. find
  44. Maloyan, N., Verma, E., Nutfullin, B., and Ashinov, B. (2024). Trojan Detection in Large Language Models: Insights from The Trojan Detection Challenge. arXiv preprint arXiv:2404.13660. — [arXiv:2404.13660](https://arxiv.org/abs/2404.13660) find
  45. Meta (2024). Introducing Meta Llama 3: The most capable openly available LLM to date. https://ai.meta.com/blog/meta-llama-3/. [Accessed: 30-04-2024]. ↗
  46. METR (2024). Details about metr’s preliminary evaluation of openai o1-preview. /autonomy-evals-guide/openai-o1-preview-report/. find
  47. Ngo, R., Chan, L., and Mindermann, S. (2022). The alignment problem from a deep learning perspective. arXiv preprint arXiv:2209.00626.
  48. NIST (2024). U.S. Artificial Intelligence Safety Institute \| NIST. https://www.nist.gov/aisi. [Accessed 20-05-2024]. ↗
  49. Open AI (2024). Developing beneficial AGI safely and responsibly. https://openai.com/ safety/. [Accessed 18-05-2024]. ↗
  50. OpenAI (2023). Preparedness Framework (Beta). find
  51. OpenAI (2024). GPT-4 Technical Report.
  52. OpenAI (2024a). Models documentation. https://platform.openai.com/docs/models/ overview. [Accessed 04-05-2024]. ↗
  53. OpenAI (2024b). OpenAI 01 System Card. find
  54. OpenAI (2024c). OpenAI o1 System Card. [Online; accessed 17.
  55. Pacchiardi, L., Chan, A. J., Mindermann, S., Moscovitz, I., Pan, A. Y., Gal, Y., Evans, O., and Brauner, J. (2023). How to catch an AI liar: Lie detection in black-box LLMs by asking unrelated questions. arXiv preprint arXiv:2309.15840.
  56. Park, P. S., Goldstein, S., O’Gara, A., Chen, M., and Hendrycks, D. (2023a). AI deception: A survey of examples, risks, and potential solutions. arXiv preprint arXiv:2308.14752. — [arXiv:2308.14752](https://arxiv.org/abs/2308.14752) find
  57. Park, P. S., Goldstein, S., O’Gara, A., Chen, M., and Hendrycks, D. (2023b). AI Deception: A Survey of Examples, Risks, and Potential Solutions. find
  58. Perez, E., Ringer, S., Lukošiūtė, K., Nguyen, K., Chen, E., Heiner, S., Pettit, C., Olsson, C., Kundu, S., Kadavath, S., Jones, A., Chen, A., Mann, B., Israel, B., Seethor, B., McKinnon, C., Olah, C., Yan, D., Amodei, D., Amodei, D., Drain, D., Li, D., Tran-Johnson, E., Khundadze, G., Kernion, J.,… — Discovering Language Model Behaviors with Model-Written Evaluations
  59. Phuong, M., Aitchison, M., Catt, E., Cogan, S., Kaskasoli, A., Krakovna, V., Lindner, D., Rahtz, M., Assael, Y., Hodkinson, S., et al. (2024). Evaluating Frontier Models for Dangerous Capabilities. arXiv preprint arXiv:2403.13793.
  60. Price, S., Strickland, A. C., and Bowman, S. (2024). Future events as backdoor triggers: investigating temporal vulnerabilities in LLMs. (Forthcoming). find
  61. Roger, F. (2023). Password-locked models: a stress case for capabilities evaluation. https://www.alignmentforum.org/posts/rZs6ddqNnW8LXuJqA/password-locked-models-a-stress-case-for-capabilities. [Accessed 04-05-2024]. ↗
  62. Schaeffer, R. (2023). Pretraining on the test set is all you need. arXiv preprint arXiv:2309.08632.
  63. Shah, R., Varma, V., Kumar, R., Phuong, M., Krakovna, V., Uesato, J., and Kenton, Z. (2022a). Goal misgeneralization: why correct specifications aren’t enough for correct goals. arXiv preprint arXiv:2210.01790. — [arXiv:2210.01790](https://arxiv.org/abs/2210.01790) find
  64. Shah, R., Varma, V., Kumar, R., Phuong, M., Krakovna, V., Uesato, J., and Kenton, Z. (2022b). Goal misgeneralization: why correct specifications aren’t enough for correct goals. arXiv preprint arXiv:2210.01790. — [arXiv:2210.01790](https://arxiv.org/abs/2210.01790) find
  65. Shavit, Y., Agarwal, S., Brundage, M., Adler, S., O’Keefe, C., Campbell, R., Lee, T., Mishkin, P., Eloundou, T., Hickey, A., et al. (2023). Practices for Governing Agentic AI Systems. Research Paper, OpenAI, December. find
  66. Shevlane, T., Farquhar, S., Garfinkel, B., Phuong, M., Whittlestone, J., Leung, J., Kokotajlo, D., Marchal, N., Anderljung, M., Kolt, N., et al. (2023). Model evaluation for extreme risks. arXiv preprint arXiv:2305.15324.
  67. Talmor, A., Herzig, J., Lourie, N., and Berant, J. (2018). Commonsenseqa: A question answering challenge targeting commonsense knowledge. arXiv preprint arXiv:1811.00937.
  68. The White House (2024). Biden-Harris Administration Announces Key AI Actions 180 Days Following President Biden’s Landmark Executive Order. https://www.whitehouse.gov/briefing-room/statements-releases/2024/04/ 29/biden-harris-administration-announces-key-ai-actions-180-days-following-president-bide… ↗
  69. Thomas, R. and Uminsky, D. (2020). The problem with metrics is a fundamental problem for AI. arXiv preprint arXiv:2002.08512. — [arXiv:2002.08512](https://arxiv.org/abs/2002.08512) find
  70. Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hossei… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  71. UK AISI (2024). AI Safety Institute approach to evaluations. https://www.gov.uk/ government/publications/ai-safety-institute-approach-to-evaluations/ai-safety-institute-approach-to-evaluations. [Accessed 30-04-2024]. ↗
  72. UK DSIT (2023a). Capabilities and risks from frontier AI. https://assets.publishing.service.gov.uk/media/65c0b6bd63a23d0013c821a0/ implementing_the_uk_ai_regulatory_principles_guidance_for_regulators.pdf. [Accessed 22-05-2024]. ↗
  73. UK DSIT (2023b). Emerging processes for frontier AI safety. https://www.gov.uk/ government/publications/emerging-processes-for-frontier-ai-safety/emerging-processes-for-frontier-ai-safety\#model-evaluations-and-red-teaming. [Accessed 18-05-2024]. ↗
  74. UK DSIT (2023c). Implementing the UK’s AI Regulatory Principles. https://assets.publishing.service.gov.uk/media/65c0b6bd63a23d0013c821a0/ implementing_the_uk_ai_regulatory_principles_guidance_for_regulators.pdf. [Accessed 30-04-2024]. ↗
  75. UK DSIT (2024). AI Safety Institute approach to evaluations. https://www.gov.uk/ government/publications/ai-safety-institute-approach-to-evaluations/ai-safety-institute-approach-to-evaluations. [Accessed 21-05-2024]. ↗
  76. Villalobos, P. and Atkinson, D. (2023). Trading Off Compute in Training and Inference. https://epochai.org/blog/trading-off-compute-in-training-and-inference. [Accessed: 2024-05-04]. ↗
  77. Wan, A., Wallace, E., Shen, S., and Klein, D. (2023). Poisoning Language Models During Instruction Tuning. find
  78. Wang, G., Xie, Y., Jiang, Y., Mandlekar, A., Xiao, C., Zhu, Y., Fan, L., and Anandkumar, A. (2023). Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291. find
  79. Ward, F., Toni, F., Belardinelli, F., and Everitt, T. (2024a). Honesty is the best policy: defining and mitigating AI deception. Advances in Neural Information Processing Systems, 36. find
  80. Ward, F. R., MacDermott, M., Belardinelli, F., Toni, F., and Everitt, T. (2024b). The reasons that agents act: Intention and instrumental goals. arXiv preprint arXiv:2402.07221. — [arXiv:2402.07221](https://arxiv.org/abs/2402.07221) find
  81. Wei, F., Chen, X., and Luo, L. (2024). Rethinking Generative Large Language Model Evaluation for Semantic Comprehension. arXiv preprint arXiv:2403.07872. — [arXiv:2403.07872](https://arxiv.org/abs/2403.07872) find
  82. Wei, J., Bosma, M., Zhao, V. Y., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. (2022a). Finetuned Language Models Are Zero-Shot Learners.
  83. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022b). Emergent abilities of large language models. arXiv preprint arXiv:2206.07682.
  84. Wikipedia (2024a). Overfitting. https://en.wikipedia.org/wiki/Overfitting. [Accessed 07-06-2024]. ↗
  85. Wikipedia (2024b). Sandbagging. https://en.wikipedia.org/wiki/Sandbagging. [Accessed 29-04-2024]. ↗
  86. Wikipedia (2024c). Volkswagen emissions scandal. http://en.wikipedia.org/ w/index.php?title=Volkswagen%20emissions%20scandal&oldid= 1220907949. [Accessed 29-04-2024]. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/weij-2024-ai-sandbagging-language-models-strategically/