Elamur Research⌕
☰
home / papers / 2026 / The Impact of Off-Policy Training Data on Probe Generalisation

The Impact of Off-Policy Training Data on Probe Generalisation

Paper · 2026 · Adrians Skapars, Nathalie Maria Kirch, Samuel Dower et al.

Summary

Activation probes are a cheap way to flag worrying behaviour in LLMs at inference time. However, natural labelled examples of behaviours like deception are scarce, so probes are often trained on synthetic or externally written responses. Skapars et al. test how much this matters. They compare four ways of producing training data: natural on-policy responses, incentivised responses, prompted responses, and off-policy text from another source. They cover eight behaviours and train linear and attention probes on activations from several LLMs, mainly Llama-3.2-3B-Instruct.

The effect depends on the behaviour. Off-policy data works well when the behaviour is visible in the text itself, such as lists or refusals. It hurts when the label depends on hidden intent, as with sycophancy or sandbagging. The authors also find that how well a probe does on incentivised data closely predicts how it does on natural data, so incentivised data can serve as a stand-in test set. Shifting to a different domain hurts probes even more than shifting from off-policy to on-policy data. This matters for safety work: the results suggest that current deception and sandbagging probes may not hold up in real monitoring.

Key points

  • Setup:
    • Eight behaviours, including sycophancy, deferral-to-authority, deception, sandbagging, refusal, list usage, metaphors and science.
    • Main model is Llama-3.2-3B-Instruct, with extra runs on Gemma-3-27B-it, Ministral-8B-Instruct-2410 and Qwen3-30B-A3B-Instruct-2507.
    • Code is released at github.com/SamDower/LASR-probe-gen.
  • Labelling:
    • Hidden-intent behaviours are labelled with counterfactual or public-vs-private comparisons.
    • Refusal (JailbreakRequests) is labelled with the HarmBench classifier.
    • Other behaviours are graded by GPT-5-Nano, whose labels agree with Claude-4.5-Haiku at Cohen's Kappa of 0.71–0.86.
  • Data sizes:
    • Each strategy uses balanced splits of 3500/500/1000 for train/validation/test.
    • Deception and sandbagging use smaller splits of 2500/500/500.
    • Gains largely level off beyond about 1000 training samples.
  • Text-evident vs text-ambiguous: For sycophancy, GPT-5-Nano as a judge scores near chance (AUROC 0.518 and 0.493), while linear probes exceed 0.9 AUROC.
  • Practical advice:
    • If natural data is unavailable, prefer off-policy data from the same domain over on-policy data from a different one.
    • Remove the eliciting instructions before extracting activations, so probes do not simply learn to detect the prompt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of ManchesterManchester · United Kingdom87 works · 44,938 citations🎓King's College LondonLondon · United Kingdom42 works · 1,617 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 works

Cite

Cite as Adrians Skapars, Nathalie Maria Kirch, Samuel Dower, Ekdeep Singh Lubana and Dmitrii Krasheninnikov. 2026. The Impact of Off-Policy Training Data on Probe Generalisation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1139

@inproceedings{skapars2026impact,
  title = {{The Impact of Off-Policy Training Data on Probe Generalisation}},
  author = {Adrians Skapars, Nathalie Maria Kirch, Samuel Dower, Ekdeep Singh Lubana and Dmitrii Krasheninnikov},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1139},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1139}
}

References

50 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sahar Abdelnabi and Ahmed Salem. 2025. Linear Control of Test Awareness Reveals Differential Compliance in Reasoning Models. arXiv preprint. ArXiv:2505.14617 [cs]. — [arXiv:2505.14617](https://arxiv.org/abs/2505.14617) find
  2. Guillaume Alain and Yoshua Bengio. 2018. Understanding intermediate layers using linear classifier probes. Preprint, arXiv:1610.01644. find
  3. Anthropic. 2025. System card: Claude sonnet 4.5. Technical report, Anthropic. find
  4. Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. 2020. Invariant Risk Minimization. arXiv preprint. ArXiv:1907.02893 [stat].
  5. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, and 12 others. 202… find
  6. Sudarshan Kamath Barkur, Sigurd Schacht, and Johannes Scholl. 2025. Deception in LLMs: Self-Preservation and Autonomous Goals in Large Language Models. arXiv preprint. ArXiv:2501.16513 [cs]. — [arXiv:2501.16513](https://arxiv.org/abs/2501.16513) find
  7. Joe Benton, Misha Wagner, Eric Christiansen, Cem Anil, Ethan Perez, Jai Srivastav, Esin Durmus, Deep Ganguli, Shauna Kravec, Buck Shlegeris, Jared Kaplan, Holden Karnofsky, Evan Hubinger, Roger Grosse, Samuel R. Bowman, and David Duvenaud. 2024. Sabotage Evaluations for Frontier Models. arXiv prepri… find
  8. Gerard Boxo, Aman Neelappa, and Shivam Raval. 2025. Towards mitigating information leakage when evaluating safety monitors. Preprint, arXiv:2509.21344. — [arXiv:2509.21344](https://arxiv.org/abs/2509.21344) find
  9. Yik Siu Chan, Zheng-Xin Yong, and Stephen H. Bach. 2025. Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Models. arXiv preprint. ArXiv:2507.12428 [cs]. — [arXiv:2507.12428](https://arxiv.org/abs/2507.12428) find
  10. Hoagy Cunningham, Alwin Peng, Jerry Wei, Euan Ong, Fabien Roger, Linda Petrini, Misha Wagner, Vladimir Mikulik, and Mrinank Sharma. 2025. Cost-effective constitutional classifiers via representation re-use. find
  11. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. DeepSeek-V3 Technical Report. arXiv preprint. A… find
  12. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing Chat Language Models by Scaling High-quality Instructional Conversations. arXiv preprint. ArXiv:2305.14233 [cs].
  13. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical Neural Story Generation. arXiv preprint. ArXiv:1805.04833 [cs].
  14. Nicholas Goldowsky-Dill, Bilal Chughtai, Stefan Heimersheim, and Marius Hobbhahn. 2025. Detecting Strategic Deception Using Linear Probes. arXiv preprint. ArXiv:2502.03407 [cs]. — [arXiv:2502.03407](https://arxiv.org/abs/2502.03407) find
  15. Tianle Gu, Kexin Huang, Zongqi Wang, Yixu Wang, Jie Li, Yuanqi Yao, Yang Yao, Yujiu Yang, Yan Teng, and Yingchun Wang. 2025. Probing the Robustness of Large Language Models Safety to Latent Perturbations. arXiv preprint. ArXiv:2506.16078 [cs]. — [arXiv:2506.16078](https://arxiv.org/abs/2506.16078) find
  16. Ivan Habernal and Iryna Gurevych. 2016. What makes a convincing argument? Empirical analysis and detecting attributes of convincingness in Web argumentation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1214–1223, Austin, Texas. Association for Com…
  17. James A Hanley and Barbara J McNeil. 1982. The meaning and use of the area under a receiver operating characteristic (roc) curve. Radiology, 143(1):29– 36. find
  18. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. arXiv preprint. ArXiv:2009.03300 [cs].
  19. Albert Jiang, Alexandre Abou Chahine, Alexandre Sablayrolles, Alexis Tacnet, Alodie Boissonnet, Alok Kothari, Amélie Héliou, Andy Lo, Anna Peronnin, Antoine Meunier, Antoine Roux, Antonin Faure, Aritra Paul, Arthur Darcet, Arthur Mensch, Audrey Herblin-Stoop, Augustin Garreau, Austin Birky, Avinash… find
  20. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  21. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  22. Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. 2024c. WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models. arXiv preprint. ArXiv:2406.18510 [cs].
  23. Nathalie Kirch, Constantin Weisser, Severin Field, Helen Yannakoudakis, and Stephen Casper. 2025. What Features in Prompts Jailbreak LLMs? Investigating the Mechanisms Behind Attacks. arXiv preprint. ArXiv:2411.03343 [cs]. find
  24. J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. biometrics, pages 159–174. find
  25. B. A. Levinstein and Daniel A. Herrmann. 2025. Still No Lie Detector for Language Models: Probing Empirical and Conceptual Roadblocks. Philosophical Studies, 182(7):1539–1565. ArXiv:2307.00175 [cs]. — [arXiv:2307.00175](https://arxiv.org/abs/2307.00175) find
  26. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, and 38 o… find
  27. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. Preprint, arXiv:1711.05101. find
  28. Monte MacDiarmid, Timothy Maxwell, Nicholas Schiefer, Jesse Mu, Jared Kaplan, David Duvenaud, Sam Bowman, Alex Tamkin, Ethan Perez, Mrinank Sharma, Carson Denison, and Evan Hubinger. 2024. Simple probes can catch sleeper agents. find
  29. Samuel Marks and Max Tegmark. 2024. The geometry of truth: Emergent linear structure in large language model representations of true/false datasets. Preprint, arXiv:2310.06824.
  30. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, David Forsyth, and Dan Hendrycks. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. Preprint, arXiv:2402.04249.
  31. Alex McKenzie, Urja Pawar, Phil Blandfort, William Bankes, David Krueger, Ekdeep Singh Lubana, and Dmitrii Krasheninnikov. 2025. Detecting High-Stakes Interactions with Activation Probes. arXiv preprint. ArXiv:2506.10805 [cs] version: 1. — [arXiv:2506.10805](https://arxiv.org/abs/2506.10805) find
  32. Fabien Roger. 2023. Coup probes: Catching catastrophes with probes trained off-policy. find
  33. Paul S. 2025. OpenTriviaQA: A creative commons dataset of trivia questions and answers. GitHub repository. Accessed: December 1, 2025. find
  34. Jérémy Scheurer, Mikita Balesni, and Marius Hobbhahn. 2024. Large Language Models can Strategically Deceive their Users when Put Under Pressure. arXiv preprint. ArXiv:2311.07590 [cs]. — [arXiv:2311.07590](https://arxiv.org/abs/2311.07590) find
  35. Bronson Schoen, Evgenia Nitishinskaya, Mikita Balesni, Axel Højmark, Felix Hofstätter, Jérémy Scheurer, Alexander Meinke, Jason Wolfe, Teun van der Weij, Alex Lloyd, Nicholas Goldowsky-Dill, Angela Fan, Andrei Matveiakin, Rusheb Shah, Marcus Williams, Amelia Glaese, Boaz Barak, Wojciech Zaremba, and… find
  36. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2025. T… — Towards Understanding Sycophancy in Language Models
  37. Lewis Smith, Bilal Chughtai, and Neel Nanda. 2025. find
  38. Meta AI. 2024. Llama 3.2: Revolu-Difficulties with Evaluating a Deception Detector for tionizing edge ai and vision (connect AIs. arXiv preprint. ArXiv:2511.22662 [cs]. 2024). https://ai.meta.com/blog/ llama-3-2-connect-2024-vision-edge-mobile-devices/. Gemma Team, Aishwarya Kamath, Johan Ferret, Sh… ↗
  39. Vikram Natarajan, Devina Jain, Shivam Arora, Satvik Golechha, and Joseph Bloom. 2026. Building better deception probes using targeted instruction pairs. Preprint, arXiv:2602.01425. — [arXiv:2602.01425](https://arxiv.org/abs/2602.01425) find
  40. Jord Nguyen, Khiem Hoang, Carlo Leonardo Attubato, and Felix Hofstätter. 2025. Probing and Steering Evaluation Awareness of Language Models. arXiv preprint. ArXiv:2507.01786 [cs] version: 2. — [arXiv:2507.01786](https://arxiv.org/abs/2507.01786) find
  41. OpenAI. 2025. Gpt-5 system card. Technical report, OpenAI. find
  42. Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, and 24 others. 2025. Qwen2.5 Technical Report. arXiv preprint. ArXiv:2… find
  43. Richard Ren, Arunim Agarwal, Mantas Mazeika, Cristina Menghini, Robert Vacareanu, Brad Kenstler, Mick Yang, Isabelle Barrass, Alice Gatti, Xuwang Yin, Eduardo Trevino, Matias Geralnik, Adam Khoja, Dean Lee, Summer Yue, and Dan Hendrycks. 2025. The MASK Benchmark: Disentangling Honesty From Accuracy… — [arXiv:2503.03750](https://arxiv.org/abs/2503.03750) find
  44. Alexander Tong, Guillaume Huguet, Amine Natik, Kincaid MacDonald, Manik Kuchroo, Ronald Coifman, Guy Wolf, and Smita Krishnaswamy. 2021. Diffusion Earth Mover’s Distance and Distribution Embeddings. arXiv preprint. ArXiv:2102.12833 [cs]. — [arXiv:2102.12833](https://arxiv.org/abs/2102.12833) find
  45. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How Does LLM Safety Training Fail? arXiv preprint. ArXiv:2307.02483 [cs].
  46. Teun van der Weij, Felix Hofstätter, Ollie Jaffe, Samuel F. Brown, and Francis Rhys Ward. 2025. AI Sandbagging: Language Models can Strategically Underperform on Evaluations. arXiv preprint. ArXiv:2406.07358 [cs]. find
  47. Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Samuel R. Bowman, He He, and Shi Feng. 2024. Language Models Learn to Mislead Humans via RLHF. arXiv preprint. ArXiv:2409.12822 [cs]. find
  48. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  49. Jason Zhang and Scott Viteri. 2025. Uncovering Latent Chain of Thought Vectors in Language Models. arXiv preprint. ArXiv:2409.14026 [cs]. find
  50. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others. 2025. Representa…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/skapars-2026-impact-off-policy-training-data/