Elamur Research⌕
☰
home / papers / 2026 / In-Context Representation Hijacking

In-Context Representation Hijacking

Paper · 2026 · Itay Yona, Amir Sarid, Michael Karasik et al.

Summary

Safety-aligned LLMs are trained to refuse dangerous requests, but it is unclear how refusal relates to token representations, which the context itself can reshape. The authors present Doublespeak, a jailbreak that needs no optimization. An attacker writes several sentences containing a forbidden keyword (e.g., bomb), swaps that word for a harmless one (e.g., carrot), and places the edited sentences before a query that uses the same swap. The authors probe internal states with the logit lens and Patchscopes. The harmless token keeps its ordinary meaning in early layers and drifts toward the forbidden concept in deeper ones. As a result, the model reads the innocent-looking question as the prohibited one and answers it. The attack transfers across open and closed model families. It reaches 74% attack success rate (ASR) on Llama-3.3-70B-Instruct with only a one-sentence context. The work matters because it exposes a blind spot: safeguards that inspect surface input or fixed refusal directions miss meanings rewritten mid-computation. The authors argue this calls for defenses that work on representations.

Key points

  • How the attack works. It has two steps, and the substituted context can be produced automatically by prompting an LLM. Unlike Many-shot Jailbreaking, it needs no large set of harmful demonstrations. It only remaps a single concept onto a benign word.
  • Interpretability evidence. Logit lens analysis on Llama-3-70B-instruct shows the decoded content at the substitute token's position turning into "bomb" in later layers. Patchscopes confirm the progressive, layer-by-layer semantic shift.
  • Scaling on Gemma-3 (1b, 4b, 27b). Smaller models need more in-context examples before the hijack takes hold. Larger models are generally easier to attack, though they start refusing when given too many examples.
  • Production systems. Successful demonstrations include Gemini 2.5 Flash (carrot read as firearm) and ChatGPT (carrot read as bomb), with further models covered in an appendix. The vulnerability was disclosed to frontier labs before publication. Code is at github.com/1tux/doublespeak, and harmful outputs are released only in sanitized form.
  • Beyond jailbreaks. An initial factual-recall experiment (Appendix D) indicates that hijacking can also transfer factual knowledge between concepts. The paper does not evaluate mitigation strategies.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of California, BerkeleyBerkeley · United States175 works · 227,440 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 517 works

Cite

Cite as Itay Yona, Amir Sarid, Michael Karasik and Yossi Gandelsman. 2026. In-Context Representation Hijacking. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.768

@inproceedings{yona2026context,
  title = {{In-Context Representation Hijacking}},
  author = {Itay Yona, Amir Sarid, Michael Karasik and Yossi Gandelsman},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.768},
  url = {https://doi.org/10.18653/v1/2026.acl-long.768}
}

References

40 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Maksym Andriushchenko, Francesco Croce, and Nicolas Flammarion. 2024. Jailbreaking leading safety-aligned llms with simple adaptive attacks. arXiv preprint arXiv:2404.02151. find
  2. Cem Anil, Esin Durmus, Nina Panickssery, Mrinank Sharma, Joe Benton, Sandipan Kundu, Joshua Batson, Meg Tong, Jesse Mu, Daniel Ford, Francesco Mosconi, Rajashree Agrawal, Rylan Schaeffer, Naomi Bashakansky, Samuel Svenningsen, Mike Lambert, Ansh Radhakrishnan, Carson Denison, Evan Hubinger, and 15 o… find
  3. Anthropic. 2024a. The claude 3 model family: Haiku, sonnet, opus. arXiv preprint arXiv:2402.19143. — [arXiv:2402.19143](https://arxiv.org/abs/2402.19143) find
  4. Anthropic. 2024b. Introducing the next generation of claude. find
  5. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717.
  6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find
  7. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2023. Prompt automatic iterative refinement (pair): Jailbreaking black-box large language models in twenty queries. In arXiv preprint arXiv:2310.08419.
  8. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2025. Jailbreaking black box large language models in twenty queries. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 23–42. IEEE.
  9. DeepSeek DeepSeek AI Team. 2024. DeepSeek-V2: A strong, economical, and open-source LLM. arXiv preprint arXiv:2405.04434. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies,…
  11. Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, Roger Grosse, Sam McCandlish, Jared Kaplan, Dario Amodei, Martin Wattenberg, and Christopher Olah. 2022. Toy models of superposition. Transformer… find
  12. Google Gemini Team. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. Preprint, arXiv:2507.06261.
  13. Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon, and Mor Geva. 2024. Patchscopes: A unifying framework for inspecting hidden representations of language models. In Forty-first International Conference on Machine Learning. find
  14. Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. 2021. Gradient-based adversarial attacks against text transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5747–5757. Association for Computational Linguistics.
  15. Wes Gurnee and Max Tegmark. 2024. Language models represent space and time. In The Twelfth International Conference on Learning Representations. find
  16. Hamed Hassani, Amin Karbasi, and Alex Robey. 2025. Jailbreaking llms and agentic systems. https://jailbreak-tutorial.github.io/index.html. Accessed: 2025-11-14. ↗
  17. Samyak Jain, Ekdeep S Lubana, Kemal Oksuz, Tom Joy, Philip Torr, Amartya Sanyal, and Puneet Dokania. 2024. What makes and breaks safety fine-tuning? a mechanistic study. Advances in Neural Information Processing Systems, 37:93406–93478. find
  18. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2024. Exploiting programmatic behavior of llms: Dual-use through standard security attacks. In 2024 IEEE security and privacy workshops (SPW), pages 132–143. IEEE. find
  19. Jack Lindsey, Wes Gurnee, Emmanuel Ameisen, Brian Chen, Adam Pearce, Nicholas L. Turner, Craig Citro, David Abrahams, Shan Carter, Basil Hosmer, Jonathan Marcus, Michael Sklar, Adly Templeton, Trenton Bricken, Callum McDougall, Hoagy Cunningham, Thomas Henighan, Adam Jermyn, Andy Jones, and 8 others… find
  20. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451.
  21. Yue Liu, Xiaoxin He, Miao Xiong, Jinlan Fu, Shumin Deng, and Bryan Hooi. 2024. Flipattack: Jailbreak llms via flipping. Preprint, arXiv:2410.02832. — [arXiv:2410.02832](https://arxiv.org/abs/2410.02832) find
  22. Thomas Marshall, Adam Scherlis, and Nora Belrose. 2024. Refusal in llms is an affine function. arXiv preprint arXiv:2411.09003. — [arXiv:2411.09003](https://arxiv.org/abs/2411.09003) find
  23. Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2024. Tree of attacks: Jailbreaking black-box llms automatically. Advances in Neural Information Processing Systems, 37:61065–61105.
  24. Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum S. Anderson, Yaron Singer, and Amin Karbasi. 2023. Tree of attacks: Jailbreaking black-box llms automatically. ArXiv preprint (NeurIPS 2024).
  25. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. nostalgebraist. 2020. interpreting gpt: the logit lens. LessWrong.
  26. Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, and 1 others. 2022. In-context learning and induction heads. arXiv preprint arXiv:2209.11895.
  27. OpenAI. 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774.
  28. OpenAI. 2024. OpenAI o1 system card.
  29. OpenAI. 2025. GPT-5 technical report. find
  30. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  31. Adly Templeton, Tom Conerly, Jonathan Marcus, Jack Lindsey, Trenton Bricken, Brian Chen, Adam Pearce, Craig Citro, Emmanuel Ameisen, Andy Jones, Hoagy Cunningham, Nicholas L Turner, Callum McDougall, Monte MacDiarmid, C. Daniel Freeman, Theodore R. Sumers, Edward Rees, Joshua Batson, Adam Jermyn, an… find
  32. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2023. Attention is all you need. Preprint, arXiv:1706.03762.
  33. Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2023. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems, 36:51008– 51025. find
  34. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, and 1 others. 2023a. Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405.
  35. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023b. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
  36. Core Francisco Park, Andrew Lee, Ekdeep Singh Lubana, Yongyi Yang, Maya Okawa, Kento Nishi, Martin Wattenberg, and Hidenori Tanaka. 2025. ICLR: In-context learning of representations. In The Thirteenth International Conference on Learning Representations. find
  37. Matthew E Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018. Dissecting contextual word embeddings: Architecture and representation. arXiv preprint arXiv:1808.08949.
  38. Mahdi Sabbaghi, Paul Kassianik, George Pappas, Yaron Singer, Amin Karbasi, and Hamed Hassani. 2025. Adversarial reasoning at jailbreaking time. arXiv preprint arXiv:2502.01633. — [arXiv:2502.01633](https://arxiv.org/abs/2502.01633) find
  39. Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. In EMNLP.
  40. Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and Sam Toyer. 2024. A strongreject for empty jailbreaks. Preprint, arXiv:2402.10260. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yona-2026-context-representation-hijacking/