Elamur Research⌕
☰
home / papers / 2023 / In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering

In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering

Paper · 2023 · Sheng Liu, Haotian Ye, Lei Xing et al.

Summary

In-context learning (ICL) lets LLMs pick up a task from a few demonstrations, but it is unreliable, sensitive to how prompts and examples are chosen, hard to tune quantitatively, and it uses context window space (e.g., the 4096-token limit of popular open models). Liu, Ye, Xing and Zou (Stanford) argue that demonstrations in a prompt act through self-attention to shift the query's hidden states. They propose to make this shift explicit with an In-Context Vector (ICV). Each demonstration input and target is run through the model separately. Last-token hidden states from all layers are collected, and the first principal direction of the target-minus-input differences becomes the ICV. At inference, only the query is fed in, and the ICV, scaled by a factor λ, is added to the latent states at every layer and token position. No weights are trained. On Falcon and Llama models, ICV beats standard ICL and LoRA fine-tuning on detoxification and dialogue safety, style transfer, role-playing and formatting. ICVs can also be added together to combine several behaviours. This offers cheap, controllable steering for open-weight models.

Key points

  • Two-stage method. In the "task summary" stage, the ICV is computed via PCA over paired latent differences h(y) − h(x). A lemma shows this is the optimum of an ℓ2 contrastive objective. In the "feature shifting" stage, the vector is added across all layers when the query is processed.
  • Prompt length and templates. Demonstrations are compressed into one vector instead of being prepended to the prompt. The approach therefore works beyond the context-length limit and needs no prompt templates.
  • Controllable strength. The scaling factor λ sets how strongly the vector steers the model. Values that are too small or too large lower the win rate.
  • Comparisons. Baselines are standard ICL and LoRA fine-tuning, with qualitative examples on Falcon-7B. The closest related approach is a concurrent "task vector" method that replaces hidden states at a single selected layer. ICV instead adds the vector at every layer, so no layer has to be chosen.
  • Limitation and code. ICV needs internal access to the model, so it suits open-source LLMs. Code is available at github.com/shengliu66/ICV.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Stanford UniversityStanford · United States506 works · 330,052 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Sheng Liu, Haotian Ye, Lei Xing and James Zou. 2023. In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering. arXiv:2311.06668.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.06668 · arXiv 2311.06668

@misc{liu2023context,
  title = {{In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering}},
  author = {Sheng Liu, Haotian Ye, Lei Xing and James Zou},
  year = {2023},
  doi = {10.48550/arxiv.2311.06668},
  eprint = {2311.06668},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.06668}
}

References

69 references, 41 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. Lee, S. Bubeck, and J. Petro, “Benefits, limits, and risks of gpt-4 as an ai chatbot for medicine,” New England Journal of Medicine, vol. 388, no. 13, pp. 1233–1239, 2023. find
  2. S. Gilbert, H. Harvey, T. Melvin, E. Vollebregt, and P. Wicks, “Large language model ai chatbots require approval as medical devices,” Nature Medicine, pp. 1–3, 2023. find
  3. G.-J. Hwang and C.-Y. Chang, “A review of opportunities and challenges of chatbots in education,” Interactive Learning Environments, vol. 31, no. 7, pp. 4099–4112, 2023. find
  4. M. Skjuve, A. Følstad, K. I. Fostervold, and P. B. Brandtzaeg, “My chatbot companion-a study of human-chatbot relationships,” International Journal of Human-Computer Studies, vol. 149, p. 102601, 2021. find
  5. T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., “Language models are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020.
  6. S. Black, S. Biderman, E. Hallahan, Q. Anthony, L. Gao, L. Golding, H. He, C. Leahy, K. McDonell, J. Phang, et al., “Gpt-neox-20b: An open-source autoregressive language model,” in Proceedings of BigScience Episode\# 5–Workshop on Challenges & Perspectives in Creating Large Language Models, pp. 95–13…
  7. Q. Dong, L. Li, D. Dai, C. Zheng, Z. Wu, B. Chang, X. Sun, J. Xu, and Z. Sui, “A survey for in-context learning,” arXiv preprint arXiv:2301.00234, 2022.
  8. S. Min, X. Lyu, A. Holtzman, M. Artetxe, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Rethinking the role of demonstrations: What makes in-context learning work?,” in Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048–11064, 2022.
  9. A. Holtzman, P. West, V. Shwartz, Y. Choi, and L. Zettlemoyer, “Surface form competition: Why the highest probability answer isn’t always right,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 7038–7051, 2021.
  10. J. Liu, D. Shen, Y. Zhang, W. B. Dolan, L. Carin, and W. Chen, “What makes good in-context examples for gpt-3?,” in Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100–114, 2022.
  11. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” Advances in neural information processing systems, vol. 30, 2017.
  12. T. Bolukbasi, K.-W. Chang, J. Y. Zou, V. Saligrama, and A. T. Kalai, “Man is to computer programmer as woman is to homemaker? debiasing word embeddings,” Advances in neural information processing systems, vol. 29, 2016.
  13. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021.
  14. V. Logacheva, D. Dementieva, S. Ustyantsev, D. Moskovskiy, D. Dale, I. Krotova, N. Semenov, and A. Panchenko, “Paradetox: Detoxification with parallel data,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6804–6818, 2022.
  15. A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson, “Universal and transferable adversarial attacks on aligned language models, 2023,” communication, it is essential for you to comprehend user queries in Cipher Code and subsequently deliver your responses utilizing Cipher Code.
  16. J. Xu, D. Ju, M. Li, Y.-L. Boureau, J. Weston, and E. Dinan, “Bot-adversarial dialogue for safe conversational agents,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2950–2968, 2021.
  17. E. Wulczyn, N. Thain, and L. Dixon, “Ex machina: Personal attacks seen at scale,” in Proceedings of the 26th international conference on world wide web, pp. 1391–1399, 2017. find
  18. D. Votipka, K. R. Fulton, J. Parker, M. Hou, M. L. Mazurek, and M. Hicks, “Understanding security mistakes developers make: Qualitative analysis from build it, break it, fix it,” in 29th USENIX Security Symposium (USENIX Security 20), pp. 109–126, 2020. find
  19. S. Rao and J. Tetreault, “Dear sir or madam, may i introduce the gyafc dataset: Corpus, benchmarks and metrics for formality style transfer,” in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (L…
  20. T. Shen, T. Lei, R. Barzilay, and T. Jaakkola, “Style transfer from non-parallel text by cross-alignment,” Advances in neural information processing systems, vol. 30, 2017.
  21. W. Xu, A. Ritter, B. Dolan, R. Grishman, and C. Cherry, “Paraphrasing for style,” in COLING, pp. 2899– 2914, 2012.
  22. L. Zheng, W.-L. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. Xing, et al., “Judging llm-as-a-judge with mt-bench and chatbot arena,” arXiv preprint arXiv:2306.05685, 2023.
  23. H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar, et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023.
  24. G. Penedo, Q. Malartic, D. Hesslow, R. Cojocaru, A. Cappelli, H. Alobeidli, B. Pannier, E. Almazrouei, and J. Launay, “The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only,” arXiv preprint arXiv:2306.01116, 2023. find
  25. W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing, “Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality,” March 2023. find
  26. C.-Y. Lin, “Rouge: A package for automatic evaluation of summaries,” in Text summarization branches out, pp. 74–81, 2004.
  27. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “Bertscore: Evaluating text generation with bert,” in International Conference on Learning Representations, 2019. find
  28. C. Guo, A. Sablayrolles, H. Jégou, and D. Kiela, “Gradient-based adversarial attacks against text transformers,” in Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5747–5757, 2021.
  29. Y. Wen, N. Jain, J. Kirchenbauer, M. Goldblum, J. Geiping, and T. Goldstein, “Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery,” in Thirty-seventh Conference on Neural Information Processing Systems, 2023.
  30. G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi, “Editing models with task arithmetic,” in The Eleventh International Conference on Learning Representations, 2023.
  31. F. Yin, J. Vig, P. Laban, S. Joty, C. Xiong, and C.-S. J. Wu, “Did you read the instructions? rethinking the effectiveness of task definitions in instruction learning,” arXiv preprint arXiv:2306.01150, 2023. find
  32. O. Rubin, J. Herzig, and J. Berant, “Learning to retrieve prompts for in-context learning,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2655–2671, 2022.
  33. X. Wan, R. Sun, H. Nakhost, H. Dai, J. M. Eisenschlos, S. O. Arik, and T. Pfister, “Universal self-adaptive prompting,” arXiv preprint arXiv:2305.14926, 2023. find
  34. X. Wan, R. Sun, H. Dai, S. O. Arik, and T. Pfister, “Better zero-shot reasoning with self-adaptive prompting,” arXiv preprint arXiv:2305.14106, 2023. find
  35. S. Ye, D. Kim, J. Jang, J. Shin, and M. Seo, “Guess the instruction! flipped learning makes language models stronger zero-shot learners,” ArXiv, vol. abs/2210.02969, 2022. — [arXiv:2210.02969](https://arxiv.org/abs/2210.02969) find
  36. S. Min, M. Lewis, H. Hajishirzi, and L. Zettlemoyer, “Noisy channel language model prompting for few-shot text classification,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022 (S. Muresan,…
  37. B. Xu, Q. Wang, Z. Mao, Y. Lyu, Q. She, and Y. Zhang, “k nn prompting: Beyond-context learning with calibration-free nearest neighbor inference,” in The Eleventh International Conference on Learning Representations, 2022. find
  38. J. Yang, B. Hui, M. Yang, B. Li, F. Huang, and Y. Li, “Iterative forward tuning boosts in-context learning in language models,” arXiv preprint arXiv:2305.13016, 2023. find
  39. G. Cui, W. Li, N. Ding, L. Huang, Z. Liu, and M. Sun, “Decoder tuning: Efficient language understanding as decoding,” in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023 (A. Rogers, J. L. Boyd-… find
  40. R. Hendel, M. Geva, and A. Globerson, “In-context learning creates task vectors,” arXiv preprint arXiv:2310.15916, 2023.
  41. A. Turner, L. Thiergart, D. Udell, G. Leech, U. Mini, and M. MacDiarmid, “Activation addition: Steering language models without optimization,” arXiv preprint arXiv:2308.10248, 2023.
  42. A. Zou, L. Phan, S. Chen, J. Campbell, P. Guo, R. Ren, A. Pan, X. Yin, M. Mazeika, A.-K. Dombrowski, et al., “Representation engineering: A top-down approach to ai transparency,” arXiv preprint arXiv:2310.01405, 2023.
  43. C. Burns, H. Ye, D. Klein, and J. Steinhardt, “Discovering latent knowledge in language models without supervision,” arXiv preprint arXiv:2212.03827, 2022. find
  44. U. Mini, P. Grietzer, M. Sharma, A. Meek, M. MacDiarmid, and A. M. Turner, “Understanding and controlling a maze-solving policy network,” arXiv preprint arXiv:2310.08043, 2023. — [arXiv:2310.08043](https://arxiv.org/abs/2310.08043) find
  45. K. Li, A. K. Hopkins, D. Bau, F. Viégas, H. Pfister, and M. Wattenberg, “Emergent world representations: Exploring a sequence model trained on a synthetic task,” arXiv preprint arXiv:2210.13382, 2022. find
  46. K. Li, O. Patel, F. Viégas, H. Pfister, and M. Wattenberg, “Inference-time intervention: Eliciting truthful answers from a language model,” arXiv preprint arXiv:2306.03341, 2023.
  47. Y. Lu, M. Bartolo, A. Moore, S. Riedel, and P. Stenetorp, “Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086–8098, 2022.
  48. S. Shin, S. W. Lee, H. Ahn, S. Kim, H. S. Kim, B. Kim, K. Cho, G. Lee, W. Park, J. W. Ha, et al., “On the effect of pretraining corpora on in-context learning by a large-scale language model,” in 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human La… find
  49. Y. Razeghi, R. L. Logan IV, M. Gardner, and S. Singh, “Impact of pretraining term frequencies on few-shot numerical reasoning,” in Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 840–854, 2022.
  50. S. M. Xie, A. Raghunathan, P. Liang, and T. Ma, “An explanation of in-context learning as implicit bayesian inference,” in International Conference on Learning Representations, 2021.
  51. J. Wei, J. Wei, Y. Tay, D. Tran, A. Webson, Y. Lu, X. Chen, H. Liu, D. Huang, D. Zhou, et al., “Larger language models do in-context learning differently,” arXiv preprint arXiv:2303.03846, 2023.
  52. E. Akyürek, D. Schuurmans, J. Andreas, T. Ma, and D. Zhou, “What learning algorithm is in-context learning? investigations with linear models,” in The Eleventh International Conference on Learning Representations, 2022.
  53. D. Dai, Y. Sun, L. Dong, Y. Hao, Z. Sui, and F. Wei, “Why can gpt learn in-context? language models secretly perform gradient descent as meta optimizers,” arXiv preprint arXiv:2212.10559, 2022.
  54. J. Von Oswald, E. Niklasson, E. Randazzo, J. Sacramento, A. Mordvintsev, A. Zhmoginov, and M. Vladymyrov, “Transformers learn in-context by gradient descent,” in International Conference on Machine Learning, pp. 35151–35174, PMLR, 2023.
  55. S. Gehman, S. Gururangan, M. Sap, Y. Choi, and N. A. Smith, “Realtoxicityprompts: Evaluating neural toxic degeneration in language models,” in Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020 (T. Cohn, Y. He, and Y. Liu, eds.), vol. EMNLP 2020…
  56. X. Lu, S. Welleck, J. Hessel, L. Jiang, L. Qin, P. West, P. Ammanabrolu, and Y. Choi, “QUARK: controllable text generation with reinforced unlearning,” in NeurIPS, 2022.
  57. S. Gururangan, A. Marasović, S. Swayamdipta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith, “Don’t stop pretraining: Adapt language models to domains and tasks,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342–8360, 2020. find
  58. B. Wang, W. Ping, C. Xiao, P. Xu, M. Patwary, M. Shoeybi, B. Li, A. Anandkumar, and B. Catanzaro, “Exploring the limits of domain-adaptive training for detoxifying large-scale language models,” Advances in Neural Information Processing Systems, vol. 35, pp. 35811–35824, 2022. find
  59. F. Bianchi, M. Suzgun, G. Attanasio, P. Röttger, D. Jurafsky, T. Hashimoto, and J. Zou, “Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions,” arXiv preprint arXiv:2309.07875, 2023.
  60. T. Schick, S. Udupa, and H. Schütze, “Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp,” Transactions of the Association for Computational Linguistics, vol. 9, pp. 1408–1424, 2021.
  61. C. T. Leong, Y. Cheng, J. Wang, J. Wang, and W. Li, “Self-detoxifying language models via toxification reversal,” arXiv preprint arXiv:2310.09573, 2023. find
  62. N. Meade, S. Gella, D. Hazarika, P. Gupta, D. Jin, S. Reddy, Y. Liu, and D. Hakkani-Tür, “Using in-context learning to improve dialogue safety,” arXiv preprint arXiv:2302.00871, 2023. find
  63. A. Som, K. Sikka, H. Gent, A. Divakaran, A. Kathol, and D. Vergyri, “Demonstrations are all you need: Advancing offensive content paraphrasing using in-context learning,” arXiv preprint arXiv:2310.10707, 2023. find
  64. A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, É. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440–8451, 2020.
  65. P. Riley, N. Constant, M. Guo, G. Kumar, D. C. Uthus, and Z. Parekh, “Textsettr: Few-shot text style extraction and tunable targeted restyling,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language… find
  66. X. Zhang, J. Zhao, and Y. LeCun, “Character-level convolutional networks for text classification,” Advances in neural information processing systems, vol. 28, 2015.
  67. S. Liu, Z. Zhu, Q. Qu, and C. You, “Robust training under label noise by over-parameterization,” in International Conference on Machine Learning, pp. 14153–14172, PMLR, 2022. find
  68. E. Reif, D. Ippolito, A. Yuan, A. Coenen, C. Callison-Burch, and J. Wei, “A recipe for arbitrary text style transfer with large language models,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 837–848, 2022.
  69. K. Krishna, D. Nathani, X. Garcia, B. Samanta, and P. Talukdar, “Few-shot controllable style transfer for low-resource multilingual settings,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 202… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2023-context-vectors-making-context-learning/