Elamur Research⌕
☰
home / papers / 2026 / Label Words as Local Task Vectors in In-Context Learning

Label Words as Local Task Vectors in In-Context Learning

Paper · 2026 · Bowen Zheng, Ming Ma, Zhongqiao Lin et al.

Summary

This paper asks whether the "task vector" idea, a single averaged hidden state at the query's final position that is assumed to encode the whole in-context task, actually holds across task types. The authors compare knowledge tasks, such as Country→Capital, with rule-induction tasks, where a categorization rule can only be inferred by combining several demonstrations. Examples are labeling random strings 0/1 by a length threshold and 2-D boundary problems. They use saliency scores, lesions of answer tokens, demixed PCA (dPCA) and activation patching into dummy-input prompts. They find that each demonstration first writes task information into the hidden state at its own answer (label) token, which forms a local task vector. In knowledge tasks these local vectors become aligned in later layers and merge into a usable global vector. In categorization tasks they stay heterogeneous and complementary, so averaging them loses information. Patching the local vectors restores few-shot accuracy more reliably than patching a global vector. This supports a distributed, aggregation-based account of how ICL works.

Key points

  • Representative tasks:
    • Country→capital (4 demonstrations by default).
    • String-length categorization: strings of 1–10 characters, answer 0 if shorter than 6 characters, otherwise 1 (8 demonstrations by default).
    • Additional tasks include Simple/Complex String, Digit, 2-D (8 and 16 demos), Antonyms, Language, Profession and Religion.
  • Saliency and lesion evidence (layer 14): Saliency peaks between demonstration answer tokens and the query's final "is" token.
    • In the knowledge task, erasing answer tokens hurts performance only when all of them are removed.
    • In categorization, accuracy drops gradually as more answer tokens are erased.
  • dPCA analysis: dPCA is used to project out the subspace of answer-position states that encodes string length. The authors use this to argue that local vectors carry an abstract form of the rule.
  • Patching across models: Patching local vectors beats global-vector patching in categorization tasks on LLaMA-7B (Table 2, averaged over 5 trials). The pattern also holds for other model families, for example Vicuna-7B on the Simple String and Digit tasks.
  • Multi-token generation on LLaMA3-8B: Local patching outperforms global patching on translation (0.81 vs 0.79 exact match) and arithmetic (0.61 vs 0.50). The authors note that neither method succeeds for every model and task.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Chinese University of Hong KongHong Kong · China159 works · 16,154 citations🏛️Chinese Academy of SciencesBeijing · China246 works · 24,952 citations🎓University of Chinese Academy of SciencesBeijing · China188 works · 8,744 citations🎓Chinese University of Hong Kong, ShenzhenShenzhen · China56 works · 3,212 citations🎓Center for Excellence in Brain Science and Intelligence TechnologyShanghai · China2 works · 245 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Bowen Zheng, Ming Ma, Zhongqiao Lin and Tianming Yang. 2026. Label Words as Local Task Vectors in In-Context Learning. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.314

@inproceedings{zheng2026label,
  title = {{Label Words as Local Task Vectors in In-Context Learning}},
  author = {Bowen Zheng, Ming Ma, Zhongqiao Lin and Tianming Yang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.314},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.314}
}

References

34 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Kwangjun Ahn, Xiang Cheng, Hadi Daneshmand, and Suvrit Sra. 2024. Transformers learn to implement preconditioned gradient descent for in-context learning. Advances in Neural Information Processing Systems, 36. find
  2. Kabir Ahuja, Madhur Panwar, and Navin Goyal. 2023. In-context learning through the bayesian prism. arXiv preprint arXiv:2306.04891. — [arXiv:2306.04891](https://arxiv.org/abs/2306.04891) find
  3. Uchida, and Christian K Machens. 2016. Demixed principal component analysis of neural population data. elife, 5:e10989. find
  4. Ekin Akyürek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. 2023. What learning algorithm is in-context learning? investigations with linear models. Preprint, arxiv:2211.15661.
  5. Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691.
  6. David GT Barrett, Ari S Morcos, and Jakob H Macke. 2019. Analyzing biological and artificial neural networks: challenges with opportunities for synergy? Current opinion in neurobiology, 55:55–64. find
  7. Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. 2023. Pythia: A suite for analyzing large language models across training and scaling. In International Conferenc… find
  8. Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120. find
  9. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, and Jared Kaplan. 2020. Language models are few-shot learners. NeurIPS.
  10. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys. org (accessed 14 April 2023), 2(3):6. ↗
  11. Joy Crosbie and Ekaterina Shutova. 2025. Induction heads as an essential mechanism for pattern matching in in-context learning. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 5034–5096. find
  12. Damai Dai, Yutao Sun, Li Dong, Yaru Hao, and Shuming Ma. 2023. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. Preprint, arxiv:2212.10559. — [arXiv:2212.10559](https://arxiv.org/abs/2212.10559) find
  13. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, and Zhiyong Wu. 2023. A survey on in-context learning. Preprint, arxiv:2301.00234.
  14. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  15. Dongfang Li, Zhenyu Liu, Xinshuo Hu, Zetian Sun, Baotian Hu, and Min Zhang. 2024. In-context learning state vector with inner and momentum optimization. Advances in Neural Information Processing Systems, 37:7797–7820. find
  16. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35.
  17. Sheng Liu, Haotian Ye, Lei Xing, and James Y. Zou. 2024. In-context vectors: Making in context learning more effective and controllable through latent space steering. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, page…
  18. Paul Michel, Omer Levy, and Graham Neubig. 2019. Are sixteen heads really better than one? Advances in neural information processing systems, 32.
  19. Yael Niv. 2009. Reinforcement learning in the brain. Journal of Mathematical Psychology, 53(3):139–154. find
  20. A Yang Qwen, Baosong Yang, B Zhang, B Hui, B Zheng, B Yu, Chengpeng Li, D Liu, F Huang, H Wei, et al. 2024. Qwen2. 5 technical report. arXiv preprint. find
  21. Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. 2019. A deep learning framework for neuroscience. Nature neuroscience, 22(11):1761–1770. find
  22. Baturay Saglam, Xinyang Hu, Zhuoran Yang, Dionysis Kalogerias, and Amin Karbasi. 2025. Learning task representations from in-context learning. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6634–6663. find
  23. Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. 2013. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034.
  24. Roee Hendel, Mor Geva, and Amir Globerson. 2023. In-context learning creates task vectors. Preprint, arxiv:2310.15916.
  25. Pavel Tikhonov, Ivan Oseledets, and Elena Tutubalina. 2025. One task vector is not enough: A large-scale study for in-context learning. arXiv preprint arXiv:2505.23911.
  26. Dmitry Kobak, Wieland Brendel, Christos Constantini-dis, Claudia E Feierstein, Adam Kepecs, Zachary F Mainen, Xue-Lian Qi, Ranulfo Romo, Naoshige Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal A… — LLaMA: Open and Efficient Foundation Language Models
  27. Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. 2023. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151–35174. PMLR.
  28. Lean Wang, Lei Li, Damai Dai, Deli Chen, and Hao Zhou. 2023a. Label words are anchors: An information flow perspective for understanding in-context learning. Preprint, arxiv:2305.14160.
  29. Xinyi Wang, Wanrong Zhu, Michael Saxon, Mark Steyvers, and William Yang Wang. 2023b. Large language models are implicitly topic models: Explaining and finding good demonstrations for in-context learning. In Workshop on Efficient Systems for Foundation Models@ ICML2023. find
  30. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837.
  31. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An explanation of in-context learning as implicit bayesian inference. Preprint, arxiv:2111.02080.
  32. Daniel LK Yamins and James J DiCarlo. 2016. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3):356–365. find
  33. Safoora Yousefi, Leo Betthauser, Hosein Hasanbeig, Raphaël Millière, and Ida Momennejad. 2023. Decoding in-context learning: Neuroscience-inspired analysis of representations in large language models. arXiv preprint arXiv:2310.00313. — [arXiv:2310.00313](https://arxiv.org/abs/2310.00313) find
  34. Zhewei Zhang, Zhenbo Cheng, Zhongqiao Lin, Chechang Nie, and Tianming Yang. 2018. A neural network model for the orbitofrontal cortex and task space acquisition during reinforcement learning. PLOS Computational Biology, 14(1):e1005925. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zheng-2026-label-words-as-local-task/