Elamur Research⌕
☰
home / papers / 2026 / One Task Vector is not Enough: A Large-Scale Study for In-Context Learning

One Task Vector is not Enough: A Large-Scale Study for In-Context Learning

Paper · 2026 · Pavel Tikhonov, Ivan Oseledets and Elena Tutubalina

Summary

Task vectors are hidden-state activations that may compress the demonstrations of an in-context learning (ICL) prompt into a single representation of the task. Earlier work tested this idea only on small, hand-built task sets. This paper builds a much larger benchmark, QuiteAFew, to check whether the idea holds at scale. The authors started from Alpaca instructions and used Qwen-2.5-72B to keep only those that could support many varied, checkable examples. They then generated about 30 input-output pairs for each of 3,096 tasks. Task vectors from Llama-3-8B and Qwen-3-8B were taken at the separator token and patched into zero-shot queries. An LLM judge scored the outputs for format and correctness.

Across task categories, performance rises and then falls with layer depth, peaking at an intermediate layer. How well a single vector works depends strongly on the kind of task. A case study on structured JSON output shows that composite tasks need several subtask-specific vectors placed along the output, not one global vector. This suggests that task knowledge in LLMs is distributed rather than packed into a single activation.

Key points

  • QuiteAFew: 3,096 few-shot tasks built from Alpaca, each with 30 input-output pairs. Qwen-3-235B-A22B generated 2,072 of the tasks and Qwen-2.5-72B generated 1,024. Instructions were filtered beforehand into GOOD, LIMITED or INVALID.
  • Evaluation protocol: an LLM judge gave each output a format score and a correctness score, both on a 0–10 scale, and a human validation study showed strong agreement with the judge. Each trial used 6 demonstrations, one "dummy" query from which the vector was extracted, and one test item. Every task was run 10 times.
  • Best layer: performance peaked around layer 15 of 32 in Llama-3-8B (relative depth ≈0.47) and around layer 24 of 36 in Qwen-3-8B (≈0.67). Both models showed an inverted-U curve.
  • Category effects: convert, classify and analyze tasks gained clearly at the best layer. Edit and describe tasks stayed roughly flat across all layers.
  • JSON case study: on 100 automobile descriptions, sub-task vectors taken at matching tokens raised the probability of the correct next token, for both attribute keys and values. They beat both plain generation and a single classic task vector.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Pavel Tikhonov, Ivan Oseledets and Elena Tutubalina. 2026. One Task Vector is not Enough: A Large-Scale Study for In-Context Learning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-srw.127

@inproceedings{tikhonov2026one,
  title = {{One Task Vector is not Enough: A Large-Scale Study for In-Context Learning}},
  author = {Pavel Tikhonov, Ivan Oseledets and Elena Tutubalina},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-srw.127},
  url = {https://doi.org/10.18653/v1/2026.acl-srw.127}
}

References

14 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  2. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.00234.
  3. Mor Geva, Jasmijn Bastings, Katja Filippova, and Amir Globerson. 2023. Dissecting recall of factual associations in auto-regressive language models. arXiv preprint arXiv:2304.14767. find
  4. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  5. Etai Hendel, Ori Lieber, Ido Dalmedigos, and Yoav Shoham. 2023. In-context learning creates task vectors. arXiv preprint arXiv:2310.15916.
  6. Dmitrii Kharlapenko, neverix, Neel Nanda, and Arthur Conmy. 2024. Extracting sae task features for in-context learning. https://www.alignmentforum.org/posts/5FGXmJ3wqgGRcbyH7/extracting-s ae-task-features-for-in-context-learning. Accessed: 2025-05-18. ↗
  7. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. find
  8. Guangliang Liu, Haitao Mao, Jiliang Tang, and Kristen Marie Johnson. 2024. Intrinsic self-correction for enhanced morality: An analysis of internal mechanisms and the superficial hypothesis. arXiv preprint arXiv:2407.15286. find
  9. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35.
  10. Ziyi Luo, Yujie Wang, Yifan Zhang, Jiacheng Chen, and Saining Liu. 2024. Task vectors are cross-modal. arXiv preprint arXiv:2410.22330. find
  11. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://gi thub.com/tatsu-lab/stanford alpaca. find
  12. Nelson Todd, Ori Lieber, Ido Dalmedigos, Etai Hendel, and Yoav Shoham. 2024. Function vectors in large language models. arXiv preprint arXiv:2310.15213. find
  13. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  14. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tikhonov-2026-one-task-vector-is-not/