Elamur Research⌕
☰
home / papers / 2024 / The Surprising Effectiveness of Test-Time Training for Few-Shot Learning

The Surprising Effectiveness of Test-Time Training for Few-Shot Learning

Paper · 2024 · Ekin Akyürek, Mehul Damani, Adam Zweiger et al.

Summary

Language models handle tasks similar to their training data well. However, on structurally novel problems they often fail, even when the prompt contains a few examples. The authors, from MIT, investigate test-time training (TTT). During inference, the model is temporarily fine-tuned through adapters, using data derived from the provided few-shot examples. The key idea is synthetic in-context tasks built with a leave-one-out scheme. Each example in turn plays the role of the "test" example, and the rest serve as demonstrations. For structured data, the set is expanded with invertible transformations: rotations, reflections, and color permutations. The final answer is chosen by voting in the spirit of self-consistency. The authors systematically compare data-generation methods, loss-function variants, and parameterizations: a separate adapter for each task or a shared one for all. On ARC, an 8B-parameter model reaches 53.0%. In an ensemble with program synthesis, the result rises to 61.9%, i.e., to the average human level. On BIG-Bench Hard, TTT clearly outperforms standard 10-shot prompting. The work shows the limits of in-context learning and supports the idea of adapting the model during inference.

Key points

  • ARC: TTT yields up to 6× higher accuracy than fine-tuned base models. On a subset of 80 validation tasks, the gain for the FT model is 27.5 pp (Pass@2). On the public validation set, the 8B model reaches 53.0%. In an ensemble with program synthesis from the BARC team, the result was 61.9%, and this ensemble was submitted as a joint official entry.
  • BBH (10-shot): accuracy rises from 50.5% to 57.8% (+7.3 pp), while zero-shot yields 40.9%. Of 27 tasks, 12 improved by at least 2%, 9 of them by at least 5%. A noticeable degradation occurs in only 2 tasks, the largest in Boolean Expressions.
  • The largest gains (20–50 pp) were obtained on tasks with clear structural rules or distribution shift: Dyck Languages, Ruin Names, Movie Recommendation, Hyperbaton.
  • Design: synthetic leave-one-out in-context tasks work better than Direct I/O. A shared adapter helps on BBH, because tasks there are easy to distinguish by their instructions. On ARC, separate adapters work better, because all tasks have the same input format and their gradients may conflict.
  • Resources and limitations: the code is published in the repositories ekinakyurek/marc (ARC) and adamzweiger/Fewshot-TTT (BBH). Hyperparameters were tuned on the same 80 ARC tasks, so the evaluation may be biased. The public benchmarks may have leaked into the Llama-3 pretraining data.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 works

Cite

Cite as Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim and Jacob Andreas. 2024. The Surprising Effectiveness of Test-Time Training for Few-Shot Learning. arXiv:2411.07279.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2411.07279 · arXiv 2411.07279

@misc{akyurek2024surprising,
  title = {{The Surprising Effectiveness of Test-Time Training for Few-Shot Learning}},
  author = {Ekin Akyürek, Mehul Damani, Adam Zweiger, Linlu Qiu, Han Guo, Jyothish Pari, Yoon Kim and Jacob Andreas},
  year = {2024},
  doi = {10.48550/arxiv.2411.07279},
  eprint = {2411.07279},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2411.07279}
}

References

45 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Acquaviva, S., Pu, Y., Kryven, M., Sechopoulos, T., Wong, C., Ecanow, G. E., Nye, M. I., Tessler, M. H., and Tenenbaum, J. Communicating natural programs to humans and machines. In Advances in Neural Information Processing Systems 35, 2022. URL http://papers.nips.cc/paper files/paper/2022/hash/182a… ↗
  2. Akyürek, E., Schuurmans, D., Andreas, J., Ma, T., and Zhou, D. What learning algorithm is in-context learning? Investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/pdf?id=0g0X4H8yN4 I. ↗
  3. Behrouz, A., Zhong, P., and Mirrokni, V. Titans: Learning to memorize at test time, 2025. URL https://arxiv.org/abs/2501.00663. ↗
  4. Bober-Irizar, M. and Banerjee, S. Neural networks for abstraction and reasoning. Scientific Reports, 2024. ISSN 2045-2322. doi: 10.1038/s41598-024-73582-7. URL https://doi.org/10.1038/s41598-024-73582-7. — [doi:10.1038/s41598-024-73582-7](https://doi.org/10.1038/s41598-024-73582-7) ↗
  5. Bottou, L. and Vapnik, V. Local learning algorithms. Neural Computation, 1992. ISSN 0899-7667. doi: 10.1162/neco .1992.4.6.888. URL https://doi.org/10.1162/neco .1992.4.6.888. — [doi:10.1162/neco](https://doi.org/10.1162/neco) ↗
  6. Brown, B., Juravsky, J., Ehrlich, R., Clark, R., Le, Q. V., Ré, C., and Mirhoseini, A. Large language monkeys: Scaling inference compute with repeated sampling, 2024. URL https://arxiv.org/abs/2407.21787. ↗
  7. Brown, T. B., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D. M., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Ch… — Language Models are Few-Shot Learners
  8. Butt, N., Manczak, B., Wiggers, A., Rainone, C., Zhang, D. W., Defferrard, M., and Cohen, T. CodeIt: Self-improving language models with prioritized hindsight replay. In Proceedings of the 41st International Conference on Machine Learning. PMLR, 2024. URL https://dl.acm.org/doi/10.5555/3692070.3692… ↗
  9. Chollet, F. On the measure of intelligence, 2019. URL https://arxiv.org/abs/1911.01547. ↗
  10. Chollet, F., Knoop, M., Kamradt, G., and Landers, B. ARC Prize 2024: Technical report, 2025. URL https://arxi v.org/abs/2412.04604. find
  11. Damani, M., Shenfeld, I., Peng, A., Bobu, A., and Andreas, J. Learning how hard to think: Input-adaptive allocation of LM computation. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=6qUUgw9bAZ. ↗
  12. Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. QLoRA: Efficient finetuning of quantized LLMs. In Advances in Neural Information Processing Systems 36, 2023. URL http://papers.nips.cc/paper files/p aper/2023/hash/1feb87871436031bdc0f2beaa62a0 49b-Abstract-Conference.html. ↗
  13. Gandelsman, Y., Sun, Y., Chen, X., and Efros, A. A. Test-time training with masked autoencoders. In Advances in Neural Information Processing Systems 35, 2022. URL http://papers.nips.cc/paper files/paper/2022/ hash/bcdec1c2d60f94a93b6e36f937aa0530-Abstr act-Conference.html. ↗
  14. Greenblatt, R. Getting 50% (SoTA) on ARC-AGI with GPT-4o, 2024. URL https://redwoodresearch.substa ck.com/p/getting-50-sota-on-arc-agi-with-gpt. Accessed 09-11-2024. ↗
  15. Hardt, M. and Sun, Y. Test-time training on nearest neighbors for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=CNL2bku4 ra. ↗
  16. Hodel, M. Addressing the Abstraction and Reasoning Corpus via procedural example generation, 2024. URL https://arxiv.org/abs/2404.07353. ↗
  17. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9. ↗
  18. Hübotter, J., Bongni, S., Hakimi, I., and Krause, A. Efficiently learning at test-time: Active fine-tuning of LLMs. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=NS1G1Uhny3. ↗
  19. Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. GPT-4o system card. ArXiv preprint, 2024. URL https://arxiv.org/abs/2410.21276. ↗
  20. Joachims, T. Transductive inference for text classification using support vector machines. In Proceedings of the 16th International Conference on Machine Learning. Morgan Kaufmann Publishers Inc., 1999. ISBN 1558606122.
  21. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J., Zhang, H., and Stoica, I. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP ’23. Association for Computing Machinery…
  22. LeGris, S., Vong, W. K., Lake, B. M., and Gureckis, T. M. H-ARC: A robust estimate of human performance on the Abstraction and Reasoning Corpus benchmark. ArXiv preprint, 2024. URL https://arxiv.org/abs/2409.0 1374. ↗
  23. Li, W.-D., Hu, K., Larsen, C., Wu, Y., Alford, S., Woo, C., Dunn, S. M., Tang, H., Zheng, W.-L., Pu, Y., and Ellis, K. Combining induction and transduction for abstract reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL https://op enreview.net/forum?id=Umdot… find
  24. Llama Team. The Llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783. ↗
  25. Loshchilov, I. and Hutter, F. Fixing weight decay regularization in Adam, 2018. URL https://openreview.net/forum?id=rk6qdGgCZ. ↗
  26. McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D., and Griffiths, T. L. Embers of autoregression show how large language models are shaped by the problem they are trained to solve. Proceedings of the National Academy of Sciences, 2024. doi: 10.1073/pnas.2322420121. URL https://www.pnas.org/doi/abs/1… — [doi:10.1073/pnas.2322420121](https://doi.org/10.1073/pnas.2322420121) ↗
  27. Min, S., Lewis, M., Zettlemoyer, L., and Hajishirzi, H. MetaICL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computational Linguistics, 2022a. doi: 10.… find
  28. Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics…
  29. OpenAI. GPT-4 technical report, 2024. URL https://ar xiv.org/abs/2303.08774.
  30. Opiełka, G., Rosenbusch, H., Vijverberg, V., and Stevenson, C. E. Do large language models solve ARC visual analogies like people do?, 2024. URL https://arxiv.org/abs/2403.09734. — [arXiv:2403.09734](https://arxiv.org/abs/2403.09734) ↗
  31. Ravi, S. and Larochelle, H. Optimization as a model for few-shot learning. In The Fifth International Conference on Learning Representations, 2017. URL https://op enreview.net/forum?id=rJY0-Kcll. find
  32. Snell, C. V., Lee, J., Xu, K., and Kumar, A. Scaling test-time compute optimally can be more effective than scaling LLM parameters. In The Thirteenth International Conference on Learning Representations, 2025. URL https://openreview.net/forum?id=4FWAwZtd2n. ↗
  33. Srivastava, A., Rastogi, A., Rao, A., Shoeb, A. A. M., Abid, A., Fisch, A., Brown, A. R., Santoro, A., Gupta, A., Garriga-Alonso, A., et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transactions on Machine Learning Research, 2023. ISSN 2835-8856.…
  34. Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. A., and Hardt, M. Test-time training with self-supervision for generalization under distribution shifts. In Proceedings of the 37th International Conference on Machine Learning. PMLR, 2020. URL http://proceedings.mlr.press/v119/sun20b.html. ↗
  35. Sun, Y., Li, X., Dalal, K., Xu, J., Vikram, A., Zhang, G., Dubois, Y., Chen, X., Wang, X., Koyejo, S., Hashimoto, T., and Guestrin, C. Learning to (learn at test time): RNNs with expressive hidden states, 2024. URL https://arxiv.org/abs/2407.04620. ↗
  36. Suzgun, M., Scales, N., Schärli, N., Gehrmann, S., Tay, Y., Chung, H. W., Chowdhery, A., Le, Q., Chi, E., Zhou, D., and Wei, J. Challenging BIG-Bench tasks and whether chain-of-thought can solve them. In Findings of the 2023 Conference of the Association for Computational Linguistics. Association f…
  37. Todd, E., Li, M., Sharma, A. S., Mueller, A., Wallace, B. C., and Bau, D. Function vectors in large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=AwyxtyMwaG. torchtune Maintainers and Contributors. torchtune: Py-Torch… ↗
  38. Veldkamp, K., Rosenbusch, H., Thoms, L., and Stevenson, C. Solving ARC visual analogies with neural embeddings and vector arithmetic: A generalized method. OSF, 2023. doi: 10.17605/OSF.IO/AKP86. URL https://osf.io/akp86/. — [doi:10.17605/osf.io/akp86](https://doi.org/10.17605/osf.io/akp86) ↗
  39. Wang, K. A., Shi, J., and Fox, E. B. Test-time regression: a unifying framework for designing sequence models with associative memory. ArXiv preprint, 2025. URL https://arxiv.org/abs/2501.12352. — [arXiv:2501.12352](https://arxiv.org/abs/2501.12352) ↗
  40. Wang, R., Zelikman, E., Poesia, G., Pu, Y., Haber, N., and Goodman, N. Hypothesis Search: Inductive reasoning with language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=G7UtIGQmjm. ↗
  41. Wang, X., Wei, J., Schuurmans, D., Le, Q. V., Chi, E. H., Narang, S., Chowdhery, A., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=1PL1NIMMrw. ↗
  42. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Ichter, B., Xia, F., Chi, E. H., Le, Q. V., and Zhou, D. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems 35, 2022. URL http://papers.nips.cc/pap er files/paper/2022/hash/9d560961… ↗
  43. Wu, Z., Qiu, L., Ross, A., Akyürek, E., Chen, B., Wang, B., Kim, N., Andreas, J., and Kim, Y. Reasoning or reciting? Exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for… find
  44. Yao, S., Yu, D., Zhao, J., Shafran, I., Griffiths, T., Cao, Y., and Narasimhan, K. Tree of Thoughts: Deliberate problem solving with large language models. In Advances in Neural Information Processing Systems 36, 2023. URL http://papers.nips.cc/paper files/paper/2023/ hash/271db9922b8d1f4dd7aaef84ed… ↗
  45. Zhao, S., Nguyen, T., and Grover, A. Probing the decision boundaries of in-context learning in large language models. In ICML 2024 Workshop on In-Context Learning, 2024. URL https://openreview.net/forum?id=rf CtCcPuSt. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/akyurek-2024-surprising-effectiveness-test-time-training/