Elamur Research⌕
☰
home / papers / 2023 / Future Lens: Anticipating Subsequent Tokens from a Single Hidden State

Future Lens: Anticipating Subsequent Tokens from a Single Hidden State

Paper · 2023 · Koyena Pal, Jiuding Sun, Andrew Yuan et al.

Summary

The paper asks whether a transformer's internal state for one token carries information about tokens beyond the immediate next one, specifically positions t+2 and later. Working with GPT-J-6B, the authors compare three ways of reading that information out of a single hidden state taken at one layer and one position.

  • Linear models: these map the state either directly to vocabulary logits or to a future final-layer hidden state, which is then decoded with the model's own output head. This extends Logit lens and Tuned Lens.
  • Fixed-prompt causal intervention: a hidden state is transplanted into an unrelated generic prompt to see whether the original continuation still emerges.
  • Learned-prompt intervention: a soft prefix is optimized with a KL objective so that the transplanted state reveals its future tokens.

The learned prompt performs best. At some layers it matches the model's subsequent-token prediction more than 48% of the time. The work also yields a "Future Lens" visualization of predicted multi-token continuations per state, giving interpretability researchers a view beyond single next-token predictions.

Key points

  • The model studied is GPT-J-6B. The goal is to anticipate the model's own predictions N tokens ahead from one hidden state h at layer l, position T.
  • A running example illustrates the transplant idea: a state from "Madison Square Garden is located in" placed into a generic prompt should still steer generation toward "New York City."
  • Both the linear and learned-prompt methods beat an empirical bigram baseline at N = 1. Reusing the pretrained decoder head helps only marginally, and mainly at late layers.
  • For later tokens, accuracy of the learned prompt peaks at middle layers. Immediate next-token (N = 0) accuracy instead peaks at the last layer.
  • Code, data, and the Future Lens demo are available at future.baulab.info.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Massachusetts AmherstAmherst Center · United States114 works · 68,731 citations🎓Northeastern UniversityBoston · United States68 works · 7,316 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

Cite

Cite as Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C. Wallace and David Bau. 2023. Future Lens: Anticipating Subsequent Tokens from a Single Hidden State. arXiv:2311.04897.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.conll-1.37 · arXiv 2311.04897

@misc{pal2023future,
  title = {{Future Lens: Anticipating Subsequent Tokens from a Single Hidden State}},
  author = {Koyena Pal, Jiuding Sun, Andrew Yuan, Byron C. Wallace and David Bau},
  year = {2023},
  doi = {10.18653/v1/2023.conll-1.37},
  eprint = {2311.04897},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2023.conll-1.37}
}

References

31 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. 2023. Eliciting latent predictions from transformers with the tuned lens.
  2. Wes Gurnee, Neel Nanda, Matthew Pauly, Katherine Harvey, Dmitrii Troitskii, and Dimitris Bertsimas. 2023. Finding neurons in a haystack: Case studies with sparse probing.
  3. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. In The Eleventh International Conference on Learning Representations. find
  4. Adi Haviv, Ido Cohen, Jacob Gidron, Roei Schuster, Yoav Goldberg, and Mor Geva. 2023. Understanding transformer memorization recall through idioms. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 248–264, Dubrovnik, Croatia. Assoc…
  5. Nicholas Carlini, Chang Liu, Úlfar Erlingsson, Jernej Kos, and Dawn Song. 2019. The secret sharer: Evaluating and testing unintended memorization in neural networks. In Proceedings of the 28th USENIX Conference on Security Symposium, SEC’19, page 267–284, USA. USENIX Association.
  6. Evan Hernandez, Arnab Sen Sharma, Tal Haklay, Kevin Meng, Martin Wattenberg, Jacob Andreas, Yonatan Belinkov, and David Bau. 2023. Linearity of relation decoding in transformer language models.
  7. Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, Alina Oprea, and Colin Raffel. 2021. Extracting training data from large language models. In USENIX Security Symposium.
  8. Daphne Ippolito, Florian Tramèr, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A. Choquette-Choo, and Nicholas Carlini. 2023. Preventing verbatim memorization in language models gives a false sense of privacy. find
  9. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
  10. Michael I Jordan. 1997. Serial order: A parallel distributed processing approach. In Advances in psychology, volume 121, pages 471–495. Elsevier. find
  11. Shahar Katz and Yonatan Belinkov. 2023. Interpreting transformer’s attention dynamic memory and visualizing the semantic information flow of gpt. find
  12. Alexander Yom Din, Taelin Karidi, Leshem Choshen, and Mor Geva. 2023. Jump to conclusions: Short-cutting transformers with linear transformations. find
  13. Jun Kong, Jin Wang, Liang-Chih Yu, and Xuejie Zhang. 2022. Accelerating inference for pretrained language models by unified multi-perspective early exiting. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4677–4686, Gyeongju, Republic of Korea. International C… find
  14. Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179–211. find
  15. Vitaly Feldman and Chiyuan Zhang. 2020. What neural networks memorize and why: Discovering the long tail via influence estimation. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA. Curran Associates Inc.
  16. Eric Lehman, Sarthak Jain, Karl Pichotta, Yoav Goldberg, and Byron C Wallace. 2021. Does bert pre-trained on clinical notes reveal sensitive data? arXiv preprint arXiv:2104.07762. — [arXiv:2104.07762](https://arxiv.org/abs/2104.07762) find
  17. Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. 2020. The pile: An 800gb dataset of diverse text for language modeling. find
  18. Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190. find
  19. Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022a. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 36. fine-tuning and extension to regression. In Proceedings of the 16th Conference of the European Chapter of the Association for Co… find
  20. Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. 2022b. Mass editing memory in a transformer. arXiv preprint arXiv:2210.07229. find
  21. Minjia Zhang and Yuxiong He. 2020. Accelerating training of transformer-based language models with progressive layer dropping. nostalgebraist. 2020. interpreting gpt: the logit lens. find
  22. Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. find
  23. Tal Schuster, Adam Fisch, Jai Gupta, Mostafa Dehghani, Dara Bahri, Vinh Q. Tran, Yi Tay, and Donald Metzler. 2022. Confident adaptive language modeling. In Advances in Neural Information Processing Systems. find
  24. Yixuan Su, Deng Cai, Yan Wang, David Vandyke, Simon Baker, Piji Li, and Nigel Collier. 2021. Non-autoregressive text generation with pre-trained language models. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 234–243… find
  25. Jiuding Sun, Chantal Shaib, and Byron C Wallace. 2023. Evaluating the zero-shot robustness of instruction-tuned language models. arXiv preprint arXiv:2306.11270. — [arXiv:2306.11270](https://arxiv.org/abs/2306.11270) find
  26. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
  27. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. arXiv preprint arXiv:1908.07125. find
  28. Ben Wang and Aran Komatsuzaki. 2021. Gpt-j-6b: A 6 billion parameter autoregressive language model. find
  29. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Transactions on Ma…
  30. Yisheng Xiao, Lijun Wu, Junliang Guo, Juntao Li, Min Zhang, Tao Qin, and Tie yan Liu. 2023. A survey on non-autoregressive generation for neural machine translation and beyond. find
  31. Ji Xin, Raphael Tang, Yaoliang Yu, and Jimmy Lin. 2021. BERxiT: Early exiting for BERT with better find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/pal-2023-future-lens-anticipating-subsequent-tokens/