Elamur Research⌕
☰
home / papers / 2023 / Learning to Reason and Memorize with Self-Notes

Learning to Reason and Memorize with Self-Notes

Paper · 2023 · Jack Lanchantin, Shubham Toshniwal, Jason Weston et al.

Summary

Transformer language models use the same amount of computation for every token and keep no explicit record of what they have inferred so far. This makes multi-step reasoning hard, and so is tracking how the state of an entity changes across a text. Chain-of-thought and Scratchpad help only partly, because they begin reasoning after the whole context and question have been read. Lanchantin et al. propose Self-Notes. The model may stop at any point in the input, emit a special start token and write a short note, such as a sub-question with its answer or an updated variable value. It then continues reading, with the note kept inside the context. Notes therefore act both as intermediate reasoning steps and as a recurrent working memory. The method is tested in four regimes: supervised, semi-supervised, unsupervised and few-shot prompting of a pretrained LLM. The evaluation uses seven datasets covering synthetic stories, program evaluation, chess and math word problems. In every regime, Self-Notes outperform both a vanilla LM and post-context CoT/Scratchpad baselines. This suggests that reasoning while reading generalizes better than reasoning only at the end.

Key points

  • Benchmarks: the paper uses seven datasets.
    • A newly proposed synthetic Toy-Story task.
    • Two synthetic algorithmic/program-evaluation tasks.
    • Two real chess-game tasks.
    • MultiArith and GSM8K.
  • Note mechanism: a note opens with a token from a predefined start set N_sta. It closes with an end token from N_end or after a fixed token limit. The generated tokens are then spliced into the context where the note began.
  • Semi-supervised training: examples without gold notes get a prefix token s. Because test inputs lack this prefix, the model is pushed to write notes at inference.
  • Unsupervised training: this regime relies on the model learning to generate questions from ordinary LM loss on QA data. Two further techniques support it:
    • Start-token probabilities are scaled by a boosting constant B > 1.
    • Several note variants are sampled, and the most confident answer is kept.
  • Data efficiency on Toy-Story: 10k training samples carry about 70k extra Self-Note QA pairs, roughly 80k QA pairs in total. A Vanilla model given 1500% more samples still performs far worse.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Psychology · 1,673 worksExperimental and Cognitive Psychology · 950 worksDevelopmental and Educational Psychology · 533 worksNeuroscience · 192 worksCognitive Neuroscience · 191 works

🧭 Topics 1Research subjects and languages the work is about.Reasoning · 1,948 works

Cite

Cite as Jack Lanchantin, Shubham Toshniwal, Jason Weston, Arthur Szlam and Sainbayar Sukhbaatar. 2023. Learning to Reason and Memorize with Self-Notes. arXiv:2305.00833.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2305.00833 · arXiv 2305.00833

@misc{lanchantin2023learning,
  title = {{Learning to Reason and Memorize with Self-Notes}},
  author = {Jack Lanchantin, Shubham Toshniwal, Jason Weston, Arthur Szlam and Sainbayar Sukhbaatar},
  year = {2023},
  doi = {10.48550/arxiv.2305.00833},
  eprint = {2305.00833},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2305.00833}
}

References

50 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NeurIPS, 2017.
  2. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  3. Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, and Augustus Odena. Show Your Work: Scratchpads for Intermediate Computation with Language Models. arXiv, abs/2112.00114, 2021.
  4. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. In NeurIPS, 2022.
  5. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712, 2023.
  6. Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V. Le, and Charles Sutton. Program Synthesis with Large Language Models. arXiv, abs/2108.07732, 2021.
  7. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, and Mike Lewis. Measuring and Narrowing the Compositionality Gap in Language Models. arXiv:2210.03350, abs/2210.03350, 2022. find
  8. Antonia Creswell, Murray Shanahan, and Irina Higgins. Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning. In ICLR, 2023. find
  9. E. Zelikman, Yuhuai Wu, and Noah D. Goodman. STaR: Bootstrapping Reasoning With Reasoning. In NeurIPS, 2022.
  10. Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, Pierre Sermanet, Noah Brown, Tomas Jackson, Linda Luu, Sergey Levine, Karol Hausman, and Brian Ichter. Inner Monologue: Embodied Reasoning through Planning with Lan… find
  11. Angela Fan, Thibaut Lavril, Edouard Grave, Armand Joulin, and Sainbayar Sukhbaatar. Addressing Some Limitations of Transformers with Feedback Memory. arXiv, 2020. find
  12. Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural computation, 1997. find
  13. Da Ju, Stephen Roller, Sainbayar Sukhbaatar, and Jason Weston. Staircase Attention for Recurrent Processing of Sequences. In NeurIPS, 2022. find
  14. DeLesley Hutchins, Imanol Schlag, Yuhuai Wu, Ethan Dyer, and Behnam Neyshabur. Block-Recurrent Transformers. In NeurIPS, 2022. find
  15. Paul van den Broek, Mary Jane White, Panayiota Kendeou, and Sarah Carlson. Reading between the lines. Developmental and individual differences in cognitive processes in reading comprehension. In. K. Wagner, C. Schatschneider, & C. Plythian-Sence (Eds.), Beyond decoding: The behavioral and biological… find
  16. Cem Anil, Yuhuai Wu, Anders Andreassen, Aitor Lewkowycz, Vedant Misra, Vinay Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring Length Generalization in Large Language Models. In NeurIPS, 2022.
  17. Shubham Toshniwal, Sam Wiseman, Karen Livescu, and Kevin Gimpel. Chess as a Testbed for Language Model State Tracking. In AAAI, 2022. find
  18. Subhro Roy and Dan Roth. Solving general arithmetic word problems. arXiv preprint arXiv:1608.01413, 2016.
  19. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. ArXiv, abs/2110.14168, 2021.
  20. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language Models are Unsupervised Multitask Learners. In OpenAI blog, 2019. find
  21. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. HuggingFace’s Transformers: State-of-the-art Natural Language Processing. In EMNLP: System Demonstrations, 2020.
  22. Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks. In ICLR, 2016.
  23. Ben Wang and Aran Komatsuzaki. Gpt-j-6b: A 6 billion parameter autoregressive language model, 2021. find
  24. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023.
  25. Léon Bottou. From machine learning to machine reasoning: An essay. Machine learning, 94: 133–149, 2014. find
  26. Sainbayar Sukhbaatar, Arthur Szlam, Jason Weston, and Rob Fergus. End-to-End Memory Networks. NeurIPS, 2015. find
  27. Mikael Henaff, Jason Weston, Arthur Szlam, Antoine Bordes, and Yann LeCun. Tracking the world state with recurrent entity networks. arXiv preprint arXiv:1612.03969, 2016.
  28. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
  29. Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal Transformers. In ICLR, 2019. find
  30. Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. e-SNLI: Natural Language Inference with Natural Language Explanations. In NeurIPS, 2018.
  31. Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale generation: Learning to solve and explain algebraic word problems. arXiv preprint arXiv:1705.04146, 2017. find
  32. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large Language Models are Zero-Shot Reasoners. In NeurIPS, 2022.
  33. H. Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions. arXiv, abs/2212.10509, 2022.
  34. Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al. Do As I Can, Not As I Say: Grounding Language in Robotic Affordances. arXiv:2204.01691, 2022.
  35. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023.
  36. Ofir Press, Noah Smith, and Mike Lewis. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. In ICLR, 2022. find
  37. Masato Neishi and Naoki Yoshinaga. On the Relation between Position Information and Sentence Length in Neural Machine Translation. In CoNLL, 2019. find
  38. Shun Kiyono, Sosuke Kobayashi, Jun Suzuki, and Kentaro Inui. SHAPE: Shifted Absolute Position Embedding for Transformers. In EMNLP, 2021. find
  39. Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionality decomposed: How do neural networks generalise? In IJCAI, 2020.
  40. Koustuv Sinha, Amirhossein Kazemnejad, Siva Reddy, Joelle Pineau, Dieuwke Hupkes, and Adina Williams. The Curious Case of Absolute Position Embeddings. Findings of EMNLP, 2022. find
  41. Alex Graves. Adaptive Computation Time for Recurrent Neural Networks. arXiv:1603.08983, 2016. find
  42. Tolga Bolukbasi, Joseph Wang, Ofer Dekel, and Venkatesh Saligrama. Adaptive Neural Networks for Efficient Inference. In ICML, 2017. find
  43. Andrea Banino, Jan Balaguer, and Charles Blundell. PonderNet: Learning to Ponder. In ICML Workshop on Automated Machine Learning, 2021.
  44. Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. PEER: A Collaborative Language Model. In ICLR, 2023.
  45. Jiatao Gu, Changhan Wang, and Junbo Zhao. Levenshtein Transformer. NeurIPS, 2019. find
  46. Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion Transformer: Flexible Sequence Generation via Insertion Operations. In ICML, 2019. find
  47. Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. Can You Unpack That? Learning to Rewrite Questions-in-Context. In EMNLP-IJCNLP, 2019.
  48. Gangwoo Kim, Sungdong Kim, Kang Min Yoo, and Jaewoo Kang. Generating Information-Seeking Conversations from Unlabeled Documents. In EMNLP, 2022. find
  49. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv:2104.09864, 2021.
  50. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling Language Modeling with Pathways. arXiv:2204.02311, 2022.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lanchantin-2023-learning-reason-memorize-self-notes/