Elamur Research⌕
☰
home / papers / 2026 / Did You Forget What I Asked? Prospective Memory Failures in Large Language Models

Did You Forget What I Asked? Prospective Memory Failures in Large Language Models

Paper · 2026 · Avni Mittal

Summary

The paper asks whether LLMs show something like human prospective-memory lapses. In humans, a deferred intention is more often forgotten when an absorbing ongoing task competes for attention. Here, the "intention" is a verifiable IFEval formatting rule, and the competing work is an added benchmark problem of increasing operational load: TriviaQA, MMLU, GSM8K (including 3- and 5-problem chains) and CNN/DailyMail.

Three models were tested on more than 8,000 prompts: o4-mini, DeepSeek-V3.1 and Llama-3.3-70B-Instruct. Both format adherence and task correctness were scored with deterministic code checkers, with no LLM judges. The authors also compared plain embedding of the constraint against a salience-boosted template, which adds an emphatic header and a closing reminder, and they ran experiments that stack several constraints at once.

Adding a task reduces compliance by 2–21%. Rules that must be executed at the end of the response suffer most, and the interference also runs the other way, hurting task accuracy. The reminder largely restores compliance, but it becomes unreliable once several constraints are combined. The results are directly useful for prompt design and for building more trustworthy evaluation.

Key points

  • Setup: the study uses 15 of the 25 IFEval instruction types, with 90 constraints (6 per type) grouped into seven categories. These are paired with TriviaQA, MMLU, GSM8K and CNN/DailyMail and run over 3 seeds with greedy decoding.
  • Type dependence: terminal constraints lose up to 50% compliance under load, while avoidance constraints stay almost unaffected.
  • Reminder effect: the reminder template lifts compliance to roughly 90–100% in many conditions, often above the no-task baseline. Under stacking, however, its effect swings between +25% and −20%.
  • Reverse interference: for one model, adding formatting constraints cut GSM8K accuracy from 93% to 27%.
  • Stacking results: joint compliance collapses as constraints and load are combined. For example, o4-mini keeps only 32.5% with 5 constraints and 3 GSM8K problems.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksPsychology · 1,552 worksExperimental and Cognitive Psychology · 870 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Avni Mittal. 2026. Did You Forget What I Asked? Prospective Memory Failures in Large Language Models. Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.trustnlp-main.33

@inproceedings{mittal2026did,
  title = {{Did You Forget What I Asked? Prospective Memory Failures in Large Language Models}},
  author = {Avni Mittal},
  booktitle = {Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.trustnlp-main.33},
  url = {https://doi.org/10.18653/v1/2026.trustnlp-main.33}
}

References

15 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Marcel Binz and Eric Schulz. 2023. Using cognitive psychology to understand gpt-3. Proceedings of the National Academy of Sciences, 120(6):e2218523120. find
  2. Xinyun Chen, Ryan A Chi, Xuezhi Wang, and Denny Zhou. 2024. Premise order matters in reasoning with large language models. arXiv preprint arXiv:2402.08939. find
  3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  4. Gilles O Einstein, Mark A McDaniel, Ruthann Thomas, Sara Mayfield, Hilary Shank, Nova Morrisette, and Jennifer Breneiser. 2005. Multiple processes in prospective memory retrieval: factors determining monitoring versus spontaneous retrieval. Journal of Experimental Psychology: General, 134(3):327. find
  5. Bryan Guan, Tanya Roosta, Peyman Passban, and Mehdi Rezagholizadeh. 2025. The order effect: investigating prompt sensitivity to input order in llms. arXiv preprint arXiv:2502.04134. — [arXiv:2502.04134](https://arxiv.org/abs/2502.04134) find
  6. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300.
  7. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. Ruler: What’s the real context size of your long-context language models? arXiv preprint arXiv:2404.06654.
  8. Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. 2024. Followbench: A multi-level fine-grained constraints following benchmark for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computa…
  9. Erik Jones and Jacob Steinhardt. 2022. Capturing failures of large language models via human cognitive biases. Advances in Neural Information Processing Systems, 35:11785–11799. find
  10. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601–1611.
  11. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the middle: How language models use long contexts. Transactions of the association for computational linguistics, 12:157–173.
  12. Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. 2024. Keeping llms aligned after fine-tuning: The crucial role of prompt templates. Advances in Neural Information Processing Systems, 37:118603–118631. find
  13. Richard L Marsh, Jason L Hicks, Gabriel I Cook, Jeffrey S Hansen, and Andrew L Pallos. 2003. Interference to ongoing activities covaries with the characteristics of an event-based intention. Journal of Experimental Psychology: Learning, Memory, and Cognition, 29(5):861. find
  14. Mark A McDaniel and Gilles O Einstein. 2000. Strategic and automatic processes in prospective memory retrieval: A multiprocess framework. Applied Cognitive Psychology: The Official Journal of the Society for Applied Research in Memory and Cognition, 14(7):S127–S144. find
  15. Yunjia Qi, Hao Peng, Xiaozhi Wang, Amy Xin, Youfeng Liu, Bin Xu, Lei Hou, and Juanzi Li. 2025. Agentif: Benchmarking instruction following of large language models in agentic scenarios. arXiv preprint arXiv:2505.16944. — [arXiv:2505.16944](https://arxiv.org/abs/2505.16944) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mittal-2026-did-you-forget-what-i/