Elamur Research⌕
☰
home / papers / 2026 / Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMs

Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMs

Paper · 2026 · Yuxuan Jiang and Francis Ferraro

Summary

Research on memorization in LLMs has mostly focused on math and logic, where recalling previously seen problems is treated as a pure shortcut. The authors argue that story understanding works differently. There, models should use background knowledge, but selectively and together with the narrative at hand. They borrow the cognitive-science split between verbatim and gist memory and build a two-level intervention scheme:

  • Inductive Setting: the input is left unchanged, and gist-oriented prompts nudge the model toward context-driven recall.
  • Restrictive Setting: character names are swapped for culturally distinct alternatives, which blocks exact recall.

GPT-4o, LLaMA 3.3-70B and DeepSeek V3 are tested on CharBench, a collection of six character-centric benchmarks. A new dataset, CharScript, checks how familiar the models are with the underlying fiction. Name replacement cuts accuracy by up to roughly 45%, while inductive prompting keeps scores largely stable. The results suggest that current benchmarks overstate narrative reasoning. They also show that prompting can steer models toward bounded, context-relevant use of memory.

Key points

  • CharBench combines MovieCoref, TVSHOWGUESS, PERSONET, CSI Corpus, FriendsQA and SummScreen. It keeps their original splits and metrics (exact match, F1, ROUGE) and covers TV shows, movies and novels.
  • CharScript contains 200 passages averaging about 312 words. They come from 40 fictional sources (11 TV series, 9 movies, 20 novels), with 5 per source, and models must attribute each passage to its source as a test of verbatim memorization.
  • Size of the drop: the abstract gives a maximum Restrictive-Setting loss of 45.2%, while the introduction says 45.8%. For every setting, the authors repeated each model's evaluation three times and averaged the scores, and the total API cost was about $100.
  • Task type matters: action-oriented tasks (Character Guessing, CSI role detection) lose more under the combined GIST+NR intervention than motivation-oriented ones (PERSONET, FriendsQA). On some benchmarks, constraining memory even helps by reducing knowledge conflicts.
  • Stated limitations: sources are mostly Western popular culture, which may exaggerate the effects. Results may be sensitive to prompt wording, and GPT-4o's training data cannot be verified.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Maryland, Baltimore CountyBaltimore · United States14 works · 915 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 works

Cite

Cite as Yuxuan Jiang and Francis Ferraro. 2026. Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMs. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.261

@inproceedings{jiang2026beyond,
  title = {{Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMs}},
  author = {Yuxuan Jiang and Francis Ferraro},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.261},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.261}
}

References

60 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Meta AI. 2024. Llama 3.3 model card. https://github.com/meta-llama/llama-models/ blob/main/models/llama3_3/MODEL_CARD.md. Accessed: 2025-02-09. ↗
  2. Keivan Alizadeh, Seyed Iman Mirzadeh, Dmitry Belenko, S Khatamifard, Minsik Cho, Carlo C Del Mundo, Mohammad Rastegari, and Mehrdad Farajtabar. 2024. Llm in a flash: Efficient large language model inference with limited memory. In Proceedings of the 62nd Annual Meeting of the Association for Computa…
  3. David Bamman, Olivia Lewke, and Anya Mansoor. 2020. An Annotated Dataset of Coreference in English Literature. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 44–54, Marseille, France. European Language Resources Association.
  4. Sabyasachee Baruah and Shrikanth Narayanan. 2023. Character coreference resolution in movie screenplays. In Findings of the Association for Computational Linguistics: ACL 2023, pages 10300–10313.
  5. Stella Biderman, Usvsn Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivanshu https://chatgpt.com/ ↗
  6. Purohit, and Edward Raff. 2024. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems, 36. find
  7. Terra Blevins and Luke Zettlemoyer. 2022. Language Contamination Helps Explain the Cross-lingual Capabilities of English Pretrained Models. arXiv preprint. ArXiv:2204.08110. find
  8. Lea Frermann, Shay B. Cohen, and Mirella Lapata. 2018. Whodunnit? Crime Drama as a Case for Natural Language Understanding. Transactions of the Association for Computational Linguistics, 6:1–15. Place: Cambridge, MA Publisher: MIT Press.
  9. Tanya Goyal, Junyi Jessy Li, and Greg Durrett. 2023. News Summarization and Evaluation in the Era of GPT-3. arXiv preprint. ArXiv:2209.12356 [cs]. find
  10. Gordon H Bower and Daniel G Morrow. 1990. Mental models in narrative comprehension. Science, 247(4938):44–48. find
  11. Yuxuan Jiang and Francis Ferraro. 2026. Scribe: Structured mid-level supervision for tool-using language models. arXiv preprint arXiv:2601.03555. — [arXiv:2601.03555](https://arxiv.org/abs/2601.03555) find
  12. Faeze Brahman, Meng Huang, Oyvind Tafjord, Chao Zhao, Mrinmaya Sachan, and Snigdha Chaturvedi. 2021. "Let Your Characters Tell Their Story": A Dataset for Character-Centric Narrative Understanding. arXiv preprint. ArXiv:2109.05438 [cs]. find
  13. Yuxuan Jiang, Dawei Li, and Frank Ferraro. 2025. Drp: Distilled reasoning pruning with skill-aware step decomposition for efficient large reasoning models. arXiv preprint arXiv:2505.13975.
  14. Charles J Brainerd and Valerie F Reyna. 2002. Fuzzy-trace theory: Dual processes in memory, reasoning, and cognitive neuroscience. Advances in child development and behavior, 28:41–100. find
  15. Mingyu Jin, Weidi Luo, Sitao Cheng, Xinyi Wang, Wenyue Hua, Ruixiang Tang, William Yang Wang, and Yongfeng Zhang. 2024. Disentangling memory and reasoning ability in large language models. arXiv preprint arXiv:2411.13504.
  16. Nicholas Carlini, Daphne Ippolito, Katherine Lee, Florian Tramèr, Matthew Jagielski, and Chiyuan Zhang. 2023. QUANTIFYING MEMORIZATION ACROSS NEURAL LANGUAGE MODELS. arXiv preprint arXiv:2202.07646. find
  17. Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Generalization through Memorization: Nearest Neighbor Language Models. arXiv preprint. ArXiv:1911.00172. find
  18. Tommaso Caselli and Piek Vossen. 2017. The Event StoryLine Corpus: A New Benchmark for Causal and Temporal Relation Extraction. In Proceedings of the Events and Stories in the News Workshop, pages 77–86, Vancouver, Canada. Association for Computational Linguistics. find
  19. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611–626.
  20. Kent Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023a. Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7312– 7327, Singapore. Association for Computational Linguistics.
  21. Kent K Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023b. Speak, memory: An archaeology of books known to chatgpt/gpt-4. arXiv preprint arXiv:2305.00118.
  22. Bowen Chen, Namgi Han, and Yusuke Miyao. 2024. A multi-perspective analysis of memorization in large language models. arXiv preprint arXiv:2405.11577. find
  23. Mingda Chen, Zewei Chu, Sam Wiseman, and Kevin Gimpel. 2022. SummScreen: A Dataset for Abstractive Screenplay Summarization. arXiv preprint. ArXiv:2104.07091 [cs].
  24. Yu-Hsin Chen and Jinho D Choi. 2016. Character identification on multiparty conversation: Identifying mentions of characters in tv shows. In Proceedings of the 17th annual meeting of the special interest group on discourse and dialogue, pages 90–100. find
  25. Pietro Lesci, Clara Meister, Thomas Hofmann, Andreas Vlachos, and Tiago Pimentel. 2024. Causal estimation of memorisation profiles. arXiv preprint arXiv:2406.04327. find
  26. Bo Li, Qinghua Zhao, and Lijie Wen. 2024. Rome: Memorization insights from text, logits and representation. Preprint, arXiv:2403.00510. — [arXiv:2403.00510](https://arxiv.org/abs/2403.00510) find
  27. Dawei Li, Hengyuan Zhang, Yanran Li, and Shiping Yang. 2023. Multi-level contrastive learning for script-based character understanding. arXiv preprint arXiv:2310.13231. find
  28. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  29. Robert McKee. 2010. Story: style, structure, substance, and the principles of screenwriting. Harper Collins. find
  30. OpenAI. 2023. Chatgpt: Optimizing language models for dialogue. https://openai.com/research/ chatgpt. Accessed: 2025-02-09. ↗
  31. Alison H Paris and Scott G Paris. 2003. Assessing narrative comprehension in young children. Reading Research Quarterly, 38(1):36–76. find
  32. Akshara Prabhakar, Thomas L Griffiths, and R Thomas McCoy. 2024. Deciphering the factors influencing the efficacy of chain-of-thought: Probability, memorization, and noisy reasoning. arXiv preprint arXiv:2407.01687. find
  33. USVSN Sai Prashanth, Alvin Deng, Kyle O’Brien, Jyothir SV, Mohammad Aflah Khan, Jaydeep Borkar, Christopher A Choquette-Choo, Jacob Ray Fuehne, Stella Biderman, Tracy Ke, et al. 2024. Recite, reconstruct, recollect: Memorization in lms as a multifaceted phenomenon. arXiv preprint arXiv:2406.17746. — [arXiv:2406.17746](https://arxiv.org/abs/2406.17746) find
  34. Valerie F Reyna and Charles J Brainerd. 1998. Fuzzy-trace theory and false memory: New frontiers. Journal of experimental child psychology, 71(2):194–209. find
  35. Yisi Sang, Xiangyang Mou, Jing Li, Jeffrey Stanton, and Mo Yu. 2022a. A survey of machine narrative reading comprehension assessments. arXiv preprint arXiv:2205.00299. — [arXiv:2205.00299](https://arxiv.org/abs/2205.00299) find
  36. Yisi Sang, Xiangyang Mou, Mo Yu, Shunyu Yao, Jing Li, and Jeffrey Stanton. 2022b. Tvshowguess: Character comprehension in stories as speaker guessing. arXiv preprint arXiv:2204.07721. find
  37. Yisi Sang, Xiangyang Mou, Mo Yu, Shunyu Yao, Jing Li, and Jeffrey Stanton. 2022c. TVShowGuess: Character Comprehension in Stories as Speaker Guessing. arXiv preprint. ArXiv:2204.07721 [cs]. find
  38. Tianhao Shen, Sun Li, Quan Tu, and Deyi Xiong. 2024. RoleEval: A Bilingual Role Evaluation Benchmark for Large Language Models. arXiv preprint. ArXiv:2312.16132 [cs]. — [arXiv:2312.16132](https://arxiv.org/abs/2312.16132) find
  39. Dominik Stammbach, Maria Antoniak, and Elliott Ash. 2022. Heroes, Villains, and Victims, and GPT-3: Automated Extraction of Character Roles Without Training Data. In Proceedings of the 4th Workshop of Narrative Understanding (WNU2022), pages 47–56, Seattle, United States. Association for Computation… find
  40. Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016. MovieQA: Understanding Stories in Movies through Question-Answering. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4631–4640, Las Vegas, NV, USA. IEEE. find
  41. Yongqi Tong, Yifan Wang, Dawei Li, Sizhe Wang, Zi Lin, Simeng Han, and Jingbo Shang. 2023. Eliminating reasoning via inferring with planning: A new framework to guide llms’ non-linear thinking. arXiv preprint arXiv:2310.12342. — [arXiv:2310.12342](https://arxiv.org/abs/2310.12342) find
  42. Sai Vallurupalli, Katrin Erk, and Francis Ferraro. 2024. Saga: A participant-specific examination of story alternatives and goal applicability for a deeper understanding of complex events. arXiv preprint arXiv:2408.05793. find
  43. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
  44. Yike Wang, Shangbin Feng, Heng Wang, Weijia Shi, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. 2024. Resolving Knowledge Conflicts in Large Language Models. arXiv preprint. ArXiv:2310.00935.
  45. Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. 2024. On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.23123.
  46. Siheng Xiong, Ali Payani, Ramana Kompella, and Faramarz Fekri. 2024. Large language models can learn temporal reasoning. arXiv preprint arXiv:2401.06853. find
  47. Ningning Xu, Yuxuan Jiang, Shubhashis Roy Dipta, and Zhang Hengyuan. 2025. Learning how to use tools, not just when: Pattern-aware tool-integrated reasoning. MATH-AI @ NeurIPS 2025. find
  48. Zhaochen Su, Jun Zhang, Xiaoye Qu, Tong Zhu, Yanshu Li, Jiashuo Sun, Juntao Li, Min Zhang, and Yu Cheng. 2024. ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM. arXiv preprint. ArXiv:2408.12076. — [arXiv:2408.12076](https://arxiv.org/abs/2408.12076) find
  49. Shiping Yang, Renliang Sun, and Xiaojun Wan. 2023a. A new benchmark and reverse validation method for passage-level hallucination detection. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 3898–3908. find
  50. Renliang Sun, Mengyuan Liu, Shiping Yang, Rui Wang, Junqing He, and Jiaxing Zhang. 2024. Fostering natural conversation in large language models with nico: a natural interactive conversation dataset. arXiv preprint arXiv:2408.09330. — [arXiv:2408.09330](https://arxiv.org/abs/2408.09330) find
  51. Shiping Yang, Jie Wu, Wenbiao Ding, Ning Wu, Shining Liang, Ming Gong, Hengyuan Zhang, and Dongmei Zhang. 2025. Quantifying the robustness of retrieval-augmented language models against spurious features in grounding data. arXiv preprint arXiv:2503.05587. — [arXiv:2503.05587](https://arxiv.org/abs/2503.05587) find
  52. Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024. Large language models for data annotation: A survey. arXiv preprint arXiv:2402.13446. — [arXiv:2402.13446](https://arxiv.org/abs/2402.13446) find
  53. Yuan Yang, Siheng Xiong, Ali Payani, Ehsan Shareghi, and Faramarz Fekri. 2023b. Harnessing the power of large language models for natural language to first-order logic translation. arXiv preprint arXiv:2305.15541. find
  54. Zhengzhe Yang and Jinho D. Choi. 2019. FriendsQA: Open-Domain Question Answering on TV Show Transcripts. In Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue, pages 188– 197, Stockholm, Sweden. Association for Computational Linguistics. find
  55. Mo Yu, Jiangnan Li, Shunyu Yao, Wenjie Pang, Xiaochen Zhou, Zhou Xiao, Fandong Meng, and Jie Zhou. 2023a. Personality Understanding of Fictional Characters during Book Reading. arXiv preprint. ArXiv:2305.10156 [cs]. find
  56. Mo Yu, Jiangnan Li, Shunyu Yao, Wenjie Pang, Xiaochen Zhou, Zhou Xiao, Fandong Meng, and Jie Zhou. 2023b. Personality understanding of fictional characters during book reading. arXiv preprint arXiv:2305.10156. find
  57. Mo Yu, Qiujing Wang, Shunchi Zhang, Yisi Sang, Kangsheng Pu, Zekai Wei, Han Wang, Liyan Xu, Jing Li, Yue Yu, et al. 2022. Few-shot character understanding in movies as an assessment to meta-learning of theory-of-mind. arXiv preprint arXiv:2211.04684. — [arXiv:2211.04684](https://arxiv.org/abs/2211.04684) find
  58. Yijiong Yu. 2025. Do llms really think step-by-step in implicit reasoning? Preprint, arXiv:2411.15862.
  59. Hengyuan Zhang, Shiping Yang, Xiao Liang, Chenming Shang, Yuxuan Jiang, Chaofan Tao, Jing Xiong, Hayden Kwok-Hay So, Ruobing Xie, Angel X Chang, et al. 2025. Find your optimal teacher: Personalized data synthesis via router-guided multi-teacher distillation. arXiv preprint arXiv:2510.10925.
  60. Kun Zhou, Yutao Zhu, Zhipeng Chen, Wentong Chen, Wayne Xin Zhao, Xu Chen, Yankai Lin, Ji-Rong Wen, and Jiawei Han. 2023. Don’t Make Your LLM an Evaluation Benchmark Cheater. arXiv preprint. ArXiv:2311.01964. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jiang-2026-beyond-math-stories-as-testbed/