Elamur Research⌕
☰
home / papers / 2026 / Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understanding

Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understanding

Paper · 2026 · Adam Štorek, Mukur Gupta, Samira Hajizadeh et al.

Summary

LLMs are increasingly used to reason over large codebases. It is unclear, however, whether they track what the supplied code actually does or fall back on familiar patterns absorbed during pretraining. The authors separate two abilities. Lexical recall is reproducing in-context code exactly. Semantic recall is grasping the code's operational semantics, meaning its behaviour at runtime.

They test 10 state-of-the-art LLMs and use the position of a target function among unrelated distractor functions as a controlled probe. Lexical recall stays nearly perfect wherever the snippet sits. Semantic recall, by contrast, shows a strong lost-in-the-middle decline.

They also define semantic recall sensitivity: how strongly a task forces a model to use the specific code rather than shortcuts. They estimate it counterfactually by deleting code lines step by step. By this measure CRUXEval is only weakly sensitive. The authors therefore build SemTrace, an output-prediction task built on unpredictable arithmetic operations, where positional degradation becomes far sharper.

The main implication is that current benchmarks understate how fragile long-context code understanding is, which matters for novel, project-specific code and for spotting subtle vulnerabilities.

Key points

  • On CRUXEval, the median accuracy drop as the relevant snippet moves toward the middle is 53.36% for semantic recall but only 2.39% for lexical recall. The lexical–semantic dissociation holds for Python, JavaScript and PHP.
  • In the line-removal test, CRUXEval accuracy falls only gradually, losing 44.15–59.74% when half the lines are deleted. This gradual decline signals pattern matching, whereas a real interpreter would fail abruptly.
  • On SemTrace the median positional accuracy drop reaches 92.73%, and some frontier models hit zero accuracy when the code sits in the centre.
  • GPT-4.1 looks robust on 2- and 3-digit SemTrace, which the authors attribute to memorized arithmetic. At 4 digits it loses up to 31 points, and at 6 digits it shows up to 43% relative loss with a U-shaped curve (80 distractors, about 16k tokens).
  • Setup limits: 800 examples per task, contexts up to about 16k tokens, and semantically unrelated distractors. Code is available at github.com/adamstorek/long-context-code-understanding.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Columbia UniversityNew York · United States241 works · 54,053 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 worksSoftware · 92 works

Cite

Cite as Adam Štorek, Mukur Gupta, Samira Hajizadeh, Prashast Srivastava and Suman Jana. 2026. Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understanding. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.19

@inproceedings{storek2026sense,
  title = {{Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Understanding}},
  author = {Adam Štorek, Mukur Gupta, Samira Hajizadeh, Prashast Srivastava and Suman Jana},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.19},
  url = {https://doi.org/10.18653/v1/2026.acl-long.19}
}

References

44 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2024. LongBench: A bilingual, multi-task benchmark for long context understanding. In Proceedings of the 62nd Annual Meeting of the Associat… find
  2. Yushi Bai, Shangqing Tu, Jiajie Zhang, Hao Peng, Xiaozhi Wang, Xin Lv, Shulin Cao, Jiazheng Xu, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2025. LongBench v2: Towards deeper understanding and reasoning on realistic long-context multitasks. Preprint, arXiv:2412.15204.
  3. Kent Chang, Mackenzie Cramer, Sandeep Soni, and David Bamman. 2023. Speak, memory: An archaeology of books known to ChatGPT/GPT-4. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7312–7327, Singapore. Association for Computational Linguistics.
  4. Tri Dao, Daniel Y Fu, Stefano Ermon, Atri Rudra, and Christopher Re. 2022. FlashAttention: Fast and memory-efficient exact attention with IO-awareness. In Advances in Neural Information Processing Systems. find
  5. Yangruibo Ding, Yanjun Fu, Omniyyah Ibrahim, Chawin Sitawarin, Xinyun Chen, Basel Alomair, David Wagner, Baishakhi Ray, and Yizheng Chen. 2025. Vulnerability detection with code language models: How far are we? In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pages 1729… find
  6. Benj Edwards. 2024. Google CEO says over 25% of new Google code is generated by AI. find
  7. Lizhe Fang, Yifei Wang, Khashayar Gatmiry, Lei Fang, and Yisen Wang. 2025. Rethinking invariance in in-context learning. In The Thirteenth International Conference on Learning Representations. find
  8. Muhan Gao, TaiMing Lu, Kuai Yu, Adam Byerly, and Daniel Khashabi. 2024. Insights into LLM long-context failures: When transformers know but don’t tell. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7611–7625, Miami, Florida, USA. Association for Computational Lingui… find
  9. Alex Gu, Baptiste Roziere, Hugh James Leather, Armando Solar-Lezama, Gabriel Synnaeve, and Sida Wang. 2024. CRUXEval: A benchmark for code reasoning, understanding and execution. In Forty-first International Conference on Machine Learning.
  10. Xiangming Gu, Tianyu Pang, Chao Du, Qian Liu, Fengzhuo Zhang, Cunxiao Du, Ye Wang, and Min Lin. 2025. When attention sink emerges in language models: An empirical view. In The Thirteenth International Conference on Learning Representations. find
  11. Daya Guo, Canwen Xu, Nan Duan, Jian Yin, and Julian McAuley. 2023. LongCoder: a long-range pre-trained language model for code completion. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org. find
  12. Valentin Hartmann, Anshuman Suri, Vincent Bindschaedler, David Evans, Shruti Tople, and Robert West. 2023. SoK: Memorization in general-purpose large language models. Preprint, arXiv:2310.18362. — [arXiv:2310.18362](https://arxiv.org/abs/2310.18362) find
  13. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, and Boris Ginsburg. 2024. RULER: What’s the real context size of your long-context language models? In First Conference on Language Modeling.
  14. Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065, Toronto, Canada. Association for Computational Linguistics.
  15. Jiawei Liu, Jia Le Tian, Vijay Daita, Yuxiang Wei, Yifeng Ding, Yuhan Katherine Wang, Jun Yang, and Lingming Zhang. 2024a. RepoQA: Evaluating long context code understanding. In First Workshop on Long-Context Foundation Models @ ICML 2024. find
  16. Kaixuan Huang, Jiacheng Guo, Zihao Li, Xiang Ji, Jiawei Ge, Wenzhe Li, Yingqing Guo, Tianle Cai, Hui Yuan, Runzhe Wang, Yue Wu, Ming Yin, Shange Tang, Yangsibo Huang, Chi Jin, Xinyun Chen, Chiyuan Zhang, and Mengdi Wang. 2025. MATH-perturb: Benchmarking LLMs’ math reasoning abilities against hard pe… find
  17. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024b. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157–173.
  18. Hamel Husain, Ho-Hsiang Wu, Tiferet Gazit, Miltiadis Allamanis, and Marc Brockschmidt. 2019. Code-SearchNet challenge: Evaluating the state of semantic code search. arXiv preprint arXiv:1909.09436.
  19. Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. 2024. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations.
  20. William Jurayj, Nils Holzenberger, and Benjamin Van Durme. 2026. Language models and logic programs for trustworthy tax reasoning. Proceedings of the AAAI Conference on Artificial Intelligence, 40(45):38688–38698. find
  21. Greg Kamradt. 2023. Needle In A Haystack - Pressure Testing LLMs. find
  22. Yuri Kuratov, Aydar Bulatov, Petr Anokhin, Ivan Rodkin, Dmitry Igorevich Sorokin, Artyom Sorokin, and Mikhail Burtsev. 2024. BABILong: Testing the limits of LLMs with long context reasoning-in-a-haystack. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks… find
  23. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with PagedAttention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles.
  24. Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same task, more tokens: the impact of input length on the reasoning performance of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15339–15353, Bangkok, T…
  25. Dacheng Li, Rulin Shao, Anze Xie, Ying Sheng, Lianmin Zheng, Joseph E. Gonzalez, Ion Stoica, Xuezhe Ma, and Hao Zhang. 2023. How long can open-source LLMs truly promise on context length? find
  26. Tianyang Liu, Canwen Xu, and Julian McAuley. 2024c. RepoBench: Benchmarking repository-level code auto-completion systems. In The Twelfth International Conference on Learning Representations. find
  27. Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
  28. Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 11048–11064, Ab…
  29. Ali Modarressi, Hanieh Deilamsalehy, Franck Dernoncourt, Trung Bui, Ryan A. Rossi, Seunghyun Yoon, and Hinrich Schütze. 2025. NoLiMa: Long-context evaluation beyond literal matching. Preprint, arXiv:2502.05167. — [arXiv:2502.05167](https://arxiv.org/abs/2502.05167) find
  30. Amirkeivan Mohtashami and Martin Jaggi. 2023. Random-access infinite context length for transformers. In Thirty-seventh Conference on Neural Information Processing Systems. find
  31. Neel Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt. 2023. Progress measures for grokking via mechanistic interpretability. In The Eleventh International Conference on Learning Representations.
  32. OpenAI. 2025. Introducing GPT-4.1 in the API. find
  33. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, and Enrico Shippole. 2024. YaRN: Efficient context window extension of large language models. In The Twelfth International Conference on Learning Representations. find
  34. Uri Shaham, Maor Ivgi, Avia Efrat, Jonathan Berant, and Omer Levy. 2023. ZeroSCROLLS: A zero-shot benchmark for long text understanding. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7977–7989, Singapore. Association for Computational Linguistics.
  35. Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. 2024. RoFormer: Enhanced transformer with rotary position embedding. Neurocomput., 568(C).
  36. Glynn Winskel. 1993. The formal semantics of programming languages: an introduction. MIT Press, Cambridge, MA, USA. find
  37. Xinyi Wu, Yifei Wang, Stefanie Jegelka, and Ali Jadbabaie. 2025. On the emergence of position bias in transformers. In Forty-second International Conference on Machine Learning. find
  38. Zhaofeng Wu, Linlu Qiu, Alexis Ross, Ekin Akyürek, Boyuan Chen, Bailin Wang, Najoung Kim, Jacob Andreas, and Yoon Kim. 2024. Reasoning or reciting? exploring the capabilities and limitations of language models through counterfactual tasks. In Proceedings of the 2024 Conference of the North American… find
  39. Ruiyang Xu, Jialun Cao, Yaojie Lu, Ming Wen, Hongyu Lin, Xianpei Han, Ben He, Shing-Chi Cheung, and Le Sun. 2025. CRUXEVAL-X: A benchmark for multilingual code reasoning, understanding and execution. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1… find
  40. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 23 others. 2025a. Qwen2.5 technical report. Preprint, arXiv:24… find
  41. Rem Yang, Julian Dai, Nikos Vasilakis, and Martin Rinard. 2025b. Evaluating the generalization capabilities of large language models on code reasoning. Preprint, arXiv:2504.05518. — [arXiv:2504.05518](https://arxiv.org/abs/2504.05518) find
  42. Xinrong Zhang, Yingfa Chen, Shengding Hu, Zihang Xu, Junhao Chen, Moo Hao, Xu Han, Zhen Thai, Shuo Wang, Zhiyuan Liu, and Maosong Sun. 2024a. ∞Bench: Extending long context evaluation beyond 100K tokens. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volu… find
  43. Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, and Zhangyang Wang. 2024b. Found in the middle: How language models use long contexts better via plug-and-play positional encoding. In The Thirty-eighth Annual Conference on Neural Information Processing Syst… find
  44. Model DeepSeek Coder V2 Lite Gemma 3 12B Llama 3.1 8B Qwen 2.5 Coder 7B CRUXEval-I/O SemTrace 20.00/21.00 39.50/19.25 19.50/32.62 31.62/46.62 4.12 15.88 16.00 22.25 *Table 2: Baseline performance (% accuracy) without distractors. CRUXEval-I/O shows input/output prediction accuracy. Smaller models ge… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/storek-2026-sense-sensitivity-examining-influence-semantic/