Elamur Research⌕
☰
home / papers / 2026 / Ghost Context: Measuring Cross-Context Interference in Long-Context Language Models

Ghost Context: Measuring Cross-Context Interference in Long-Context Language Models

Paper · 2026 · Rohith Namboothiri

Summary

Long-context LLMs in RAG pipelines and agents receive prompts assembled from many sources, and the system assumes the model will rely on the span that answers the query. This paper identifies a failure the author calls Ghost Context. An irrelevant span causally shapes the output, producing "misattributed grounding": a claim backed by genuine evidence, but from the wrong source. Such claims are entailed by something in the prompt, so source-blind faithfulness checks (e.g., FActScore, SummaC, MiniCheck) cannot catch them, as the paper argues formally. To measure the effect, the author builds a 272-case corpus of custom fictional cases plus items adapted from MuSiQue, HotpotQA and SituatedQA. Each distractor span is masked, the model is rerun with greedy decoding, and two LLM judges compare the outputs. Three models from three families are tested. The phenomenon is uncommon overall but rises sharply when contexts conflict, especially when they contain outdated versions of a fact. It is also cheap to fix: removing the single top-attributed span corrects nearly all detected cases. The protocol gives RAG and agent builders a practical audit tool and new robustness metrics.

Key points

  • Metrics and taxonomy. The paper proposes four metrics:

    • Ghost Context Rate (GCR)
    • Fix@k
    • Collateral Damage Rate (CDR)
    • Contextual Invariance Rate (CIR)

    It also defines four interference patterns: Temporal Bleed, Cross-Document Fusion, Distractor Dominance, and Instruction-Data Interference.

  • Prevalence. Strict GCR is only 3.2% across all cases. Temporal contradictions push it to 38.3% on the most susceptible model, and open-ended (non-strictly-factual) distractor influence reaches 20.4%.

  • Remediation. Masking the single highest-attributed distractor fixes 95.5% of detected errors (Fix@1). This comes with 2.4% collateral damage and no false positives on negative controls.

  • Judging setup. The primary judge is Claude Sonnet 4.6, cross-checked by an unrelated second judge; raw agreement between the two is 97–99%. One tested model, Claude Haiku 4.5, shares a family with the primary judge. No human annotation study was run.

  • Limitations and release. Contexts are short (about 479 words) compared with production RAG contexts of 5K–50K tokens, and the security implications are discussed but not tested. The corpus, pipeline and results are publicly released.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Rohith Namboothiri. 2026. Ghost Context: Measuring Cross-Context Interference in Long-Context Language Models. Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.trustnlp-main.19

@inproceedings{namboothiri2026ghost,
  title = {{Ghost Context: Measuring Cross-Context Interference in Long-Context Language Models}},
  author = {Rohith Namboothiri},
  booktitle = {Proceedings of the 6th Workshop on Trustworthy NLP (TrustNLP 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.trustnlp-main.19},
  url = {https://doi.org/10.18653/v1/2026.trustnlp-main.19}
}

References

46 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In ICLR.
  2. Scott Barnett, Stefanus Kurniawan, Srikanth Thudumu, Zach Brannelly, and Mohamed Abdelrazek. 2024. Seven failure points when engineering a retrieval augmented generation system. arXiv:2401.05856. — [arXiv:2401.05856](https://arxiv.org/abs/2401.05856) find
  3. Bernd Bohnet, Vinh Q. Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Jacob Eisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Liber Osber, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Metzler, and 2 others. 2022. Attr… — Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
  4. Cheng-Han Chiang and Hung yi Lee. 2023. Can large language models be an alternative to human evaluations? arXiv:2305.01937. find
  5. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2024. RAGAS: Automated evaluation of retrieval augmented generation. In EACL System Demonstrations, pages 150–163.
  6. Alvan R. Feinstein and Domenic V. Cicchetti. 1990. High agreement but low kappa: I. the problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549. find
  7. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. In EMNLP, pages 6465–6488.
  8. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-augmented generation for large language models: A survey. arXiv:2312.10997.
  9. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world LLM-integrated applications with indirect prompt injection. In AISec.
  10. Or Honovich, Leshem Choshen, Roee Aharoni, Ella Neeman, Idan Szpektor, and Omri Abend. 2022. TRUE: Re-evaluating factual consistency evaluation. In NAACL, pages 3905–3920.
  11. Cheng-Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang, and Boris Ginsburg. 2024. RULER: What’s the real context size of your long-context language models? arXiv:2404.06654.
  12. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv:2311.05232.
  13. Sarthak Jain and Byron C. Wallace. 2019. Attention is not explanation. In NAACL, pages 3543–3556.
  14. Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2023. Can large language models truly follow your instructions? In EMNLP Findings. find
  15. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38.
  16. Jieun Kim and 1 others. 2025. Semantic inversion, identical replies: Revisiting negation blindness in large language models. In EMNLP. find
  17. Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In ICML, pages 1885–1894.
  18. Helen Nissenbaum. 2004. Privacy as contextual integrity. Washington Law Review, 79(1):119–158. find
  19. Philippe Laban, Tobias Schnabel, Paul N. Bennett, and Marti A. Hearst. 2022. SummaC: Re-visiting NLI-based models for inconsistency detection in summarization. In TACL, volume 10, pages 163–177. find
  20. J. Richard Landis and Gary G. Koch. 1977. The measurement of observer agreement for categorical data. Biometrics, 33(1):159–174. find
  21. Mosh Levy, Alon Jacoby, and Yoav Goldberg. 2024. Same task, more tokens: the impact of input length on the reasoning performance of large language models. arXiv:2402.14848.
  22. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In NeurIPS, volume 33, pages 9459–9474…
  23. Mo Li, Songyang Zhang, Yunxin Liu, and Kai Chen. 2024. NeedleBench: Can LLMs do retrieval and reasoning in 1 million context window? arXiv:2407.11963. — [arXiv:2407.11963](https://arxiv.org/abs/2407.11963) find
  24. Feiyang Liu, Nikhil Kandpal, and Colin Raffel. 2024a. AttriBoT: A bag of tricks for efficiently approximating leave-one-out context attribution. arXiv:2411.15102. find
  25. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024b. Lost in the middle: How language models use long contexts. TACL, 12:157–173.
  26. Potsawee Manakul, Adian Liusie, and Mark J.F. Gales. 2023. SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models. arXiv:2303.08896. find
  27. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023a. Generative agents: Interactive simulacra of human behavior. In UIST. find
  28. Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry. 2023b. TRAK: Attributing model behavior at scale. In ICML. find
  29. Judea Pearl. 2009. Causality, 2nd edition. Cambridge University Press. find
  30. Fabián Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. In NeurIPS ML Safety Workshop.
  31. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. In ICML, pages 31210–31227.
  32. Theodore R. Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths. 2024. Cognitive architectures for language agents. arXiv:2309.02427. find
  33. Liyan Tang, Philippe Laban, and Greg Durrett. 2024a. MiniCheck: Efficient fact-checking of LLMs on grounding documents. In EMNLP.
  34. Yi Tang and 1 others. 2024b. MultiHop-RAG: Benchmarking retrieval-augmented generation for multi-hop queries. arXiv:2401.15391.
  35. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi-hop questions via single hop question composition. TACL, 10:539–554. find
  36. Van Truong and 1 others. 2025. A comprehensive taxonomy of negation for NLP and LLMs. arXiv:2507.22337. — [arXiv:2507.22337](https://arxiv.org/abs/2507.22337) find
  37. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. In ACL, pages 1906–1919.
  38. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NeurIPS, pages 5998–6008.
  39. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In EMNLP, pages 12076–12100.
  40. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nishi, Jason Ahn, Daniel Jurafsky, and Bernd Bohnet. 2020. Investigating gender bias in language models using causal mediation analysis. In NeurIPS. find
  41. Christoph Molnar. 2022. Interpretable Machine Learning: A Guide for Making Black Box Models Explainable, 2nd edition. find
  42. Mohsan Nadeem and 1 others. 2024. Negation blindness in large language models: Unveiling the NO syndrome in image generation. arXiv:2409.00105. — [arXiv:2409.00105](https://arxiv.org/abs/2409.00105) find
  43. Sarah Wiegreffe and Yuval Pinter. 2019. Attention is not not explanation. In EMNLP, pages 11–16.
  44. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP, pages 2369–2380.
  45. Jingwei Yi, Yueqi Xie, Bin Zhu, Keegan Hines, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2023. Benchmarking and defending against indirect prompt injection attacks on large language models. arXiv:2312.14197. — [arXiv:2312.14197](https://arxiv.org/abs/2312.14197) find
  46. Qiusi Zhan, Zhixiang Liang, Zifan Ying, and Daniel Kang. 2024. InjecAgent: Benchmarking indirect prompt injections in tool-integrated large language model agents. arXiv:2403.02691.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/namboothiri-2026-ghost-context-measuring-cross-context/