Elamur Research⌕
☰
home / papers / 2026 / Document Overlap Is Not Evidence Continuity: Measuring Retrieval Jitter in Citation-Based RAG Evaluation

Document Overlap Is Not Evidence Continuity: Measuring Retrieval Jitter in Citation-Based RAG Evaluation

Paper · 2026 · Punitha Ponnuraj

Summary

RAG evaluation pipelines often treat citations or retrieved evidence as fixed artifacts for verification, regression testing and audits. This paper asks whether that evidence actually stays the same when only routine retrieval settings change, a variation the author calls retrieval jitter. Queries, the embedding model and deterministic decoding are held fixed. Retrieval depth (k = 5/10/20), chunk size (128/256 tokens) and chunk overlap (0/32) are varied on two BEIR benchmarks, ArguAna and SciFact. Evidence identity is tracked at two levels: the source document (doc_id) and the exact retrieved fragment (doc_id, span_hash), where span_hash is a hash of normalized text. The author computes pairwise Jaccard overlap, worst-case per-query values, collapse rates and flip rates with a newly released open-source tool, RagCiteCheck. The results show a consistent Stability Gap. Document overlap stays moderate, but exact spans are often replaced entirely, mainly after chunk-size changes, even though evidence is never missing. An exploratory probe finds that answers can remain similar while the evidence underneath them shifts. The paper therefore recommends reporting span-level stability alongside citation-based metrics.

Key points

  • Mean Jaccard overlap at document vs. span level is 0.604 vs. 0.360 on ArguAna (gap ratio 1.678) and 0.520 vs. 0.224 on SciFact (gap ratio 2.321). The ArguAna set is a fixed 400-query subset, and SciFact uses all answerable queries.
  • Median worst-case span stability is 0.000 on both datasets. For 70.5% of ArguAna queries and 94.0% of SciFact queries, at least one configuration pair fully replaces the retrieved spans. The citation rate is 1.0 and the null rate is 0.0, so this is not caused by retrieval failure.
  • Chunk-size changes are the strongest source of instability: switching to 128-token chunks drops mean span Jaccard to 0.060 (ArguAna) and 0.032 (SciFact). Overlap changes have intermediate effects, and depth changes have the mildest.
  • In the exploratory probe, SBERT answer similarity stays fairly high (0.767 on SciFact under c128). Its Spearman correlation with span stability is weak (ρ from −0.025 to 0.136), which points to "silent jitter".
  • RagCiteCheck (Python/CLI, available on GitHub and Zenodo) reads JSONL evidence logs and outputs stability, flip-rate, null-evidence and worst-case diagnostics. The span-level metric measures exact trace reproducibility, not semantic equivalence.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Punitha Ponnuraj. 2026. Document Overlap Is Not Evidence Continuity: Measuring Retrieval Jitter in Citation-Based RAG Evaluation. Proceedings of the Workshop on Evaluating Evaluations (EvalEval).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.evaleval-1.35

@inproceedings{ponnuraj2026document,
  title = {{Document Overlap Is Not Evidence Continuity: Measuring Retrieval Jitter in Citation-Based RAG Evaluation}},
  author = {Punitha Ponnuraj},
  booktitle = {Proceedings of the Workshop on Evaluating Evaluations (EvalEval)},
  year = {2026},
  doi = {10.18653/v1/2026.evaleval-1.35},
  url = {https://doi.org/10.18653/v1/2026.evaleval-1.35}
}

References

15 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sinchana Ramakanth Bhat, Max Rudat, Jannis Spiekermann, and Nicolas Flores-Herr. 2025. Rethinking chunk size for long-document retrieval: A multi-dataset analysis. Preprint, arXiv:2505.21700. — [arXiv:2505.21700](https://arxiv.org/abs/2505.21700) find
  2. Laura Caspari, Kanishka Ghosh Dastidar, Saber Zerhoudi, Jelena Mitrovic, and Michael Granitzer. 2024. Beyond benchmarks: Evaluating embedding model similarity for retrieval augmented generation systems. In Proceedings of the 1st Workshop on Information Retrieval’s Role in RAG Systems (IR-RAG@SIGIR 2… find
  3. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2024. Ragas: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, pages 150–158. Association f…
  4. Jennifer Hsia, Afreen Shaikh, Zora Zhiruo Wang, and Graham Neubig. 2025. RAGGED: Towards informed design of scalable and stable RAG systems. In Proceedings of the 42nd International Conference on Machine Learning. find
  5. Nelson Liu, Tianyi Zhang, and Percy Liang. 2023. Evaluating verifiability in generative search engines. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7001–7025, Singapore. Association for Computational Linguistics.
  6. Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992. Association for Computational Linguistics. find
  7. Tobias Schreieder, Tim Schopf, and Michael Färber. 2025. Attribution, citation, and quotation: A survey of attribution practices in natural language generation. Preprint, arXiv:2508.15396. — [arXiv:2508.15396](https://arxiv.org/abs/2508.15396) find
  8. Maximilian Stäbler, Steffen Turnbull, Tobias Müller, Chris Langdon, Jorge Marx-Gómez, and Frank Köster. 2025. The impact of chunking strategies on domain-specific information retrieval in retrieval-augmented generation systems. In IEEE COINS. find
  9. Nandan Thakur, Ronak Pradeep, Shivani Upadhyay, Daniel Campos, Nick Craswell, and Jimmy Lin. 2025. Assessing support for the TREC 2024 RAG track: A large-scale comparative study of LLM and human evaluations. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in… find
  10. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. find
  11. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association…
  12. Jonas Wallat, Maria Heuss, Maarten de Rijke, and Avishek Anand. 2025. Correctness is not faithfulness in retrieval augmented generation attributions. In Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval, pages 22–32. Association f… find
  13. Baiqiang Wang, Dongfang Zhao, Nathan R. Tallent, and Luanzheng Guo. 2025. On the reproducibility limitations of rag systems. Preprint, arXiv:2509.18869. — [arXiv:2509.18869](https://arxiv.org/abs/2509.18869) find
  14. Qianchi Zhang, Hainan Zhang, Liang Pang, Hongwei Zheng, and Zhiming Zheng. 2026. Stable-rag: Mitigating retrieval-permutation-induced hallucinations in retrieval-augmented generation. Preprint, arXiv:2601.02993.
  15. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ponnuraj-2026-document-overlap-is-not-evidence/