Elamur Research⌕
☰
home / papers / 2026 / TempPerturb-Eval: On the Joint Effects of Internal Temperature and External Perturbations in RAG Robustness

TempPerturb-Eval: On the Joint Effects of Internal Temperature and External Perturbations in RAG Robustness

Summary

RAG evaluations usually treat retrieval noise and decoding settings such as temperature as separate concerns, although in deployment they act together. This paper studies how the two interact. The authors treat the LLM as a black box and give it HotpotQA multi-hop questions (bridge and comparison types) whose supporting context is deliberately corrupted in three ways: sentence replacement, sentence removal, and named-entity substitution. An unperturbed baseline is kept for comparison. Each prompt is run three times at temperatures from 0.0 to 2.0. Outputs are scored against the reference with BERTScore and ROUGE, which captures both correctness and run-to-run variability. Both open-source and proprietary models are tested. The main finding is that higher temperatures make models more fragile to noisy context, and some perturbation types change quality non-linearly as temperature rises. Robustness measured at a single temperature can therefore be misleading. The work provides a reusable benchmark, a literature-based taxonomy of perturbation methods, and practical guidance for choosing models and tuning temperature when retrieval is imperfect.

Key points

  • Data: HotpotQA (about 113k Wikipedia-based QA pairs, CC BY-SA 4.0), split into bridge and comparison questions; the gold supporting sentences serve as the clean baseline context.
  • Experimental grid: three perturbations (sentence replacement, sentence removal, NER replacement), 11 temperature values (0.0–2.0, step 0.2) and three runs per sample, giving 440 experimental conditions across models and question types.
  • Metrics: BERTScore and ROUGE-1/2/L, reported as mean, variance and standard deviation over repeated runs, plus qualitative analysis.
  • Findings: high temperature consistently increases sensitivity to perturbed context, and certain perturbation types respond non-linearly across the temperature range.
  • Resources: code at github.com/yongxin2020/TempPerturb-Eval; all model inputs and generations on Hugging Face (yongxin2020/TempPerturb-Eval-data); the paper also includes a taxonomy table of IR/RAG perturbation methods (removal, reordering, entity/synonym/antonym replacement, noise and prefix injection, and others).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

1 paper of our library cites it (of the 140673 whose bibliographies we read); each has a page here.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3v28zfbo6agu

@inproceedings{zhou2026tempperturb,
  title = {{TempPerturb-Eval: On the Joint Effects of Internal Temperature and External Perturbations in RAG Robustness}},
  author = {Zhou, Yongxin and Mulhem, Philippe and Schwab, Didier},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/3v28zfbo6agu},
  url = {https://doi.org/10.63317/3v28zfbo6agu}
}

References

32 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Avishek Anand, Lijun Lyu, Maximilian Idahl, Yumeng Wang, Jonas Wallat, and Zijian Zhang. 2022. Explainable information retrieval: A survey. find
  2. Weihua Du, Yiming Yang, and Sean Welleck. 2025. Optimizing temperature for language models with multi-sample inference. In Forty-second International Conference on Machine Learning. find
  3. Feiteng Fang, Yuelin Bai, Shiwen Ni, Min Yang, Xiaojun Chen, and Ruifeng Xu. 2024. Enhancing noise robustness of retrieval-augmented language models with adaptive adversarial training. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),… find
  4. Gregory Goren, Oren Kurland, Moshe Tennenholtz, and Fiana Raiber. 2020. Ranking-incentivized quality preserving content modification. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 259–268, New York, NY, USA. Associ… find
  5. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on llm-as-a-judge. find
  6. Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. Open-Review.net. find
  7. Zhibo Hu, Chen Wang, Yanfeng Shu, Hye-Young Paik, and Liming Zhu. 2024. Prompt perturbation in retrieval-augmented generation based large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD ’24, page 1119–1130, New York, NY, USA. Association… find
  8. Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969– 7992, Singapore. Association f… find
  9. Jeongwoo Kang, Markarit Vartampetian, Felix Herron, Yongxin Zhou, Diandra Fabre, and Gabriela Gonzalez-Saez. 2025. Getalp@automin 2025: Leveraging rag to answer questions based on meeting transcripts. find
  10. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Proc… find
  11. Lujun Li, Lama Sleem, Niccolo’ Gentile, Geoffrey Nichil, and Radu State. 2025. Exploring the impact of temperature on large language models: Hot or cold? Procedia Computer Science, 264:242–251. International Neural Network Society Workshop on Deep Learning Innovations and Applications 2025. find
  12. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics. find
  13. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pre-training approach. find
  14. Yu-An Liu, Ruqing Zhang, Jiafeng Guo, Maarten de Rijke, Yixing Fan, and Xueqi Cheng. 2024. Robust neural information retrieval: An adversarial and out-of-distribution perspective. find
  15. Antonin Poché, Lucas Hervier, and Mohamed-Chafik Bakkay. 2023. Natural example-based explainability: A survey. In Explainable Artificial Intelligence, pages 24–47, Cham. Springer Nature Switzerland. find
  16. Nisarg Raval and Manisha Verma. 2020. One word at a time: adversarial attacks on retrieval models. find
  17. Matthew Renze. 2024. The effect of sampling temperature on problem solving in large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7346–7356, Miami, Florida, USA. Association for Computational Linguistics. find
  18. Joel Rorseth, Parke Godfrey, Lukasz Golab, Mehdi Kargar, Divesh Srivastava, and Jaroslaw Szlichta. 2023. Credence: Counterfactual explanations for document ranking. 2023 IEEE 39th International Conference on Data Engineering (ICDE), pages 3631–3634. find
  19. Joel Rorseth, Parke Godfrey, Lukasz Golab, Divesh Srivastava, and Jaroslaw Szlichta. 2024. Rage against the machine: Retrieval-augmented llm explanations. find
  20. Viju Sudhi, Sinchana Ramakanth Bhat, Max Rudat, and Roman Teucher. 2024. Rag-ex: A generic framework for explaining retrieval augmented generation. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’24, page 2776–2780, New York,… find
  21. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2014. Going deeper with convolutions. find
  22. Fei Wang, Xingchen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan O Arik. 2025. Astute RAG: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pag… find
  23. Chen Wu, Ruqing Zhang, Jiafeng Guo, Maarten De Rijke, Yixing Fan, and Xueqi Cheng. 2023. Prada: Practical black-box adversarial attacks against neural ranking models. ACM Trans. Inf. Syst., 41(4). find
  24. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  25. Yongxin Zhou, Philippe Mulhem, and Didier Schwab. 2025a. Explicabilité par Perturbations pour les Systèmes RAG. In Actes de l’atelier Accès à l’information basé sur le dialogue et grands modèles de langage 2025 (DIAG-LLM), pages 1–6, Marseille, France. ATALA and ARIA. find
  26. Yongxin Zhou, Fabien Ringeval, and François Portet. 2025b. Can GPT models follow human summarization guidelines? a study for targeted communication goals. In Proceedings of the 18th International Natural Language Generation Conference, pages 249–273, Hanoi, Vietnam. Association for Computational Lin… find
  27. Language Resource References Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural La… find our card
  28. Field Content Question New Faces of 1952 is a musical revue with songs and comedy skits, it helped jump start the career of which young performer, and American actress? Carol Lawrence Answer Supporting Facts New Faces of 1952 (sent 2) find
  29. It helped jump start the careers of several young performers including Paul Lynde, Alice Ghostley, Eartha Kitt, Robert Clary, Carol Lawrence, Ronny Graham, performer/writer Mel Brooks (as Melvin Brooks), and lyricist Sheldon Harnick. Carol Lawrence (born September 5, 1932) is an American actress, mo… find
  30. Carol Lawrence (sent 0) Example Distractor Context Guess Who I Saw Today (sent 0) Guess Who I Saw Today (sent 1) "Guess Who I Saw Today" is a popular jazz song written by Murray Grand with lyrics by Elisse Boyd. The song was originally composed for Leonard Sillman’s Broadway musical revue "New Faces… find
  31. Monotonous (sent 0) "Monotonous" is a popular song written by June Carroll and Arthur Siegel for Leonard Sillman’s Broadway revue "New Faces of 1952". The song was written based on the experiences of its singer Eartha Kitt. find
  32. Monotonous (sent 1) Table 2: A HotpotQA bridge question example (ID: 5a76a401554299373536012b) from our evaluation set, showing supporting facts and sample distractor context. find

↑ back to top

This page in Markdown →