Elamur Research⌕
☰
home / papers / 2026 / Context-attended Adversarial Reinforcement Learning for Robust Multi-step Retrieval Augmented Generation

Context-attended Adversarial Reinforcement Learning for Robust Multi-step Retrieval Augmented Generation

Paper · 2026 · Yingtao Ren, Xiao Luo, Yu-Cheng Chang et al.

Summary

Multi-step RAG systems retrieve evidence over several rounds. The resulting long contexts gather irrelevant noise and can also contain deliberately planted misinformation. Earlier defenses such as fine-tuning on fixed noisy data, verifier agents, or self-reflective reasoning were mostly tested with a single retrieval step and benign noise. The authors present CARE (Context-attended Adversarial Reinforcement Learning), which treats robustness as a game between two players. An attacker built on a large LLM uses TextGrad to keep rewriting fake documents that look relevant, with the goal of lowering a reward model's correctness score. The generator sees real and poisoned documents mixed together and is trained with GRPO to recover the right answer. The two sides are updated in alternation. A curriculum sorts training questions by difficulty and moves from easy to hard to keep optimization from collapsing. On knowledge-intensive multi-hop QA benchmarks, CARE improves on robustness baselines in both answer accuracy and attack resistance, across models of different sizes and families. This matters for any RAG deployment that relies on untrusted retrieved text.

Key points

  • The authors describe this as the first method to alternate RL (GRPO) with TextGrad input-space optimization. The attacker refines its poisoned documents step by step to exploit the generator's current weaknesses.
  • Training data is built by running multi-step retrieval for seed questions and sampling an evaluation model K times per question. Each question's pass rate is compared with a threshold τ_easy to label it easy or hard, and the labeled questions populate three progressive stages (D_R1–D_R3).
  • When more poisoned documents are injected, backbone models reach attack success rates (ASR) of roughly 45–55%, and a "thinking" reasoning model is the most vulnerable. With CARE, ASR stays under 7% for counterfactual attacks and under 2% for poisoned ones.
  • In ablations, training only against static attacks sharply reduces defense ability, which supports the need for a dynamic attacker.
  • The authors list several limitations:
    • The retriever is assumed to be static.
    • Only context-injection attacks are covered; index manipulation and jailbreak-style prompt injection are not addressed.
    • Code is released at github.com/yingtaoren/CARE.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Wisconsin–MadisonMadison · United States42 works · 48,486 citations🎓University of Technology SydneySydney · Australia18 works · 12,833 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

Cite

Cite as Yingtao Ren, Xiao Luo, Yu-Cheng Chang and Chin-teng Lin. 2026. Context-attended Adversarial Reinforcement Learning for Robust Multi-step Retrieval Augmented Generation. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.856

@inproceedings{ren2026context,
  title = {{Context-attended Adversarial Reinforcement Learning for Robust Multi-step Retrieval Augmented Generation}},
  author = {Yingtao Ren, Xiao Luo, Yu-Cheng Chang and Chin-teng Lin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.856},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.856}
}

References

43 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Conference on Learning Representations.
  2. Chi-Min Chan, Chunpu Xu, Ruibin Yuan, Hongyin Luo, Wei Xue, Yike Guo, and Jie Fu. 2024. RQ-RAG: Learning to refine queries for retrieval augmented generation. In First Conference on Language Modeling. find
  3. Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2024. Benchmarking large language models in retrieval-augmented generation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17754–17762.
  4. Zhuo Chen, Yuyang Gong, Jiawei Liu, Miaokun Chen, Haotan Liu, Qikai Cheng, Fan Zhang, Wei Lu, and Xiaozhong Liu. 2025. Flippedrag: Black-box opinion manipulation adversarial attacks to retrieval-augmented generation models. In Proceedings of the 2025 ACM SIGSAC Conference on Computer and Communicati… find
  5. Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on rag meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD conference on knowledge discovery and data mining, pages 6491– 6501.
  6. Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. 2024. Retrieval-augmented generation for large language models: A survey. Preprint, arXiv:2312.10997.
  7. Runpeng Geng, Yanting Wang, Ying Chen, and Jinyuan Jia. 2025. Unic-rag: Universal knowledge corruption attacks to retrieval-augmented generation. Preprint, arXiv:2508.18652. — [arXiv:2508.18652](https://arxiv.org/abs/2508.18652) find
  8. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Peiyi Wang, Qihao Zhu, Runxin Xu, Ruoyu Zhang, Shirong Ma, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, and 9 others. 2025. DeepSeek-R1 incentivizes reasoning in LLMs through reinfo… find
  9. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Comm…
  10. Wentao Hu, Wengyu Zhang, Yiyang Jiang, Chen Jason Zhang, Xiaoyong Wei, and Li Qing. 2025. Removal of hallucination on hallucination: Debate-augmented RAG. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vienna, Austria. Association…
  11. Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and 1 others. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems,…
  12. Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992.
  13. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan O Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025a. Search-r1: Training LLMs to reason and leverage search engines with reinforcement learning. In Second Conference on Language Modeling.
  14. Jiajie Jin, Yutao Zhu, Zhicheng Dou, Guanting Dong, Xinyu Yang, Chenghao Zhang, Tong Zhao, Zhao Yang, and Ji-Rong Wen. 2025b. Flashrag: A modular toolkit for efficient retrieval-augmented generation research. In Companion Proceedings of the ACM on Web Conference 2025, page 737–740, New York, NY, USA… find
  15. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association f…
  16. Jaehyung Kim, Jaehyun Nam, Sangwoo Mo, Jongjin Park, Sang-Woo Lee, Minjoon Seo, Jung-Woo Ha, and Jinwoo Shin. 2024. Sure: Summarizing retrievals using answer candidates for open-domain qa of llms. CoRR, abs/2404.13081. — [arXiv:2404.13081](https://arxiv.org/abs/2404.13081) find
  17. Ruochang Li, Xiao Luo, Zhiping Xiao, Wei Ju, and Ming Zhang. 2025a. HEAL: Hybrid enhancement with LLM-based agents for text-attributed hypergraph self-supervised representation learning. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 6815–6829, Suzhou, China. Associa… find
  18. Yangning Li, Weizhi Zhang, Yuyao Yang, Wei-Chieh Huang, Yaozu Wu, Junyu Luo, Yuanchen Bei, Henry Peng Zou, Xiao Luo, Yusheng Zhao, Chunkit Chan, Yankai Chen, Zhongfen Deng, Yinghui Li, Hai-Tao Zheng, Dongyuan Li, Renhe Jiang, Ming Zhang, Yangqiu Song, and Philip S. Yu. 2025b. Towards agentic rag wit…
  19. Xun Liang, Simin Niu, Zhiyu Li, Sensen Zhang, Hanyu Wang, Feiyu Xiong, Zhaoxin Fan, Bo Tang, Jihao Zhao, Jiawei Yang, Shichao Song, and Mengwei Wang. 2025. SafeRAG: Benchmarking security in retrieval-augmented generation of large language model. In Proceedings of the 63rd Annual Meeting of the Assoc… find
  20. Thang Nguyen, Peter Chin, and Yu-Wing Tai. 2025. Ma-rag: Multi-agent retrieval-augmented generation via collaborative chain-of-thought reasoning. Preprint, arXiv:2505.20096.
  21. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5687–5711, Singapore. Association for Computational Linguistics. find
  22. Zihan Qiu, Zekun Wang, Bo Zheng, Zeyu Huang, Kaiyue Wen, Songlin Yang, Rui Men, Le Yu, Fei Huang, Suozhi Huang, Dayiheng Liu, Jingren Zhou, and Junyang Lin. 2025. Gated attention for large language models: Non-linearity, sparsity, and attention-sink-free. In The Thirty-ninth Annual Conference on Neu… find
  23. Zhihong Shao, Yeyun Gong, yelong shen, Minlie Huang, Nan Duan, and Weizhu Chen. 2023. Enhancing retrieval-augmented large language models with iterative retrieval-generation synergy. In The 2023 Conference on Empirical Methods in Natural Language Processing.
  24. Zeyu Shen, Basileal Yoseph Imana, Tong Wu, Chong Xiang, Prateek Mittal, and Aleksandra Korolova. 2025. ReliabilityRAG: Effective and provably robust defense for RAG-based web-search. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  25. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554. find
  26. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p…
  27. Yiteng Tu, Weihang Su, Yujia Zhou, Yiqun Liu, and Qingyao Ai. 2025. Robust fine-tuning for retrieval augmented generation against retrieval defects. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’25, page 1272–1282, New York… find
  28. Fei Wang, Xingchen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan O Arik. 2025. Astute RAG: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pag…
  29. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2024. Text embeddings by weakly-supervised contrastive pre-training. Preprint, arXiv:2212.03533.
  30. Zhepei Wei, Wei-Lin Chen, and Yu Meng. 2025. InstructRAG: Instructing retrieval-augmented generation via self-synthesized rationales. In The Thirteenth International Conference on Learning Representations. find
  31. Peilin Wu, Mian Zhang, Xinlu Zhang, Xinya Du, and Zhiyu Chen. 2025. Search wisely: Mitigating suboptimal agentic searches by reducing uncertainty. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 19734–19745. find
  32. Yu Xia, Rui Wang, Xu Liu, Mingyan Li, Tong Yu, Xiang Chen, Julian McAuley, and Shuai Li. 2025. Beyond chain-of-thought: A survey of chain-of-X paradigms for LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 10795–10809, Abu Dhabi, UAE. Association for Comp…
  33. Shijia Xu, Zhou Wu, Xiaolong Jia, Yu Wang, Kai Liu, and April Xiaowen Dong. 2026. Self-correcting rag: Enhancing faithfulness via mmkp context selection and nli-guided mcts. Preprint, arXiv:2604.10734.
  34. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  35. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369…
  36. Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024a. Making retrieval-augmented language models robust to irrelevant context. In The Twelfth International Conference on Learning Representations.
  37. Ori Yoran, Tomer Wolfson, Ori Ram, and Jonathan Berant. 2024b. Making retrieval-augmented language models robust to irrelevant context. In The Twelfth International Conference on Learning Representations.
  38. Zhiyin Yu, Chao Zheng, Chong Chen, Xian-Sheng Hua, and Xiao Luo. 2025. scRAG: Hybrid retrieval-augmented generation for LLM-based cross-tissue single-cell annotation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 954– 970, Vienna, Austria. Association for Computationa… find
  39. Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Pan Lu, Zhi Huang, Carlos Guestrin, and James Zou. 2025. Optimizing generative AI by backpropagating language model feedback. Nature, 639(8055):609–616. find
  40. Baolei Zhang, Haoran Xin, Minghong Fang, Zhuqing Liu, Biao Yi, Tong Li, and Zheli Liu. 2025a. Traceback of poisoning attacks to retrieval-augmented generation. In Proceedings of the ACM on Web Conference 2025, WWW ’25, page 2085–2097, New York, NY, USA. Association for Computing Machinery. find
  41. Yufeng Zhang, Xuepeng Wang, Lingxiang Wu, and Jinqiao Wang. 2025b. Enhancing chain of thought prompting in large language models via reasoning patterns. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 25985– 25993. find
  42. Junda Zhu, Lingyong Yan, Haibo Shi, Dawei Yin, and Lei Sha. 2024. ATM: Adversarial tuning multi-agent system makes a robust retrieval-augmented generator. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 10902–10919, Miami, Florida, USA. Association fo… find
  43. Wei Zou, Runpeng Geng, Binghui Wang, and Jinyuan Jia. 2025. Poisonedrag: knowledge corruption attacks to retrieval-augmented generation of large language models. In Proceedings of the 34th USENIX Conference on Security Symposium, SEC ’25, USA. USENIX Association. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ren-2026-context-attended-adversarial-reinforcement-learning/