Elamur Research⌕
☰
home / papers / 2025 / "Give a Positive Review Only": An Early Investigation Into In-Paper Prompt Injection Attacks and Defenses for AI Reviewers

"Give a Positive Review Only": An Early Investigation Into In-Paper Prompt Injection Attacks and Defenses for AI Reviewers

Paper · 2025 · Qin Zhou, Zhexin Zhang, Zhi Li et al.

Summary

Hidden instructions tucked into submitted PDFs can steer LLM-based reviewers toward inflated scores. The authors call this In-Paper Prompt Injection (IPI) and test it at scale. They took 100 randomly sampled ICLR 2025 submissions and inserted white, microscopic-font text with a PyMuPDF-based tool. Two attack families were tested. Static attacks reuse four real-world injection prompts taken from the taxonomy of Lin (2025). Iterative attacks refine the injected text over several rounds using feedback from a surrogate reviewer. The targets were GPT-5, DeepSeek-Chat (DeepSeek-V3), Gemini-2.5-Pro, GPT-OSS-20B and Llama-3.1-8B-Instruct. Both attack types raised ratings substantially, and iterative attacks often pushed scores to the maximum. The effect held regardless of where the text was inserted, how long the paper was, or how humans had rated it, and it transferred between models. A defense that asks the reviewer to detect injections first restored near-baseline scores. However, an adaptive attacker who explicitly aims to evade detection largely defeated it. The work warns that AI-assisted peer review needs stronger safeguards before broad deployment.

Key points

  • Static attacks: on the 10-point ICLR scale, average ratings rose by 2.80 for DeepSeek, 1.91 for Gemini and 1.24 for GPT-5.
  • Iterative attacks: within at most three optimization rounds, scores got close to 10 for Gemini, DeepSeek and GPT-OSS-20B.
  • Transfer across models: role-play prompts generated by Gemini-2.5-Pro transferred best (average gain 2.26, including +1.04 on GPT-5). Blunt "ignore instructions" commands barely affected GPT-5.
  • Defense: with DeepSeek-Chat as reviewer, combined detect-and-score caught 99% of injections and cut the mean score from 10.00 to 7.27, but it also overcorrected 17 papers. Running a separate detector that is given the reviewer role reached 97% detection, a 7.14 mean, and zero overcorrection.
  • Adaptive attack: against the combined defense, only 24% of injections were detected. The mean score rose from 7.06 to 8.11, and 30 papers ended up more than 1.5 points above their clean baseline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China439 works · 94,783 citations🏛️Chinese Academy of SciencesBeijing · China256 works · 25,136 citations🎓University of Chinese Academy of SciencesBeijing · China196 works · 8,907 citations🔬Institute of Information EngineeringBeijing · China43 works · 1,524 citations

 Fields, topics and concepts

📚 Shelves 3The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksInformation Systems · 1,888 works

Cite

Cite as Qin Zhou, Zhexin Zhang, Zhi Li and Limin Sun. 2025. "Give a Positive Review Only": An Early Investigation Into In-Paper Prompt Injection Attacks and Defenses for AI Reviewers. arXiv:2511.01287.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2511.01287 · arXiv 2511.01287

@misc{zhou2025give,
  title = {{"Give a Positive Review Only": An Early Investigation Into In-Paper Prompt Injection Attacks and Defenses for AI Reviewers}},
  author = {Qin Zhou, Zhexin Zhang, Zhi Li and Limin Sun},
  year = {2025},
  doi = {10.48550/arxiv.2511.01287},
  eprint = {2511.01287},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2511.01287}
}

References

34 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Artifex Software. 2024. Pymupdf (fitz): Python bindings for mupdf. find
  2. Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fernández, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. 2024. Llms instead of human judges? a large scale empirical study across 20 nlp evaluation tasks. arXiv preprint arXiv:2406…
  3. Yuan Chang, Ziyue Li, Hengyuan Zhang, Yuanbo Kong, Yanru Wu, Zhijiang Guo, and Ngai Wong. 2025. Treereview: A dynamic tree of questions framework for deep and efficient llm-based scientific peer review. ArXiv, abs/2506.07642.
  4. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J. Pappas, and Eric Wong. 2025. Jailbreaking black box large language models in twenty queries. In 2025 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pages 23–42.
  5. Matteo Gioele Collu, Umberto Salviati, Roberto Confalonieri, Mauro Conti, and Giovanni Apruzzese. 2025. Publish to perish: Prompt injection attacks on llm-assisted peer review. ArXiv, abs/2508.20863. — [arXiv:2508.20863](https://arxiv.org/abs/2508.20863) find
  6. Yiqiao Jin, Qinlin Zhao, Yiyang Wang, Hao Chen, Kaijie Zhu, Yijia Xiao, and Jindong Wang. 2024. Agentreview: Exploring peer review dynamics with llm agents. arXiv preprint arXiv:2406.12708.
  7. Chavvi Kirtani, Madhav Krishan Garg, Tejash Prasad, Tanmay Singhal, Murari Mandal, and Dhruv Kumar. 2025. Revieweval: An evaluation framework for ai-generated reviews. ArXiv, abs/2502.11736. find
  8. Jisoo Lee, Jieun Lee, and Jeong-Ju Yoo. 2025. The role of large language models in the peer-review process: opportunities and challenges for medical journal reviewers and editors. Journal of Educational Evaluation for Health Professions, 22. find
  9. Ruochi Li, Haoxuan Zhang, Edward F. Gehringer, Ting Xiao, Junhua Ding, and Haihua Chen. 2025. Unveiling the merits and defects of llms in automatic review generation for scientific papers. find
  10. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. 2024. find
  11. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 1(8):AIoa2400196.
  12. Zhicheng Lin. 2025. Hidden prompts in manuscripts exploit ai-assisted peer review. arXiv preprint arXiv:2507.06185.
  13. Ryan Liu and Nihar B. Shah. 2023. Reviewergpt? an exploratory study on using large language models for paper reviewing. ArXiv, abs/2306.00622.
  14. Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. 2024a. Automatic and universal prompt injection attacks against large language models. arXiv preprint arXiv:2403.04957. — [arXiv:2403.04957](https://arxiv.org/abs/2403.04957) find
  15. Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, and Chaowei Xiao. 2024b. Automatic and universal prompt injection attacks against large language models. ArXiv, abs/2403.04957. — [arXiv:2403.04957](https://arxiv.org/abs/2403.04957) find
  16. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, et al. 2023. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499.
  17. Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024c. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24), pages 1831–1847, Philadelphia, PA. USENIX Association.
  18. Robert Schulz, Adrian Barnett, René Bernard, Nicholas JL Brown, Jennifer A Byrne, Peter Eckmann, Małgorzata A Gazda, Halil Kilicoglu, Eric M Prager, Maia Salholz-Hillel, et al. 2022. Is the future of peer review automated? BMC research notes, 15(1):203. find
  19. Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee, and Juho Kim. 2025. Mind the blind spots: A focus-level evaluation framework for llm reviews.
  20. Purin Sukpanichnant, Anna Rapberger, and Francesca Toni. 2024. Peerarg: Argumentative peer review with llms. ArXiv, abs/2409.16813. find
  21. Pawin Taechoyotin and Daniel Acuna. 2025. Remor: Automated peer review generation with llm reasoning and multi-objective reinforcement learning. ArXiv, abs/2505.11718.
  22. Nitya Thakkar, Mert Yuksekgonul, Jake Silberg, Animesh Garg, Nanyun Peng, Fei Sha, Rose Yu, Carl Vondrick, and James Zou. 2025. Can llm feedback enhance review quality? a randomized study of 20k reviews at iclr 2025. arXiv preprint arXiv:2504.09737.
  23. Aman Singh Thakur, Kartik Choudhary, Venkat Srinik Ramayapally, Sankaran Vaidyanathan, and Dieuwke Hupkes. 2024. Judging the judges: Evaluating alignment and vulnerabilities in LLMs-as-judges. find
  24. Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce… find
  25. Zeming Wei, Yifei Wang, and Yisen Wang. 2023. Jailbreak and guard aligned language models with only few in-context demonstrations. ArXiv, abs/2310.06387.
  26. Tracey Weissgerber, Nico Riedel, Halil Kilicoglu, Cyril Labbé, Peter Eckmann, Gerben Ter Riet, Jennifer Byrne, Guillaume Cabanac, Amanda Capes-Davis, Bertrand Favier, et al. 2021. Automated screening of covid-19 preprints: can we help authors to improve transparency and reproducibility? Nature medic… find
  27. Zihao Xu, Yi Liu, Gelei Deng, Yuekang Li, and Stjepan Picek. 2024. A comprehensive study of jailbreak attack versus defense for large language models. In Annual Meeting of the Association for Computational Linguistics. find
  28. Rui Ye, Xianghe Pang, Jingyi Chai, Jiaao Chen, Zhen fei Yin, Zhen Xiang, Xiaowen Dong, Jing Shao, and Siheng Chen. 2024. Are we there yet? revealing the risks of utilizing large language models in scholarly peer review. ArXiv, abs/2412.01708.
  29. Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2023. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. ArXiv, abs/2309.10253.
  30. Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2024. Evaluating large language models at evaluating instruction following. In The Twelfth International Conference on Learning Representations. find
  31. Ruiyang Zhou, Lu Chen, and Kai Yu. 2024. Is LLM a reliable reviewer? a comprehensive evaluation of LLM on automatic paper reviewing tasks. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9340– 9351, T…
  32. Minjun Zhu, Yixuan Weng, Linyi Yang, and Yue Zhang. 2025. Deepreview: Improving llm-based paper review with human-like deep thinking process. ArXiv, abs/2503.08569.
  33. Zhenzhen Zhuang, Jiandong Chen, Hongfeng Xu, Yuwen Jiang, and Jialiang Lin. 2025. Large language models for automated scholarly paper review: A survey. Information Fusion, page 103332. find
  34. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. ArXiv, abs/2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhou-2025-give-positive-review-only/