Elamur Research⌕
☰
home / papers / 2024 / Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Review

Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Review

Paper · 2024 · Sungduk Yu, Man Luo, Avinash Madasu et al.

Summary

Growing submission volumes at AI conferences may tempt overloaded reviewers to quietly hand their work to LLMs, which undermines trust in peer review. This paper asks whether such reviews can be caught one at a time, rather than only estimated in aggregate. The authors pulled ICLR papers and human reviews through the OpenReview API. They then produced 16,000 synthetic reviews with GPT-4o and Llama-3.1 (70b) for 500 papers per year (2021–2024), varying the prompt across four reviewer personas and supplying ICLR reviewer guidelines. They benchmarked fine-tuned RoBERTa and Longformer detectors, the commercial Originality AI API, and LLM-as-a-judge setups. They also introduced an "anchor" method: an LLM writes its own review of the same paper, and a test review is flagged when its embedding is highly cosine-similar to that anchor. Existing tools missed many GPT-4o reviews unless they tolerated many false alarms. The anchor approach caught most GPT-4o reviews at a 5% false positive rate. This matters because conferences need detectors that rarely accuse honest reviewers.

Key points

  • Data: ICLR 2019–2024 submissions and reviews via the OpenReview API. AI reviews were generated by GPT-4o and Llama-3.1 (70b) using prompts adapted from Lu et al., with four personas: balanced, nitpicky, innovative and conservative.
  • On ICLR 2022 at a false positive rate of 0.05, the GPT-4o and Llama 3.1 anchor detectors found about 97% and 95% of GPT-4o reviews, and 87–90% of Llama reviews.
  • At the same false positive rate, Originality AI flagged only slightly over half of GPT-4o reviews, although it detected nearly all Llama-written ones.
  • AUC on GPT-4o reviews: RoBERTa 0.71, Longformer 0.88, Originality AI 0.94, both anchor methods 0.99. On Llama reviews, Originality AI led with 0.99.
  • An analysis of GPT-4o judge rationales, clustered with SFR-Embedding and t-SNE, showed it relies on style cues such as formal tone and generic critique. It sometimes cited non-native phrasing as a sign of AI text, which raises fairness concerns.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢IntelSanta Clara · United States22 works · 43,146 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 worksSocial Sciences · 2,009 worksSafety Research · 665 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Sungduk Yu, Man Luo, Avinash Madasu, Vasudev Lal and Phillip Howard. 2024. Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Review. arXiv:2410.03019.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2410.03019 · arXiv 2410.03019

@misc{yu2024is,
  title = {{Is Your Paper Being Reviewed by an LLM? Investigating AI Text Detectability in Peer Review}},
  author = {Sungduk Yu, Man Luo, Avinash Madasu, Vasudev Lal and Phillip Howard},
  year = {2024},
  doi = {10.48550/arxiv.2410.03019},
  eprint = {2410.03019},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2410.03019}
}

References

32 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Weixin Liang, Zachary Izzo, Yaohui Zhang, Haley Lepp, Hancheng Cao, Xuandong Zhao, Lingjiao Chen, Haotian Ye, Sheng Liu, Zhi Huang, et al. Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews. arXiv preprint arXiv:2403.07183, 2024.
  2. . URL https://github.com/openreview/openreview-py. ↗
  3. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  4. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
  5. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024.
  6. . URL https://iclr.cc/Conferences/2022/ReviewerGuide. ↗
  7. Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu. How close is chatgpt to human experts? comparison corpus, evaluation, and detection. differences, 14:18.
  8. Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
  9. Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150, 2020.
  10. Yafu Li, Qintong Li, Leyang Cui, Wei Bi, Zhilin Wang, Longyue Wang, Linyi Yang, Shuming Shi, and Yue Zhang. Mage: Machine-generated text detection in the wild. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 36–53, 2024.
  11. Sebastian Gehrmann, Hendrik Strobelt, and Alexander M Rush. Gltr: Statistical detection and visualization of generated text. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 111–116, 2019.
  12. Armand Joulin, Édouard Grave, Piotr Bojanowski, and Tomáš Mikolov. Bag of tricks for efficient text classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers, pages 427–431, 2017. find
  13. Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D Manning, and Chelsea Finn. Detectgpt: Zero-shot machine-generated text detection using probability curvature. In International Conference on Machine Learning, pages 24950–24962. PMLR, 2023.
  14. Arslan Akram. An empirical study of ai generated text detection tools. arXiv preprint arXiv:2310.01423, 2023. — [arXiv:2310.01423](https://arxiv.org/abs/2310.01423) find
  15. Jae Q. J. Liu, Kelvin T. K. Hui, Fadi Al Zoubi, Zing Z. X. Zhou, Dino Samartzis, Curtis C. H. Yu, Jeremy R. Chang, and Arnold Y. L. Wong. The great detectives: humans versus AI detectors in catching large language model-generated medical writing. International Journal for Educational Integrity, 20(1… — [doi:10.1007/s40979-024-00155-6](https://doi.org/10.1007/s40979-024-00155-6) find
  16. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–46623, 2023.
  17. . URL https://huggingface.co/Salesforce/SFR-Embedding-Mistral. ↗
  18. Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. Gpt detectors are biased against non-native english writers. Patterns, 4(7), 2023.
  19. Anton Bakhtin, Sam Gross, Myle Ott, Yuntian Deng, Marc’Aurelio Ranzato, and Arthur Szlam. Real or fake? learning to discriminate machine from human generated text. arXiv preprint arXiv:1906.03351, 2019. — [arXiv:1906.03351](https://arxiv.org/abs/1906.03351) find
  20. Ganesh Jawahar, Muhammad Abdul-Mageed, and VS Laks Lakshmanan. Automatic detection of machine generated text: A critical survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 2296–2309, 2020.
  21. Tiziano Fagni, Fabrizio Falchi, Margherita Gambini, Antonio Martella, and Maurizio Tesconi. Tweepfake: About detecting deepfake tweets. Plos one, 16(5):e0251415, 2021.
  22. Irene Solaiman, Miles Brundage, Jack Clark, Amanda Askell, Ariel Herbert-Voss, Jeff Wu, Alec Radford, Gretchen Krueger, Jong Wook Kim, Sarah Kreps, et al. Release strategies and the social impacts of language models. arXiv preprint arXiv:1908.09203, 2019.
  23. Y Liu, M Ott, N Goyal, J Du, M Joshi, D Chen, O Levy, M Lewis, L Zettlemoyer, and V Stoyanov. Roberta: A robustly optimized bert pretraining approach. arxiv [preprint](2019). arXiv preprint arXiv:1907.11692, 1907.
  24. Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and Yejin Choi. Defending against neural fake news. Advances in neural information processing systems, 32, 2019.
  25. Adaku Uchendu, Thai Le, Kai Shu, and Dongwon Lee. Authorship attribution for neural text generation. In Proceedings of the 2020 conference on empirical methods in natural language processing (EMNLP), pages 8384–8395, 2020.
  26. Daphne Ippolito, Daniel Duckworth, Chris Callison-Burch, and Douglas Eck. Automatic detection of generated text is easiest when humans are fooled. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1808–1822, 2020.
  27. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 1(8): AIoa2400196, 2024.
  28. Cheng Tan, Dongxin Lyu, Siyuan Li, Zhangyang Gao, Jingxuan Wei, Siqi Ma, Zicheng Liu, and Stan Z Li. Peer Review as A Multi-Turn and Long-Context Dialogue with Role-Based Interactions. arXiv, 2024. doi: 10.48550/arxiv.2406.05688. — [doi:10.48550/arxiv.2406.05688](https://doi.org/10.48550/arxiv.2406.05688) find
  29. Ruiyang Zhou, Lu Chen, and Kai Yu. Is LLM a reliable reviewer? a comprehensive evaluation of LLM on automatic paper reviewing tasks. In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, Proceedings of the 2024 Joint International Conferenc…
  30. Keith Tyser, Jason Lee, Avi Shporer, Madeleine Udell, Dov Te’eni, and Iddo Drori. Openreviewer: Mitigating challenges in llm reviewing. find
  31. Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, et al. What can natural language processing do for peer review? arXiv preprint arXiv:2405.06563, 2024.
  32. Edoardo Mosca, Mohamed Hesham Ibrahim Abdalla, Paolo Basso, Margherita Musumeci, and Georg Groh. Distinguishing fact from fiction: A benchmark dataset for identifying machine-generated scientific papers in the llm era. In Proceedings of the 3rd Workshop on Trustworthy Natural Language Processing (Tr… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yu-2024-paper-being-reviewed-llm/