Elamur Research⌕
☰
home / papers / 2023 / ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing

ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing

Paper · 2023 · Ryan Liu and Nihar B. Shah

Summary

Peer review is overloaded, and human reviewers often miss flaws. Liu and Shah explore whether LLMs could take over parts of the reviewing workload. A pilot challenged nine models to find a deliberately planted error in a short paper: GPT-4 along with eight others, namely StableLM, LLaMa, Alpaca, Dolly, Koala, Vicuna, OpenAssistant and Bard. Only GPT-4 succeeded, and targeted questions gave more useful output than a generic request for a review. The authors then tested GPT-4 on three tasks:

  • finding errors deliberately inserted into handcrafted short computer science papers;
  • checking authors' "Yes" answers to NeurIPS 2022 checklist items against the relevant paper sections;
  • choosing the stronger abstract from constructed pairs where one is clearly better.

GPT-4 did well on the narrow tasks. Its error detection was roughly comparable to human reviewers, and its checklist verification was largely accurate. However, it failed most abstract comparisons, swayed by positive results, inflated wording, algorithm names and prompt injection. The authors conclude that LLMs can assist with specific reviewing subtasks but cannot yet act as standalone reviewers. Their handcrafted materials also form a small benchmark for future models.

Key points

  • Error detection: GPT-4 found the planted error in 7 of 13 short papers. The errors were both mathematical and conceptual. None of the other eight models detected any error.
  • Checklist verification: The test covered 15 NeurIPS 2022 papers, 16 checklist questions and 119 {checklist question, paper} pairs. Taking the majority of 3 runs, GPT-4 reached 86.6% accuracy. Coincidentally, author answers matched the ground truth at the same 86.6% rate. GPT-4 disagreed with 75% of the incorrect author answers. Half of its own errors involved questions that the text alone could not answer, e.g. information found only in figures. The intro once says "NeurIPS 2023", but the abstract and limitations say 2022.
  • Abstract comparison: GPT-4 picked wrongly in 6 of 10 pairs. It succeeded at four things: catching a misread null result, interpreting upper bounds correctly, ignoring buzzwords, and ignoring author prestige.
  • Limitations: Token limits meant full papers could not be passed to the model. The handcrafted dataset is small. The checklist sample is biased toward accepted papers.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,225 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 worksHistory and Philosophy of Science · 62 works

Cite

Cite as Ryan Liu and Nihar B. Shah. 2023. ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing. arXiv:2306.00622.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2306.00622 · arXiv 2306.00622

@misc{liu2023reviewergpt,
  title = {{ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing}},
  author = {Ryan Liu and Nihar B. Shah},
  year = {2023},
  doi = {10.48550/arxiv.2306.00622},
  eprint = {2306.00622},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2306.00622}
}

References

91 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. R. Anil et al. PaLM 2 Technical Report. 2023. arXiv: 2305.10403 [cs.CL].
  2. W. G. Baxt et al. “Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance”. In: Annals of emergency medicine (1998). find
  3. R. A. Bradley and M. E. Terry. “Rank analysis of incomplete block designs: I. The method of paired comparisons”. In: Biometrika (1952). find
  4. S. Bubeck et al. “Sparks of artificial general intelligence: Early experiments with GPT-4”. In: arXiv preprint arXiv:2303.12712 (2023).
  5. I. Buljan et al. “Meta-Research: Large-scale language analysis of peer review reports”. In: eLife (2020). find
  6. S. Chakraborty, P. Goyal, and A. Mukherjee. “Aspect-based Sentiment Analysis of Scientific Reviews”. In: JCDL. 2020. find
  7. Q. Chen et al. “A Comprehensive Benchmark Study on Biomedical Text Generation and Mining with ChatGPT”. In: bioRxiv (2023). find
  8. S. Chen et al. “LLM-empowered Chatbots for Psychiatrist and Patient Simulation: Application and Evaluation”. In: 2023. find
  9. A. Cohan et al. “SPECTER: Document-level Representation Learning using Citation-informed Transformers”. In: ACL. 2020.
  10. M. Conover et al. Hello Dolly: Democratizing the magic of ChatGPT with open models. Blog post. Mar. 2023. url: https://www.databricks . com / blog / 2023 / 03 / 24 / hello - dolly - democratizing-magic-chatgpt-open-models.html. ↗
  11. C. Caragea, A. Uban, and L. P. Dinu. “The myth of double-blind review revisited: ACL vs. EMNLP”. In: EMNLP-IJCNLP. 2019. find
  12. L. Charlin and R. Zemel. “The Toronto Paper Matching System: An automated paper-reviewer assignment system”. In: ICML Workshop on Peer Reviewing and Publishing Models. 2013. find
  13. Y. Du et al. Improving Factuality and Reasoning in Language Models through Multiagent Debate. 2023. arXiv: 2305.14325 [cs.CL].
  14. T. Foltỳnek, N. Meuschke, and B. Gipp. “Academic plagiarism detection: a systematic literature review”. In: ACM CSUR (2019). find
  15. M. Fromm et al. “Argument Mining Driven Analysis of Peer-Reviews”. In: arXiv preprint arXiv:2012.07743 (2020). — [arXiv:2012.07743](https://arxiv.org/abs/2012.07743) find
  16. X. Geng et al. Koala: A Dialogue Model for Academic Research. Blog post. Apr. 2023. url: https://bair.berkeley.edu/blog/2023/04/03/koala/. ↗
  17. F. Godlee, C. R. Gale, and C. N. Martyn. “Effect on the quality of peer review of blinding reviewers and asking them to sign their reports: a randomized controlled trial”. In: JAMA (1998). find
  18. T. Ghosal et al. “Peer review analyze: A novel benchmark resource for computational analysis of peer reviews”. In: PLOS one (2022). find
  19. K. Greshake et al. “Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection”. In: 2023. — Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
  20. H. Ge, M. Welling, and Z. Ghahramani. A Bayesian model for calibrating conference review scores. Manuscript. 2013. find
  21. M. Hosseini and S. P. Horbach. “Fighting reviewer fatigue or amplifying bias? Considerations and recommendations for use of ChatGPT and other Large Language Models in scholarly peer review”. In: Research Integrity and Peer Review (2023). find
  22. S. Hill and F. J. Provost. “The myth of the double-blind review? Author identification using only citations”. In: SIGKDD Explorations (Jan. 2003). find
  23. T. Houle. An Introduction to StatReviewer. EMUG. 2016. find
  24. P. Hämäläinen, M. Tavast, and A. Kunnari. “Evaluating Large Language Models in Generating Synthetic HCI Research Data: a Case Study”. In: CHI (2023). find
  25. X. Hua et al. “Argument mining for understanding peer reviews”. In: arXiv preprint arXiv:1903.10104 (2019). find
  26. J.-B. Huang. “Deep paper gestalt”. In: arXiv preprint arXiv:1812.08775 (2018). — [arXiv:1812.08775](https://arxiv.org/abs/1812.08775) find
  27. J. Huber et al. “Nobel and novice: Author prominence affects peer review”. In: PNAS (2022). find
  28. S. Jecmen et al. “Mitigating manipulation in peer review via randomized reviewer assignments”. In: NeurIPS (2020). find
  29. S. Jecmen et al. “A Dataset on Malicious Paper Bidding in Peer Review”. In: TheWebConf. 2023. find
  30. Z. Ji et al. “Survey of Hallucination in Natural Language Generation”. In: ACM CSUR (2022).
  31. Z. Jin et al. “When to Make Exceptions: Exploring Language Models as Accounts of Human Moral Judgment”. In: ArXiv (2022). find
  32. D. Kang et al. “A dataset of peer reviews (peerread): Collection, insights and NLP applications”. In: arXiv preprint arXiv:1804.09635 (2018).
  33. N. N. Kennard et al. “A Dataset for Discourse Structure in Peer Review Discussions”. In: arXiv preprint arXiv:2110.08520 (2021). — [arXiv:2110.08520](https://arxiv.org/abs/2110.08520) find
  34. W. E. Kerzendorf et al. “Distributed peer review enhanced with natural language processing and machine learning”. In: Nature Astronomy (2020). find
  35. E. Kıcıman et al. “Causal reasoning and large language models: Opening a new frontier for causality”. In: arXiv preprint arXiv:2305.00050 (2023). find
  36. T. Kojima et al. Large Language Models are Zero-Shot Reasoners. 2023. arXiv: 2205.11916
  37. A. Kobren, B. Saha, and A. McCallum. “Paper Matching with Local Fairness Constraints”. In: ACM KDD. 2019. find
  38. I. Kuznetsov et al. “Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review”. In: arXiv preprint arXiv:2204.10805 (2022).
  39. LAION AI. Open Assistant. 2023. url: https://projects.laion.ai/Open-Assistant/. ↗
  40. C. J. Lee. “Commensuration bias in peer review”. In: Philosophy of Science (2015). find
  41. M. L. Littman. “Collusion rings threaten the integrity of computer science research”. In: Communications of the ACM (2021). find
  42. Y. Liu et al. RoBERTa: A Robustly Optimized BERT Pretraining Approach. 2019. arXiv: 1907. 11692 [cs.CL].
  43. A. Liu et al. WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation. 2022. arXiv: 2201.05955 [cs.CL].
  44. X. Liu et al. “Large Language Models are Few-Shot Health Learners”. In: 2023. find
  45. R. D. Luce. Individual Choice Behavior. 1959. find
  46. J. Manyika. An overview of Bard: an early experiment with generative AI. 2023. url: https://ai.google/static/documents/google-about-bard.pdf. ↗
  47. A. McCook. “Is peer review broken? Submissions are up, reviewers are overtaxed, and authors are lodging complaint after complaint about the process at top-tier journals. What’s wrong with peer review?” In: The scientist (2006). find
  48. W. Morris et al. “Using Transformer Language Models to Validate Peer-Assigned Essay Scores in Massive Open Online Courses (MOOCs)”. In: LAK (2023). find
  49. Y. Matsubara and S. Singh. “Citations Beyond Self Citations: Identifying Authors, Affiliations, and Nationalities in Scientific Papers”. In: WOSP. 2020. find
  50. E. Manzoor and N. B. Shah. “Uncovering Latent Biases in Text: Method and Application to Peer Review”. In: AAAI. 2021. find
  51. J. J. Nay. “Large Language Models as Corporate Lobbyists”. In: ArXiv (2023). find
  52. R. Noothigattu, N. Shah, and A. Procaccia. “Loss Functions, Axioms, and Peer Review”. In: JAIR (2021). find
  53. OpenAI. ChatGPT – Release Notes. Blog post. May 2023. url: https://help.openai.com/en/articles/6825453-chatgpt-release-notes\#h_9894d7b0a4. ↗
  54. OpenAI. GPT-4 Technical Report. 2023. arXiv: 2303.08774 [cs.CL].
  55. J. S. Park et al. Social Simulacra: Creating Populated Prototypes for Social Computing Systems. 2022. arXiv: 2208.04024 [cs.HC].
  56. J. S. Park et al. “Generative Agents: Interactive Simulacra of Human Behavior”. In: ArXiv (2023). find
  57. C. Rastogi et al. “How do Authors’ Perceptions of their Papers Compare with Co-authors’ Perceptions and Peer-review Decisions?” In: arXiv:2211.12966 (2022). find
  58. Y. Razeghi et al. “Impact of Pretraining Term Frequencies on Few-Shot Reasoning”. In: ArXiv (2022). find
  59. V. Sanh et al. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. 2020. arXiv: 1910.01108 [cs.CL].
  60. S. Schroter et al. “Effects of training on quality of peer review: randomised controlled trial”. In: BMJ (2004). find
  61. S. Schroter et al. “What errors do peer reviewers detect, and does training improve their ability to detect them?” In: JRSM (2008). find
  62. J. Selvi. Exploring Prompt Injection Attacks. Blog post. Dec. 2022. url: https://research.nccgroup.com/2022/12/05/exploring-prompt-injection-attacks/. ↗
  63. N. B. Shah et al. “Estimation from pairwise comparisons: sharp minimax bounds with topology dependence”. In: JMLR (2016). find
  64. N. B. Shah. An Overview of Challenges, Experiments, and Computational Solutions in Peer Review. http://bit.ly/PeerReviewOverview (Abridged version published in the Communications of the ACM). June 2022. ↗
  65. C. H. Song et al. “LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models”. In: ArXiv (2022).
  66. T. Schick and H. Schütze. “Generating Datasets with Pretrained Language Models”. In: EMNLP. ACL, Nov. 2021. doi: 10.18653/v1/2021.emnlp- main.555. url: https://aclanthology.org/2021.emnlp-main.555. ↗
  67. I. Stelmakh, N. Shah, and A. Singh. “Catch Me if I Can: Detecting Strategic Behaviour in Peer Assessment”. In: AAAI. 2021. find
  68. I. Stelmakh, N. Shah, and A. Singh. “PeerReview4All: Fair and Accurate Reviewer Assignment in Peer Review”. In: JMLR (2021). find
  69. Stability AI. Stability AI Launches the First of its StableLM Suite of Language Models. Blog post. Apr. 2023. url: https://stability.ai/blog/stability-ai-launches-the-first-of-its-stablelm-suite-of-language-models. ↗
  70. I. Stelmakh et al. “A Gold Standard Dataset for the Reviewer Assignment Problem”. In: arXiv preprint arXiv:2303.16750 (2023). find
  71. R. Taori et al. Alpaca: A Strong, Replicable Instruction-Following Model. Blog post. Mar. 2023. url: https://crfm.stanford.edu/2023/03/13/alpaca.html. ↗
  72. The AJE Team. Peer Review: How We Found 15 Million Hours of Lost Time. 2013. url: https://www.aje.com/arc/peer-review-process-15-million-hours-lost-time/. ↗
  73. The Vicuna Team. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality. 2023. url: https://lmsys.org/blog/2023-03-30-vicuna/. ↗
  74. H. Touvron et al. LLaMA: Open and Efficient Foundation Language Models. 2023. arXiv: 2302. 13971 [cs.CL].
  75. M. Turpin et al. “Language Models Don’t Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting”. In: arXiv preprint arXiv:2305.04388 (2023).
  76. A. Tomkins, M. Zhang, and W. D. Heavlin. “Reviewer bias in single-versus double-blind peer review”. In: PNAS (2017). find
  77. T. Ullman. “Large Language Models Fail on Trivial Alterations to Theory-of-Mind Tasks”. In: ArXiv (2023). find
  78. T. N. Vijaykumar. Potential Organized Fraud in ACM/IEEE Computer Architecture Conferences. 2020. find
  79. Q. Wang et al. “ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis”. In: arXiv preprint arXiv:2010.06119 (2020). find
  80. S. Wang et al. “Want To Reduce Labeling Cost? GPT-3 Can Help”. In: EMNLP. ACL, Nov. 2021. doi: 10.18653/v1/2021.findings-emnlp.354.
  81. G. Wang et al. Voyager: An Open-Ended Embodied Agent with Large Language Models. 2023. arXiv: 2305.16291 [cs.AI]. find
  82. J. Wei et al. “Chain of Thought Prompting Elicits Reasoning in Large Language Models”. In: ArXiv (2022). — Chain-of-Thought Prompting Elicits Reasoning in Large Language Models
  83. J. Wieting et al. “Simple and Effective Paraphrastic Similarity from Parallel Translations”. In: ACL. July 2019. find
  84. M. Xia et al. “Training Trajectories of Language Models Across Scales”. In: arXiv preprint arXiv:2212.09803 (2022). find
  85. W. Yuan, P. Liu, and G. Neubig. “Can We Automate Scientific Reviewing?” In: arXiv preprint arXiv:2102.00176 (2021).
  86. J. Zhang et al. “Investigating Fairness Disparities in Peer Review: A Language Model Enhanced Approach”. In: ArXiv (2022). find
  87. M. Zhang et al. How Language Model Hallucinations Can Snowball. 2023. arXiv: 2305.13534
  88. S. Zhao et al. “Prompt as Triggers for Backdoor Attack: Examining the Vulnerability in Language Models”. In: ArXiv (2023). find
  89. F. Zhou et al. “Reflection of Thought: Inversely Eliciting Numerical Reasoning in Language Models via Solving Linear Systems”. In: ArXiv (2022). find
  90. C. Ziems et al. Can Large Language Models Transform Computational Social Science? 2023. arXiv: 2305.03514 [cs.CL].
  91. K. Zhou, D. Jurafsky, and T. Hashimoto. Navigating the Grey Area: Expressions of Overconfidence and Uncertainty in Language Models. 2023. arXiv: 2302.13439 [cs.CL]. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2023-reviewergpt-exploratory-study-using/