The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates
Paper · 2024 · Giuseppe Russo Latona, Manoel Horta Ribeiro, Tim R. Davidson et al.
Summary
Large language models may now be helping reviewers write their peer reviews, and many fear this could distort scientific evaluation. This paper asks how common such AI-assisted reviewing actually is and whether it changes outcomes. The authors collected all ICLR 2024 submissions and reviews through the OpenReview API. They flagged reviews as likely AI-assisted using the commercial GPTZero detector, and they estimated the detector's false-positive rate from pre-ChatGPT years (2018–2023). They then ran two quasi-experimental analyses. The first compares AI-assisted and human reviews written for the same paper. The second matches papers with similar topics, review counts and human scores, where only one paper in each pair got an AI-assisted review. According to the results, roughly one in six reviews involved LLM help. These reviews tended to give more generous scores and raised acceptance chances, especially for borderline papers. The authors argue that this measurable influence supports concerns about the reliability of and trust in peer review. Code and annotated data are released publicly.
Key points
- Data: 7,404 ICLR 2024 submissions and 28,028 reviews from OpenReview. A review was labelled AI-assisted when GPTZero gave it a human probability below 0.5, and the results were robust to this cutoff.
- Prevalence (Study 1): At least 15.8% of reviews (4,428) were AI-assisted. This is a lower bound, because it is corrected for false positives but not for false negatives. Nearly half of the submissions (49.4%) received at least one flagged review.
- Scores (Study 2): The analysis covered 3,357 papers and 9,666 review combinations. When an AI-assisted review and a human review of the same paper disagreed, the AI-assisted one was higher in 53.4% of cases (p = 0.002). That corresponds to a +14.4% relative difference, or about 0.14 points higher on average.
- Acceptance (Study 3): In matched pairs, receiving an AI-assisted review raised acceptance by 3.1 percentage points overall. For borderline papers the increase was 4.9 percentage points (p = 0.024), which amounts to about 31.1% higher odds of acceptance.
- Resource: Code and data are available at github.com/epfl-dlab/AIReviewLottery.
Institutions
SwitzerlandFields, topics and concepts
📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 17,959 worksSocial Sciences · 2,009 worksSafety Research · 665 worksDecision Sciences · 306 worksInformation Systems and Management · 96 worksHistory and Philosophy of Science · 83 works
Cite
Cite as Giuseppe Russo Latona, Manoel Horta Ribeiro, Tim R. Davidson, Veniamin Veselovsky and Robert West. 2024. The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates. arXiv:2405.02150.
@misc{latona2024ai,
title = {{The AI Review Lottery: Widespread AI-Assisted Peer Reviews Boost Paper Scores and Acceptance Rates}},
author = {Giuseppe Russo Latona, Manoel Horta Ribeiro, Tim R. Davidson, Veniamin Veselovsky and Robert West},
year = {2024},
doi = {10.48550/arxiv.2405.02150},
eprint = {2405.02150},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2405.02150}
}Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/latona-2024-ai-review-lottery-widespread/