Elamur Research⌕
☰
home / papers / 2023 / Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments

Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments

Paper · 2023 · Alexander Goldberg, Ivan Stelmakh, Kyunghyun Cho et al.

Summary

Conferences, journals and funding bodies often score individual peer reviews. They use these scores to reward strong reviewers or as the outcome measure in experiments on the review process. This paper tests whether such scores can be trusted. At NeurIPS 2022, the authors recruited paper authors, reviewers and meta-reviewers to rate existing reviews on a 7-point scale. Raters gave an overall score plus scores on criteria such as constructiveness, coverage, understanding and substantiation.

The study combined two kinds of evidence. A randomized controlled trial showed some evaluators reviews padded with uninformative text. Observational analyses examined author ratings, agreement between evaluators, miscalibration, and how people turn criteria scores into an overall judgment.

Padding alone made reviews look better. Authors favored reviews that recommended accepting their own papers. Noise levels matched those in ordinary paper reviewing. So "reviewing reviews" carries the same weaknesses as peer review itself. Incentive schemes and experiments, including studies comparing LLM-written and human reviews, should therefore not treat these ratings as ground truth.

Key points

  • Length bias (RCT): Reviews artificially lengthened with non-informative content scored about 0.5 points higher on the 7-point scale (Mann-Whitney U, τ = 0.64, p < 0.0001). The effect also appeared in individual criteria scores. Evaluators in this trial were people who had not reviewed the paper in question.
  • Author-outcome bias: Authors rated "reject" reviews of their own papers roughly 1.4 points below "accept" reviews (τ = 0.82, p < 0.0001). This held after controlling for review length, quality (estimated from non-author evaluations) and how many accepted or rejected papers each author had.
  • Inconsistency: Two evaluators of the same review disagreed 28–32% of the time, a rate close to the disagreement among NeurIPS reviewers scoring papers.
  • Miscalibration and subjectivity: Miscalibration was estimated with a linear model of scores. Subjectivity was gauged by fitting a function that converts criteria scores into overall scores and taking that function's remaining loss as the measure. Both were comparable to NeurIPS paper review.
  • Limitations: Dishonest behavior, such as collusion rings, could not be studied because the ratings had no consequences. The 7-point scale also differs from the 10-point scale used for NeurIPS paper reviews.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States533 works · 443,885 citations🎓Carnegie Mellon UniversityPittsburgh · United States633 works · 195,218 citations🎓New York UniversityNew York · United States233 works · 104,003 citations🎓Korea Advanced Institute of Science and TechnologyDaejeon · South Korea59 works · 3,320 citations

Also on the byline: New Economic School · GlaxoSmithKline Research & Development Limited.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 17,959 worksPsychology · 1,673 worksExperimental and Cognitive Psychology · 950 worksDecision Sciences · 306 worksHistory and Philosophy of Science · 83 worksStatistics, Probability and Uncertainty · 67 works

Cite

Cite as Alexander Goldberg, Ivan Stelmakh, Kyunghyun Cho, Alice Oh, Alekh Agarwal, Danielle Belgrave and Nihar B. Shah. 2023. Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments. arXiv:2311.09497.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2311.09497 · arXiv 2311.09497

@misc{goldberg2023peer,
  title = {{Peer Reviews of Peer Reviews: A Randomized Controlled Trial and Other Experiments}},
  author = {Alexander Goldberg, Ivan Stelmakh, Kyunghyun Cho, Alice Oh, Alekh Agarwal, Danielle Belgrave and Nihar B. Shah},
  year = {2023},
  doi = {10.48550/arxiv.2311.09497},
  eprint = {2311.09497},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2311.09497}
}

References

53 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Clare Ardern, Nadia Martino, Sammy Nag, Robyn Tamblyn, David Moher, Adrian Mota, and Karim Khan. Three years of quality assurance data assessing the performance of over 4000 grant peer review contributions to the Canadian institutes of health research project grant competition. FACETS, 2023. find
  2. Stefano Balietti, Robert Goldstone, and Dirk Helbing. Peer review and competition in the art exhibition game. Proceedings of the National Academy of Sciences, 2016. find
  3. Michael L Callaham, William G Baxt, Joseph F Waeckerle, and Robert L Wears. Reliability of editors’ subjective quality ratings of peer reviews of manuscripts. Jama, 280(3):229–231, 1998. find
  4. Michael L. Callaham, Robert K. Knopp, and E. John Gallagher. Effect of Written Feedback by Editors on Quality of Reviews: Two Randomized Trials. JAMA, 287(21):2781–2783, 06 2002. find
  5. Corinna Cortes and Neil D. Lawrence. Inconsistency in conference peer review: Revisiting the 2014 NeurIPS experiment, 2021. find
  6. Michael Callaham and Charles McCulloch. Longitudinal trends in the performance of scientific peer reviewers. Annals of emergency medicine, 57(2):141–148, 2011. find
  7. Kevin C Chung, Melissa J Shauver, Sunitha Malay, Lin Zhong, Aaron Weinstein, and Rod J Rohrich. Is double-blinded peer review necessary? the effect of blinding on review quality. Plastic and reconstructive surgery, 136(6):1369–1377, 2015. find
  8. Michael L Callaham and John Tercier. The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS medicine, 4(1):e40, 2007. find
  9. Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04475, 2024. find
  10. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. MARG: multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.04259, 2024.
  11. Komal Dhull, Steven Jecmen, Pravesh Kothari, and Nihar B Shah. Strategyproofing peer assessment via partitioning: The price in terms of evaluators’ expertise. In HCOMP, 2022. find
  12. Esin Durmus, Faisal Ladhak, and Tatsunori Hashimoto. Spurious correlations in reference-free evaluation of text generation. arXiv preprint arXiv:2204.09890, 2022. find
  13. Arthur T Evans, Robert A McNutt, Suzanne W Fletcher, and Robert H Fletcher. The characteristics of peer reviewers who produce good-quality reviews. Journal of general internal medicine, 8:422–428, 1993. find
  14. Irene D Feurer, Gary J Becker, Daniel Picus, Estella Ramirez, Michael D Darcy, and Marshall E Hicks. Evaluating peer reviews: pilot testing of a grading instrument. JAMA, 272(2):98–100, 1994. find
  15. European Science Foundation. ESF survey analysis report on peer review practices, 2011. Available online https://www.esf.org/fileadmin/user_upload/esf/PeerReview-Practices_Survey2011.pdf. ↗
  16. H. Ge, M. Welling, and Z. Ghahramani. A Bayesian model for calibrating conference review scores, 2013. Available online http://mlg.eng.cam.ac.uk/hong/unpublished/nips-review-model.pdf Last accessed: April 4, 2021. ↗
  17. Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar B. Shah, Vincent Conitzer, and Fei Fang. Mitigating manipulation in peer review via randomized reviewer assignments. In NeurIPS, 2020. find
  18. Conny Kühne, Klemens Böhm, and Jing Zhi Yue. Reviewing the reviewers: A study of author perception on peer reviews in computer science. In 6th International Conference on Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom 2010), pages 1–8, 2010. find
  19. Maria K Kowalczuk, Frank Dudbridge, Shreeya Nanda, Stephanie L Harriman, Jigisha Patel, and Elizabeth C Moylan. Retrospective analysis of the quality of reports by author-suggested and non-author-suggested reviewers in journals operating on open or single-blind peer review models. Bmj Open, 5(9):e00… find
  20. Aditya Khosla, Derek Hoiem, and Serge Belongie. Analysis of reviews for cvpr 2012, 2013. find
  21. Hsuan-Tien Lin, Maria Florina Balcan, Raia Hadsell, and Marc’Aurelio Ranzato. What we learned from NeurIPS 2020 reviewing process, Oct 2020. find
  22. Carole J Lee. Commensuration bias in peer review. Philosophy of Science, 82(5):1272–1283, 2015. find
  23. Minhyeok Lee. Game-theoretical analysis of reviewer rewards in peer-review journal systems: Analysis and experimental evaluation using deep reinforcement learning. arXiv preprint arXiv:2305.12088, 2023. — [arXiv:2305.12088](https://arxiv.org/abs/2305.12088) find
  24. Michael L Littman. Collusion rings threaten the integrity of computer science research. Communications of the ACM, 64(6):43–44, 2021. find
  25. Ryan Liu and Nihar B Shah. Reviewergpt? an exploratory study on using large language models for paper reviewing. arXiv preprint arXiv:2306.00622, 2023.
  26. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Smith, Yian Yin, Daniel McFarland, and James Zou. Can large language models provide useful feedback on research papers? a large-scale empirical analysis, 2023.
  27. Abdelghani Maddi and Egidio Luis Miotti. On the peer review reports: It’s not the size that matters... really? arXiv preprint arXiv:2309.02000, 2023. — [arXiv:2309.02000](https://arxiv.org/abs/2309.02000) find
  28. Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. arXiv preprint arXiv:2405.14734, 2024. find
  29. Ritesh Noothigattu, Nihar Shah, and Ariel Procaccia. Loss functions, axioms, and peer review. J. Artif. Int. Res., 70:1481–1515, may 2021. find
  30. Konstantina Papagiannaki. Author feedback experiment at pam 2007. ACM SIGCOMM Computer Communication Review, 37(3):73–78, 2007. find
  31. Piitu Parmanne, Joonas Laajava, Noora Järvinen, Terttu Harju, Mauri Marttunen, and Pertti Saloheimo. Peer reviewers’ willingness to review, their recommendations and quality of reviews after the finnish medical journal switched from single-blind to double-blind peer review. Research Integrity and P… find
  32. Shelly M. Pranić, Mario Malički, Stjepan Ljudevit Marušić, Bahar Mehmani, and Ana Marušić. Is the quality of reviews reflected in editors’ and authors’ satisfaction with peer review? a cross-sectional study in 12 journals across four research fields. Learned Publishing, 34(2):187–197, 2021. find
  33. Magnus Roos, Jörg Rothe, Joachim Rudolph, Björn Scheuermann, and Dietrich Stoyan. A statistical approach to calibrating the scores of biased reviewers: The linear vs. the nonlinear model. In Multidisciplinary Workshop on Advances in Preference Handling, 2012. find
  34. Charvi Rastogi, Ivan Stelmakh, Alina Beygelzimer, Yann N Dauphin, Percy Liang, Jennifer Wortman Vaughan, Zhenyu Xue, Hal Daumé III, Emma Pierson, and Nihar B Shah. How do authors’ perceptions of their papers compare with co-authors’ perceptions and peer-review decisions? arXiv:2211.12966. Short blo… find
  35. Sara Schroter, Nick Black, Stephen Evans, James Carpenter, Fiona Godlee, and Richard Smith. Effects of training on quality of peer review: randomised controlled trial. Bmj, 328(7441):673, 2004. find
  36. Cecilia Superchi, José Antonio González, Ivan Solà, Erik Cobo, Darko Hren, and Isabelle Boutron. Tools used to assess the quality of peer review reports: a methodological systematic review. BMC medical research methodology, 19:1–14, 2019. find
  37. Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in RLHF. arXiv preprint arXiv:2310.03716, 2023.
  38. Nihar B Shah. Challenges, experiments, and computational solutions in peer review. Communications of the ACM, 65(6):76–87, 2022. find
  39. Siddarth Srinivasan and Jamie Morgenstern. Auctions and prediction markets for scientific peer review. arXiv preprint arXiv:2109.00923, 2021. — [arXiv:2109.00923](https://arxiv.org/abs/2109.00923) find
  40. Ivan Stelmakh, Nihar B Shah, Aarti Singh, and Hal Daumé III. A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4785–4793, 2021. find
  41. Nihar B. Shah, Behzad Tabibian, Krikamol Muandet, Isabelle Guyon, and Ulrike von Luxburg. Design and analysis of the NIPS 2016 review process. Journal of Machine Learning Research, 19(49):1–34, 2018. find
  42. Alexander Ugarov. Peer prediction for peer review: arXiv:2303.16855, 2023. — [arXiv:2303.16855](https://arxiv.org/abs/2303.16855) find
  43. T. N. Vijaykumar. Potential organized fraud in ACM/IEEE computer architecture conferences. https://medium. com/@tnvijayk/potential-organized-fraud-in-acm-ieee-computer-architecture-conferences-ccd61169370d, 2020. find
  44. Susan Van Rooyen, Nick Black, and Fiona Godlee. Development of the review quality instrument (rqi) for assessing peer reviews of manuscripts. Journal of clinical epidemiology, 52(7):625–629, 1999. find
  45. Susan Van Rooyen, Tony Delamothe, and Stephen JW Evans. Effect on peer review of telling reviewers that their signed reviews might be posted on the web: randomised controlled trial. Bmj, 341, 2010. designing a marketplace for ideas. arXiv preprint find
  46. Susan Van Rooyen, Fiona Godlee, Stephen Evans, Nick Black, and Richard Smith. Effect of open peer review on quality of reviews and on reviewers’ recommendations: a randomised trial. Bmj, 318(7175):23–27, 1999. find
  47. Ellen J Weber, Patricia P Katz, Joseph F Waeckerle, and Michael L Callaham. Author perception of peer review: impact of review quality and acceptance on satisfaction. JAMA, 287(21):2790–2793, 2002. find
  48. Elizabeth Walsh, Maeve Rooney, Louis Appleby, and Greg Wilkinson. Open peer review: A randomised controlled trial. The British Journal of Psychiatry, 176(1):47–51, 2000. find
  49. Jingyan Wang and Nihar B Shah. Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. In AAMAS, 2019. find
  50. Jingyan Wang, Ivan Stelmakh, Yuting Wei, and Nihar B Shah. Debiasing evaluations that are biased by evaluations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10120–10128, 2021. find
  51. Yuanzhang Xiao, Florian Dörfler, and Mihaela van der Schaar. Rating and matching in peer review systems. In Allerton conference, 2014. find
  52. Yuanzhang Xiao, Florian Dörfler, and Mihaela Van Der Schaar. Incentive design in peer review: Rating and repeated endogenous matching. IEEE Transactions on Network Science and Engineering, 2018. find
  53. Yichong Xu, Han Zhao, Xiaofei Shi, and Nihar Shah. On strategyproof conference review. In IJCAI, 2019. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/goldberg-2023-peer-reviews-peer-reviews/