Elamur Research⌕
☰
home / papers / 2025 / Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards

Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards

Paper · 2025 · Jaeho Kim, Yunseok Lee and Seulki Lee

Summary

This ICML 2025 position paper examines the decline in peer review quality at major AI conferences. Venues such as NeurIPS, ICLR, ICML, CVPR and AAAI now each receive more than 10,000 submissions. The authors model the review ecosystem as three stakeholders: authors, reviewers and the System (venues plus platforms like OpenReview). They then split the causes of decline into two groups. Some cannot be fixed through review design, such as submission growth, LLM-assisted writing and fast-shifting research topics. Others can be addressed, such as reviewer negligence caused by a power imbalance and reviewer labor that goes unrewarded.

Instead of new data or models, the paper proposes two procedural reforms. The first is a two-stage, bi-directional double-blind process. Reviewers' summaries, strengths and questions are shown first, and authors judge how well the reviewer understood the paper and flag likely LLM-written text. Weaknesses and scores are revealed only afterward, which limits retaliation. The second is a System-backed reward scheme that turns reviewing into verifiable academic credit. The proposals matter because they are cheap, fit existing timelines and target reviewer accountability directly.

Key points

  • Submissions to several top AI venues passed 10,000 by 2025, and ICLR grew 59.8% in that year alone.
  • Cited evidence suggests over 17.5% of 2024 computer science papers were modified or generated with LLMs. The authors' own analysis also shows a rising share of typo-free reviews, which hints at LLM involvement in reviewing.
  • About 17% of the top-20 research keywords at a major AI conference change every year (Appendix A), which widens reviewers' knowledge gaps.
  • The proposed workflow splits review release into two parts: Section 1 (summary, strengths, questions), then author feedback on comprehension plus LLM flags, then Section 2 (weaknesses, rating).
  • Suggested incentives include digital badges and impact scores for reviewers, intended as long-term career credentials rather than one-off awards.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Ulsan National Institute of Science and TechnologyUlsan · South Korea4 works · 13 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 17,959 worksDecision Sciences · 306 worksInformation Systems and Management · 96 worksHistory and Philosophy of Science · 83 works

Cite

Cite as Jaeho Kim, Yunseok Lee and Seulki Lee. 2025. Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards. arXiv:2505.04966.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.04966 · arXiv 2505.04966

@misc{kim2025position,
  title = {{Position: The AI Conference Peer Review Crisis Demands Author Feedback and Reviewer Rewards}},
  author = {Jaeho Kim, Yunseok Lee and Seulki Lee},
  year = {2025},
  doi = {10.48550/arxiv.2505.04966},
  eprint = {2505.04966},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.04966}
}

References

50 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Impact Statement Beygelzimer, A., Dauphin, Y. N., Liang, P., and Vaughan, J. W. Has the machine learning review process become more arbitrary as the field has grown? the neurips 2021 consistency experiment. arXiv preprint arXiv:2306.03262, 2023. find
  2. The views and proposals presented in this position paper reflect solely the thoughts and opinions of the authors, and do not represent the official stance of any affiliated institutions, organizations, or other parties. We believe that this position paper could spark meaningful discussion on the nee… find
  3. Jecmen, S., Shah, N. B., Fang, F., and Akoglu, L. On the detection of reviewer-author collusion rings from paper bidding. arXiv preprint arXiv:2402.07860, 2024. — [arXiv:2402.07860](https://arxiv.org/abs/2402.07860) find
  4. Centeno, R., Hermoso, R., and Fasli, M. On the inaccuracy of numerical ratings: dealing with biased opinions in social networks. Information Systems Frontiers, 17:809– 825, 2015. find
  5. Kim, J. Author-based analysis of conference versus journal publication in computer science. Journal of the Association for Information Science and Technology, 70(1): 71–82, 2019. find
  6. Cortes, C. and Lawrence, N. D. Inconsistency in conference peer review: Revisiting the 2014 neurips experiment. arXiv preprint arXiv:2109.09774, 2021. — [arXiv:2109.09774](https://arxiv.org/abs/2109.09774) find
  7. Kuznetsov, I., Afzal, O. M., Dercksen, K., Dycke, N., Goldberg, A., Hope, T., Hovy, D., Kummerfeld, J. K., Lauscher, A., Leyton-Brown, K., et al. What can natural language processing do for peer review? arXiv preprint arXiv:2405.06563, 2024.
  8. Donker, T. The dangers of using large language models for peer review. 2023. find
  9. Elkhatat, A. M., Elsaid, K., and Almeer, S. Evaluating the efficacy of ai content detection tools in differentiating between human and ai-generated text. International Journal for Educational Integrity, 19(1):17, 2023. find
  10. Liang, W., Zhang, Y., Wu, Z., Lepp, H., Ji, W., Zhao, X., Cao, H., Liu, S., He, S., Huang, Z., et al. Mapping the increasing use of llms in scientific papers. In First Conference on Language Modeling. Facey-Shaw, L., Specht, M., Van Rosmalen, P., Brner, D., and Bartley-Bryan, J. Educational function… find
  11. Liang, W., Izzo, Z., Zhang, Y., Lepp, H., Cao, H., Zhao, X., Chen, L., Ye, H., Liu, S., Huang, Z., et al. Monitoring ai-modified content at scale: A case study on the impact of chatgpt on ai conference peer reviews. In International Conference on Machine Learning, pp. 29575– 29620. PMLR, 2024a.
  12. Freyne, J., Coyle, L., Smyth, B., and Cunningham, P. Relative status of journal and conference publications in computer science. Communications of the ACM, 53(11): 124–132, 2010. find
  13. Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D. Y., Yang, X., Vodrahalli, K., He, S., Smith, D. S., Yin, Y., et al. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 1(8):AIoa2400196, 2024b.
  14. Frijters, P. and Torgler, B. Improving the peer review process: a proposed market system. Scientometrics, 119(2): 1285–1288, 2019. find
  15. Liu, R. and Shah, N. B. Reviewergpt? an exploratory study on using large language models for paper reviewing. arXiv preprint arXiv:2306.00622, 2023.
  16. Gao, Y., Eger, S., Kuznetsov, I., Gurevych, I., and Miyao, Y. Does my rebuttal matter? insights from a major NLP conference. CoRR, abs/1903.11367, 2019. URL http://arxiv.org/abs/1903.11367. ↗
  17. Lu, C., Lu, C., Lange, R. T., Foerster, J., Clune, J., and Ha, D. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024.
  18. Gibson, D., Ostashewski, N., Flintoff, K., Grant, S., and Knight, E. Digital badges in education. Education and Information Technologies, 20:403–410, 2015. find
  19. Maslej, N., Fattorini, L., Perrault, R., Parli, V., Reuel, A., Brynjolfsson, E., Etchemendy, J., Ligett, K., Lyons, T., Manyika, J., Niebles, J. C., Shoham, Y., Wald, R., and Clark, J. Artificial intelligence index report 2024, 2024. URL https://arxiv.org/abs/2405.19522. — [arXiv:2405.19522](https://arxiv.org/abs/2405.19522) ↗
  20. Goldberg, A., Stelmakh, I., Cho, K., Oh, A., Agarwal, A., Belgrave, D., and Shah, N. B. Peer reviews of peer reviews: A randomized controlled trial and other experiments. PloS one, 20(4):e0320444, 2025.
  21. Mimno, D. and McCallum, A. Expertise modeling for matching papers with reviewers. In Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 500–509, 2007. find
  22. Hargittai, E. Potential biases in big data: Omitted voices on social media. Social science computer review, 38(1): 10–24, 2020. find
  23. Mulligan, A., Hall, L., and Raphael, E. Peer review in a changing world: An international study measuring the attitudes of researchers. Journal of the American Society for Information Science and Technology, 64(1):132–161, 2013. find
  24. Huang, J., Huang, W.-b., Bu, Y., Cao, Q., Shen, H., and Cheng, X. What makes a successful rebuttal in computer science conferences?: A perspective on social interaction. Journal of Informetrics, 17(3):101427, 2023. find
  25. Nicholson, J. and Alperin, J. P. A brief survey on peer review in scholarly communication. The Winnower, pp. 1–8, 2016. find
  26. Tyser, K., Segev, B., Longhitano, G., Zhang, X.-Y., Meeks, Z., Lee, J., Garg, U., Belsten, N., Shporer, A., Udell, M., et al. Ai-driven review systems: evaluating llms in scalable and bias-aware academic reviews. arXiv preprint arXiv:2408.10365, 2024.
  27. Northcraft, G. B. and Tenbrunsel, A. E. Effective matrices, decision frames, and cooperation in volunteer dilemmas: A theoretical perspective on academic peer review. Organization Science, 22(5):1277–1285, 2011. find
  28. Ugarov, A. Peer prediction for peer review: designing a marketplace for ideas. arXiv preprint arXiv:2303.16855, 2023. — [arXiv:2303.16855](https://arxiv.org/abs/2303.16855) find
  29. Rogers, A. and Augenstein, I. What can we do to improve peer review in nlp? In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1256– 1262, 2020. find
  30. Van Dalen, H. P. and Henkens, K. Intended and unintended consequences of a publish-or-perish culture: A worldwide survey. Journal of the American Society for Information Science and Technology, 63(7):1282–1293, 2012. find
  31. Ross, J. S., Gross, C. P., Desai, M. M., Hong, Y., Grant, A. O., Daniels, S. R., Hachinski, V. C., Gibbons, R. J., Gardner, T. J., and Krumholz, H. M. Effect of blinded peer review on abstract acceptance. Jama, 295(14):1675– 1680, 2006. find
  32. Warne, V. Rewarding reviewers–sense or sensibility? a wiley study explained. Learned Publishing, 29(1):41–50, 2016. find
  33. Whittaker, E. and Kitagishi, I. Large language models for simultaneous named entity extraction and spelling correction. arXiv preprint arXiv:2403.00528, 2024. — [arXiv:2403.00528](https://arxiv.org/abs/2403.00528) find
  34. Russo, A. Some ethical issues in the review process of machine learning conferences. arXiv preprint arXiv:2106.00810, 2021. — [arXiv:2106.00810](https://arxiv.org/abs/2106.00810) find
  35. Wu, J., Xu, H., Guo, Y., and Su, W. An isotonic mechanism for overlapping ownership. arXiv preprint arXiv:2306.11154, 2023. — [arXiv:2306.11154](https://arxiv.org/abs/2306.11154) find
  36. Sculley, D., Snoek, J., and Wiltschko, A. Avoiding a tragedy of the commons in the peer review process. arXiv preprint arXiv:1901.06246, 2018. — [arXiv:1901.06246](https://arxiv.org/abs/1901.06246) find
  37. Wu, J., Xu, H., Guo, Y., and Su, W. A truth serum for eliciting self-evaluations in scientific reviews. Technical report, 2024. find
  38. Shah, N. B. Challenges, experiments, and computational solutions in peer review. Communications of the ACM, 65(6):76–87, 2022. find
  39. Wu, R., Guo, C., Wu, F., Kidambi, R., Van Der Maaten, L., and Weinberger, K. Making paper reviewing robust to bid manipulation attacks. In International Conference on Machine Learning, pp. 11240–11250. PMLR, 2021. find
  40. Srinivasan, S. and Morgenstern, J. Auctions and peer prediction for academic peer review. arXiv preprint arXiv:2109.00923, 2021. — [arXiv:2109.00923](https://arxiv.org/abs/2109.00923) find
  41. Xu, Y. E., Fang, F., Tomczak, J., Zhang, C., Xue, Z. S., Paquet, U., and Belgrave, D. Neurips 2024 experiment on improving the paper-reviewer assignment. 2024. URL https://blog.neurips.cc/ category/2024-conference/. ↗
  42. Stefaniak, J. and Carey, K. Instilling purpose and value in the implementation of digital badges in higher education. International Journal of Educational Technology in Higher Education, 16(1):44, 2019. find
  43. Stephan, P. How economics shapes science. Harvard University Press, 2012. find
  44. Ye, R., Pang, X., Chai, J., Chen, J., Yin, Z., Xiang, Z., Dong, X., Shao, J., and Chen, S. Are we there yet? revealing the risks of utilizing large language models in scholarly peer review. arXiv preprint arXiv:2412.01708, 2024.
  45. Su, B., Zhang, J., Collina, N., Yan, Y., Li, D., Cho, K., Fan, J., Roth, A., and Su, W. J. Analysis of the icml 2023 ranking data: Can authors’ opinions of their own papers assist peer review in machine learning? CoRR, 2024. find
  46. Zhang, Y., Yu, F.-Y., Schoenebeck, G., and Kempe, D. A system-level analysis of conference peer review. In Proceedings of the 23rd ACM Conference on Economics and Computation, pp. 1041–1080, 2022. find
  47. Su, W. You are the best reviewer of your own papers: An owner-assisted scoring mechanism. Advances in Neural Information Processing Systems, 34:27929–27939, 2021. find
  48. Zhou, R., Chen, L., and Yu, K. Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pp. 9340–9351, 2024.
  49. Tran, D., Valtchanov, A., Ganapathy, K., Feng, R., Slud, E., Goldblum, M., and Goldstein, T. An open review of openreview: A critical analysis of the machine learning conference review process. arXiv preprint arXiv:2010.05137, 2020. — [arXiv:2010.05137](https://arxiv.org/abs/2010.05137) find
  50. A. Top 20 Keywords in ICLR by Year find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kim-2025-position-ai-conference-peer-review/