Elamur Research⌕
☰
home / papers / 2024 / What Can Natural Language Processing Do for Peer Review?

What Can Natural Language Processing Do for Peer Review?

Paper · 2024 · Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen et al.

Summary

Scholarly peer review is under growing strain. Research output, especially in AI, is expanding faster than the pool of qualified reviewers. The result is costly, slow, sometimes biased reviewing, often done by overloaded or inexperienced people. Manuscripts, reviews and discussions are mostly text, so the authors ask where NLP could help and where it should stay out. The authors are a large group that grew out of Dagstuhl seminar 24,052.

This white paper does not introduce a new model or benchmark. Instead, it maps the problem space. It follows the reviewing workflow of AI conferences from submission to camera-ready revision, linking each stage to its difficulties and to existing NLP work. It then covers cross-cutting issues: gathering and licensing review data, operationalizing and evaluating tasks, and ethics.

The central conclusion is that fully automated reviewing is not a realistic goal for now. However, many narrower, well-defined subtasks can be supported, with or without LLMs. Alongside a companion dataset repository and a call for action, the paper serves as a reference agenda for the field.

Key points

  • The paper originated at seminar number 24,052 held at Dagstuhl, which carried the title "Reviewer No. 2" and focused on old and new problems in peer review. Coordination was led by Ilia Kuznetsov (TU Darmstadt, UKP Lab), and the organizers were Iryna Gurevych, Anna Rogers and Nihar B. Shah.
  • The paper motivates the problem with scale and bias:
    • AAAI routinely receives more than 10,000 submissions.
    • Estimates suggest millions of expert hours go into reviewing each year.
    • Documented biases include prestige, nationality, gender, race, language proficiency, confirmation bias and publication bias.
  • Peer review is framed as a chain of many tasks rather than one task. These connect to core NLP problems: reasoning, scientific text understanding, cross-document modeling, summarization, evaluation, bias mitigation, and privacy.
  • A companion repository (github.com/OAfzal/nlp-for-peer-review) collects peer-review datasets and accepts new contributions.
  • The call for action addresses the research community, NLP/AI researchers, policymakers and funders. For policy, it stresses four principles:
    • transparency about tool use;
    • preference for explainable methods;
    • rigorous, incremental and reversible deployment;
    • ongoing monitoring with feedback channels for all participants.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States633 works · 195,218 citations🎓University of WashingtonSeattle · United States422 works · 213,539 citations🔬Allen Institute for Artificial IntelligenceSeattle · United States317 works · 106,633 citations🎓Technische Universität DarmstadtDarmstadt · Germany123 works · 36,745 citations🎓Georgia Institute of TechnologyAtlanta · United States159 works · 28,096 citations🎓University of British ColumbiaVancouver · Canada107 works · 37,489 citations🎓The University of SydneySydney · Australia52 works · 68,450 citations🎓Hebrew University of JerusalemJerusalem · Israel71 works · 9,675 citations🎓Mohamed bin Zayed University of Artificial IntelligenceAbu Dhabi · United Arab Emirates145 works · 6,810 citations🎓Monash UniversityMelbourne · Australia66 works · 55,050 citations🎓Universität HamburgHamburg · Germany84 works · 5,692 citations🎓Bocconi UniversityMilan · Italy29 works · 3,110 citations🎓Radboud University NijmegenNijmegen · Netherlands26 works · 2,125 citations🎓IT University of CopenhagenCopenhagen · Denmark28 works · 2,439 citations🎓Indian Institute of Science BangaloreBengaluru · India16 works · 1,462 citations🎓Indian Institute of Technology DelhiNew Delhi · India26 works · 6,995 citations🎓Queen's UniversityKingston · Canada12 works · 1,670 citations🔬Laboratoire Interdisciplinaire des Sciences du NumériqueGif-sur-Yvette · France16 works · 630 citations

Also on the byline: University of British Columbia, Okanagan Campus.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksHistory and Philosophy of Science · 83 works

Cite

Cite as Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, Sheng Lu, Mausam, Margot Mieskes, Aurélie Névéol, Danish Pruthi, Lizhen Qu, Roy Schwartz, Noah A. Smith, Thamar Solorio, Jingyan Wang, Xiaodan Zhu, Anna Rogers, Nihar B. Shah and Iryna Gurevych. 2024. What Can Natural Language Processing Do for Peer Review?. arXiv:2405.06563.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2405.06563 · arXiv 2405.06563

@misc{kuznetsov2024what,
  title = {{What Can Natural Language Processing Do for Peer Review?}},
  author = {Ilia Kuznetsov, Osama Mohammed Afzal, Koen Dercksen, Nils Dycke, Alexander Goldberg, Tom Hope, Dirk Hovy, Jonathan K. Kummerfeld, Anne Lauscher, Kevin Leyton-Brown, Sheng Lu, Mausam, Margot Mieskes, Aurélie Névéol, Danish Pruthi, Lizhen Qu, Roy Schwartz, Noah A. Smith, Thamar Solorio, Jingyan Wang, Xiaodan Zhu, Anna Rogers, Nihar B. Shah and Iryna Gurevych},
  year = {2024},
  doi = {10.48550/arxiv.2405.06563},
  eprint = {2405.06563},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2405.06563}
}

References

166 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AbuRa’ed, A., Chiruzzo, L., and Saggion, H. (2017). What sentence are you referring to and why? Identifying cited sentences in scientific literature. In Mitkov, R. and Angelova, G., editors, Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017, pages… find
  2. AJE (2013). Peer review: How we found 15 million hours of lost time. https://www.aje.com/arc/ peer-review-process-15-million-hours-lost-time/. Online; accessed 01-March-2024. ↗
  3. Ali, Z., Kefalas, P., Muhammad, K., Ali, B., and Imran, M. (2020). Deep learning in citation recommendation models survey. Expert Systems with Applications, 162. find
  4. Altmäe, S., Sola-Leyva, A., and Salumets, A. (2023). Artificial intelligence in scientific writing: A friend or a foe? Reproductive BioMedicine Online, 47(1):3–9. find
  5. Amado Olivo, V. and Kerzendorf, W. (2023). Development of a global registry for peer review in astrophysics. In Peer Review Under Review, page 13. find
  6. Belouadi, J., Lauscher, A., and Eger, S. (2023). AutomaTikZ: Text-guided synthesis of scientific vector graphics with TikZ. In The Twelfth International Conference on Learning Representations (ICLR).
  7. Beltagy, I., Cohan, A., Feigenblat, G., Freitag, D., Ghosal, T., Hall, K., Herrmannova, D., Knoth, P., Lo, K., Mayr, P., Patton, R. M., Shmueli-Scheuer, M., de Waard, A., Wang, K., and Wang, L. L., editors (2021). Proceedings of the Second Workshop on Scholarly Document Processing. Association for C… find
  8. Belz, A. (2022). A metrological perspective on reproducibility in NLP. Computational Linguistics, 48(4):1125– 1135. find
  9. Belz, A., Thomson, C., Reiter, E., and Mille, S. (2023). Non-repeatable experiments and non-reproducible results: The reproducibility crisis in human evaluation in NLP. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Findings of the Association for Computational Linguistics: ACL 2023, page… find
  10. Bender, E. M. and Friedman, B. (2018). Data statements for natural language processing: Toward mitigating system bias and enabling better science. Transactions of the Association for Computational Linguistics, 6:587–604. find
  11. Beygelzimer, A., Dauphin, Y., Liang, P., and Wortman Vaughan, J. (2021a). Introducing the NeurIPS 2021 Paper Checklist. https://neuripsconf.medium.com/ introducing-the-neurips-2021-paper-checklist-3220d6df500b. Online; accessed 17-April-2024. ↗
  12. Beygelzimer, A., Dauphin, Y., Liang, P., and Wortman Vaughan, J. (2021b). find
  13. The NeurIPS 2021 consistency experiment. https://blog.neurips.cc/2021/12/08/the-neurips-2021-consistency-experiment/. Online; accessed 18-April-2024. ↗
  14. Bhagavatula, C., Feldman, S., Power, R., and Ammar, W. (2018). Content-based citation recommendation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 238–251. find
  15. Bharti, P., Navlakha, M., Agarwal, M., and Ekbal, A. (2023). PolitePEER: does peer review hurt? A dataset to gauge politeness intensity in the peer reviews. Language Resources and Evaluation, pages 1–23. find
  16. Biswas, S., Dobaria, D., and Cohen, H. L. (2023). ChatGPT and the future of journal reviews: A feasibility study. The Yale Journal of Biology and Medicine, 96:415–420. find
  17. Blecher, L., Cucurull, G., Scialom, T., and Stojnic, R. (2023). Nougat: Neural optical understanding for academic documents. In The Twelfth International Conference on Learning Representations (ICLR).
  18. Bornmann, L. (2011). Scientific peer review. Annual Review of Information Science and Technology, 45(1):197–245. find
  19. Cachola, I., Lo, K., Cohan, A., and Weld, D. (2020). TLDR: Extreme summarization of scientific documents. In Cohn, T., He, Y., and Liu, Y., editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4766–4777. Association for Computational Linguistics.
  20. Çano, E. and Bojar, O. (2020). Two huge title and keyword generation corpora of research articles. In Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., and Piperidis, S., editors, Proceedin… find
  21. Caragea, C., Bulgarov, F. A., Godea, A., and Das Gollapalli, S. (2014). Citation-enhanced keyphrase extraction from research papers: A supervised approach. In Moschitti, A., Pang, B., and Daelemans, W., editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (… find
  22. Chakraborty, S., Goyal, P., and Mukherjee, A. (2020). Aspect-based sentiment analysis of scientific reviews. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, JCDL ’20, page 207–216, New York, NY, USA. Association for Computing Machinery. find
  23. Chandrasekaran, M. K., de Waard, A., Feigenblat, G., Freitag, D., Ghosal, T., Hovy, E., Knoth, P., Konopnicki, D., Mayr, P., Patton, R. M., and Shmueli-Scheuer, M., editors (2020). Proceedings of the First Workshop on Scholarly Document Processing. Association for Computational Linguistics. find
  24. Charlin, L. and Zemel, R. (2013). The Toronto Paper Matching System: An automated paper-reviewer assignment system. In Proceedings of the 30th International Conference on Machine Learning, volume 28, Atlanta, Georgia, USA. Journal of Machine Learning Research Workshop and Conference Proceedings. find
  25. Cheng, L., Bing, L., Yu, Q., Lu, W., and Si, L. (2020). APE: Argument pair extraction from peer review and rebuttal via multi-task learning. In Webber, B., Cohn, T., He, Y., and Liu, Y., editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 70… find
  26. Chouldechova, A. (2017). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. Big Data, 5(2):153–163. find
  27. Church, K. W. (2020). Emerging trends: Reviewing the reviewers (again). Natural Language Engineering, 26(2):245–257. find
  28. Cohan, A., Feigenblat, G., Freitag, D., Ghosal, T., Herrmannova, D., Knoth, P., Lo, K., Mayr, P., Shmueli-Scheuer, M., de Waard, A., and Wang, L. L., editors (2022). Proceedings of the Third Workshop on Scholarly Document Processing, volume 29, Gyeongju, Republic of Korea. Association for Computatio… find
  29. Cohan, A., Feldman, S., Beltagy, I., Downey, D., and Weld, D. S. (2020). Specter: Document-level representation learning using citation-informed transformers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2270–2282.
  30. D’Arcy, M., Hope, T., Birnbaum, L., and Downey, D. (2024). MARG: Multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.04259.
  31. D’Arcy, M., Ross, A., Bransom, E., Kuehl, B., Bragg, J., Hope, T., and Downey, D. (2023). ARIES: A corpus of scientific paper edits made in response to peer reviews. arXiv preprint arXiv:2306.12587.
  32. Some NAACL 2013 statistics on author response, review quality, etc. https://nlpers.blogspot.com/2015/06/some-naacl-2013-statistics-on-author.html. Online; accessed 21-April-2024. ↗
  33. De Kock, C. and Vlachos, A. (2022). How to disagree well: Investigating the dispute tactics used on Wikipedia. In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3824–3837, Abu Dhabi, United Arab Emirat… find
  34. Ding, W., Kamath, G., Wang, W., and Shah, N. B. (2022). Calibration with privacy in peer review. In ISIT, pages 1635–1640. IEEE. find
  35. Ding, W., Shah, N. B., and Wang, W. (2020). On the privacy-utility tradeoff in peer-review data analysis. In AAAI Privacy-Preserving Artificial Intelligence (PPAI-21) workshop. find
  36. Dodge, J., Gururangan, S., Card, D., Schwartz, R., and Smith, N. A. (2019). Show Your Work: Improved reporting of experimental results. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing… find
  37. Drori, I. and Te’eni, D. (2024). Human-in-the-loop AI reviewing: Feasibility, opportunities, and risks. Journal of the Association for Information Systems, 25(1):98–109. find
  38. Du, W., Raheja, V., Kumar, D., Kim, Z. M., Lopez, M., and Kang, D. (2022). Understanding iterative revision from human-written text. In Muresan, S., Nakov, P., and Villavicencio, A., editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pape… find
  39. Dwork, C. and Roth, A. (2014). The algorithmic foundations of differential privacy. Foundations and Trends in Theoretical Computer Science, 9(3-4):211–407. find
  40. Dycke, N., Kuznetsov, I., and Gurevych, I. (2022). Yes-yes-yes: Proactive data collection for ACL rolling review and beyond. In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 300–318, Abu Dhabi, United Arab Emirates. A… find
  41. Dycke, N., Kuznetsov, I., and Gurevych, I. (2023a). NLPeer: A unified resource for the computational study of peer review. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages…
  42. Dycke, N., Kuznetsov, I., and Gurevych, I. (2023b). Overview of PragTag-2023: Low-resource multi-domain pragmatic tagging of peer reviews. In Alshomary, M., Chen, C.-C., Muresan, S., Park, J., and Romberg, J., editors, Proceedings of the 10th Workshop on Argument Mining, pages 187–196, Singapore. As…
  43. Else, H. and Van Noorden, R. (2021). The fight against fake-paper factories that churn out sham science. Nature, 591(7851):516–519. find
  44. Ernst, E. and Kienbacher, T. (1991). Chauvinism. Nature, 352(6336):560–560. find
  45. Färber, M. and Jatowt, A. (2020). Citation recommendation: Approaches and datasets. International Journal on Digital Libraries, 21(4):375–405. find
  46. Fiez, T., Shah, N., and Ratliff, L. (2020). A SUPER* algorithm to optimize paper bidding in peer review. In Conference on Uncertainty in Artificial Intelligence, pages 580–589. Proceedings of Machine Learning Research. find
  47. Fok, R., Kambhamettu, H., Soldaini, L., Bragg, J., Lo, K., Hearst, M., Head, A., and Weld, D. S. (2023). Scim: Intelligent skimming support for scientific papers. In Proceedings of the 28th International Conference on Intelligent User Interfaces, pages 476–490. find
  48. Frachtenberg, E. and Koster, N. (2020). A survey of accepted authors in computer systems conferences. PeerJ Computer Science, 6. find
  49. Fu, T., Wang, W. Y., McDuff, D., and Song, Y. (2022). DOC2PPT: Automatic presentation slides generation from scientific documents. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twe… find
  50. Gao, Y., Eger, S., Kuznetsov, I., Gurevych, I., and Miyao, Y. (2019). Does my rebuttal matter? Insights from a major NLP conference. In Burstein, J., Doran, C., and Solorio, T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics… find
  51. Ge, H., Welling, M., and Ghahramani, Z. (2013). A Bayesian model for calibrating conference review scores. Manuscript. Available online http://mlg.eng.cam.ac.uk/hong/unpublished/nips-review-model.pdf Last accessed: April 4, 2021. ↗
  52. Ghosal, T., Varanasi, K. K., and Kordoni, V. (2022). HedgePeer: A dataset for uncertainty detection in peer reviews. In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries, pages 1–5. find
  53. Ginther, D. K., Schaffer, W. T., Schnell, J., Masimore, B., Liu, F., Haak, L. L., and Kington, R. (2011). Race, ethnicity, and NIH research awards. Science, 333(6045):1015–1019. find
  54. Glockner, M., Staliūnaitė, I., Thorne, J., Vallejo, G., Vlachos, A., and Gurevych, I. (2024). Ambifc: Fact-checking ambiguous claims with evidence. Transactions of the Association for Computational Linguistics, 12:1–18.
  55. Goldberg, A., Fanti, G., and Shah, N. B. (2023a). Batching of tasks by users of pseudonymous forums: Anonymity compromise and protection. Proceedings of the ACM on Measurement and Analysis of Computing Systems, 7(1). find
  56. Goldberg, A., Stelmakh, I., Cho, K., Oh, A., Agarwal, A., Belgrave, D., and Shah, N. B. (2023b). Peer reviews of peer reviews: A randomized controlled trial and other experiments. arXiv preprint arXiv:2311.09497.
  57. Goldberg, P. (1968). Are women prejudiced against women? Trans-action, 5(5):28–30. find
  58. GSPR (2018). Global State of Peer Review 2018. Wellington: Publons. find
  59. Guo, Y., Shang, G., Rennard, V., Vazirgiannis, M., and Clavel, C. (2023). Automatic analysis of substantiation in scientific peer reviews. In Bouamor, H., Pino, J., and Bali, K., editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10198–10216, Singapore. Association… find
  60. Head, A., Xie, A., and Hearst, M. A. (2022). Math augmentation: How authors enhance the readability of formulas using novel visual design practices. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pages 1–18. find
  61. Hofstra, B., Kulkarni, V. V., Galvez, S. M.-N., He, B., Jurafsky, D., and McFarland, D. A. (2020). The diversity–innovation paradox in science. Proceedings of the National Academy of Sciences, 117(17):9284– 9291. find
  62. Hu, Y. and Wan, X. (2013). Ppsgen: learning to generate presentation slides for academic papers. In Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence, pages 2099–2105. find
  63. Hua, X., Nikolov, M., Badugu, N., and Wang, L. (2019). Argument mining for understanding peer reviews. In Burstein, J., Doran, C., and Solorio, T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  64. Jecmen, S., Shah, N. B., Fang, F., and Akoglu, L. (2024). On the detection of reviewer-author collusion rings from paper bidding. arXiv preprint arXiv:2402.07860. — [arXiv:2402.07860](https://arxiv.org/abs/2402.07860) find
  65. Jecmen, S., Shah, N. B., Fang, F., and Conitzer, V. (2022). Tradeoffs in preventing manipulation in paper bidding for reviewer assignment. arXiv preprint arXiv:2207.11315. — [arXiv:2207.11315](https://arxiv.org/abs/2207.11315) find
  66. Jecmen, S., Yoon, M., Conitzer, V., Shah, N. B., and Fang, F. (2023). A dataset on malicious paper bidding in peer review. In Proceedings of the ACM Web Conference 2023, pages 3816–3826. find
  67. Jecmen, S., Zhang, H., Liu, R., Shah, N., Conitzer, V., and Fang, F. (2020). Mitigating manipulation in peer review via randomized reviewer assignments. Advances in Neural Information Processing Systems, 33:12533–12545. find
  68. Jiang, C., Xu, W., and Stevens, S. (2022). arXivEdits: Understanding the human revision process in scientific writing. In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9420–9435, Abu Dhabi, United Ara… find
  69. Jurgens, D., Kumar, S., Hoover, R., McFarland, D., and Jurafsky, D. (2018). Measuring the evolution of a scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391–406.
  70. Kamocki, P., Mapelli, V., and Choukri, K. (2018). Data management plan (DMP) for language data under the new general data protection regulation (GDPR). In Calzolari, N., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Hasida, K., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk,… find
  71. Kamocki, P. and Witt, A. (2020). Privacy by design and language resources. In Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., and Piperidis, S., editors, Proceedings of the Twelfth Langua… find
  72. Kang, D., Ammar, W., Dalvi, B., van Zuylen, M., Kohlmeier, S., Hovy, E., and Schwartz, R. (2018). A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Walker, M., Ji, H., and Stent, A., editors, Proceedings of the 2018 Conference of the North American Chapter of the As…
  73. Kennard, N., O’Gorman, T., Das, R., Sharma, A., Bagchi, C., Clinton, M., Yelugam, P. K., Zamani, H., and McCallum, A. (2022). DISAPERE: A dataset for discourse structure in peer review discussions. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V., editors, Proceedings of the 2022 Conference… find
  74. Kim, J. S., Chen, V., Pruthi, D., Shah, N. B., and Talwalkar, A. (2023). Assisting human decisions in document matching. Transactions on Machine Learning Research. find
  75. Kleinberg, J., Mullainathan, S., and Raghavan, M. (2016). Inherent trade-offs in the fair determination of risk scores. arXiv preprint arXiv:1609.05807. — [arXiv:1609.05807](https://arxiv.org/abs/1609.05807) find
  76. Klymenko, O., Meisenbacher, S., and Matthes, F. (2022). Differential privacy in natural language processing the story so far. In Feyisetan, O., Ghanavati, S., Thaine, P., Habernal, I., and Mireshghallah, F., editors, Proceedings of the Fourth Workshop on Privacy in Natural Language Processing, pages… find
  77. Kobren, A., Saha, B., and McCallum, A. (2019). Paper matching with local fairness constraints. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1247–1257. find
  78. Kumar, A., Ghosal, T., and Ekbal, A. (2021). A deep neural architecture for decision-aware meta-review generation. In 2021 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 222–225. find
  79. Künzli, N., Berger, A., Czabanowska, K., Lucas, R., Madarasova Geckova, A., Mantwill, S., and von Dem Knesebeck, O. (2022). «I Do Not Have Time» - is this the end of peer review in public health sciences? Public Health Reviews, 43. find
  80. Kuznetsov, I., Buchmann, J., Eichler, M., and Gurevych, I. (2022). Revise and Resubmit: An intertextual model of text-based collaboration in peer review. Computational Linguistics, 48(4):949–986.
  81. Landhuis, E. (2016). Scientific literature: Information overload. Nature, 535(7612):457–458. find
  82. Lauscher, A., Glavaš, G., Ponzetto, S. P., and Eckert, K. (2018). Investigating the role of argumentation in the rhetorical analysis of scientific publications with neural multi-task learning models. In Riloff, E., Chiang, D., Hockenmaier, J., and Tsujii, J., editors, Proceedings of the 2018 Confere… find
  83. Lauscher, A., Ko, B., Kuehl, B., Johnson, S., Cohan, A., Jurgens, D., and Lo, K. (2022). MultiCite: Modeling realistic citations requires moving beyond the single-sentence single-label setting. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V., editors, Proceedings of the 2022 Conference of t… find
  84. Lawrence, N. and Cortes, C. (2014). The NIPS Experiment. http://inverseprobability.com/2014/12/ 16/the-nips-experiment. Online; accessed 17-April-2024. ↗
  85. Lee, C. J. (2015). Commensuration bias in peer review. Philosophy of Science, 82(5):1272–1283. find
  86. Lee, C. J., Sugimoto, C. R., Zhang, G., and Cronin, B. (2013). Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1):2–17. find
  87. Leyton-Brown, K., Mausam, Nandwani, Y., Zarkoob, H., Cameron, C., Newman, N., and Raghu, D. (2024). Matching papers and reviewers at large conferences. Artificial Intelligence, 331:104119. find
  88. Li, J., Sato, A., Shimura, K., and Fukumoto, F. (2020). Multi-task peer-review score prediction. In Chandrasekaran, M. K., de Waard, A., Feigenblat, G., Freitag, D., Ghosal, T., Hovy, E., Knoth, P., Konopnicki, D., Mayr, P., Patton, R. M., and Shmueli-Scheuer, M., editors, Proceedings of the First W… find
  89. Li, J. J. and Nenkova, A. (2015). Fast and accurate prediction of sentence specificity. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, AAAI’15, page 2281–2287. AAAI Press. find
  90. Li, M., Hovy, E., and Lau, J. (2023). Summarizing multiple documents with conversational structure for meta-review generation. In Bouamor, H., Pino, J., and Bali, K., editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7089–7112, Singapore. Association for Computati… find
  91. Li, N., Li, T., and Venkatasubramanian, S. (2007). t-Closeness: Privacy beyond k-anonymity and l-diversity. In Chirkova, R., Dogac, A., Özsu, M. T., and Sellis, T. K., editors, Proceedings of the 23rd International Conference on Data Engineering, pages 106–115. IEEE Computer Society. find
  92. Liang, W., Zhang, Y., Cao, H., Wang, B., Ding, D., Yang, X., Vodrahalli, K., He, S., Smith, D., Yin, Y., et al. (2023). Can large language models provide useful feedback on research papers? A large-scale empirical analysis. arXiv preprint arXiv:2310.01783.
  93. Littman, M. L. (2021). Collusion rings threaten the integrity of computer science research. Communications of the ACM, 64(6):43–44. find
  94. Liu, R., Jecmen, S., Conitzer, V., Fang, F., and Shah, N. B. (2023). Testing for reviewer anchoring in peer review: A randomized controlled trial. arXiv preprint arXiv:2307.05443. — [arXiv:2307.05443](https://arxiv.org/abs/2307.05443) find
  95. Liu, R. and Shah, N. B. (2023). ReviewerGPT? An exploratory study on using large language models for paper reviewing. arXiv preprint arXiv:2306.00622.
  96. Lo, K., Shen, Z., Newman, B., Chang, J., Authur, R., Bransom, E., Candra, S., Chandrasekhar, Y., Huff, R., Kuehl, B., Singh, A., Wilhelm, C., Zamarron, A., Hearst, M. A., Weld, D., Downey, D., and Soldaini, L. (2023). PaperMage: A unified toolkit for processing, representing, and manipulating visual… find
  97. Lo, K., Wang, L. L., Neumann, M., Kinney, R., and Weld, D. (2020). S2ORC: The semantic scholar open research corpus. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983. Associat…
  98. Malički, M. and Mehmani, B. (2024). Structured peer review: Pilot results from 23 Elsevier journals. bioRxiv preprint bioRxiv:10.1101/2024.02.01.578440. — [doi:10.1101/2024.02.01.578440](https://doi.org/10.1101/2024.02.01.578440) find
  99. Manzoor, E. and Shah, N. B. (2021). Uncovering latent biases in text: Method and application to peer review. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4767–4775. find
  100. Mao, Y., Zhong, M., and Han, J. (2022). CiteSum: Citation text-guided scientific extreme summarization and domain adaptation with limited supervision. In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages… find
  101. Meir, R., Lang, J., Lesca, J., Mattei, N., and Kaminsky, N. (2021). A market-inspired bidding scheme for peer review paper assignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4776–4784. find
  102. Mita, M., Sakaguchi, K., Hagiwara, M., Mizumoto, T., Suzuki, J., and Inui, K. (2022). Towards automated document revision: Grammatical error correction, fluency edits, and beyond. arXiv preprint arXiv:2205.11484. find
  103. Murray, D. L., Morris, D., Lavoie, C., Leavitt, P. R., MacIsaac, H., Masson, M. E., and Villard, M.-A. (2016). Bias in research grant evaluation has dire consequences for small universities. PLOS One, 11(6):e0155876. find
  104. Narimatsu, H., Koyama, K., Dohsaka, K., Higashinaka, R., Minami, Y., and Taira, H. (2021). Task definition and integration for scientific-document writing support. In Beltagy, I., Cohan, A., Feigenblat, G., Freitag, D., Ghosal, T., Hall, K., Herrmannova, D., Knoth, P., Lo, K., Mayr, P., Patton, R. M… find
  105. Neubig, G., Wieting, J., McCarthy, A., Stent, A., Schluter, N., and Cohn, T. (2021). ACL reviewer matching code. https://github.com/acl-org/reviewer-paper-matching. Online; accessed 21-April-2024. ↗
  106. Névéol, A., Fort, K., and Hwa, R. (2017). Report on EMNLP Reviewer Survey. https://hal.science/ hal-01660886/file/EMNLPReviewerSurvey2017.pdf. Online; accessed 21-April-2024. ↗
  107. Noothigattu, R., Shah, N., and Procaccia, A. (2021). Loss functions, axioms, and peer review. Journal of Artificial Intelligence Research, 70:1481–1515. find
  108. Nuijten, M. B. and Polanin, J. R. (2020). “statcheck”: Automatically detect statistical reporting inconsistencies to increase reproducibility of meta-analyses. Research synthesis methods, 11(5):574–579. find
  109. Parno, B., Erlingsson, U., and Enck, W. (2017). Report on the IEEE S&P 2017 submission and review process and its experiments. https://www.ieee-security.org/TC/Reports/2017/SP2017-PCChairReport.pdf. Online; accessed 21-April-2024. ↗
  110. Payan, J. and Zick, Y. (2021). I will have order! Optimizing orders for fair reviewer assignment. arXiv preprint arXiv:2108.02126. — [arXiv:2108.02126](https://arxiv.org/abs/2108.02126) find
  111. Peters, D. P. and Ceci, S. J. (1982). Peer-review practices of psychological journals: The fate of published articles, submitted again. Behavioral and Brain Sciences, 5(2):187–195. find
  112. Piniewski, M., Jarić, I., Koutsoyiannis, D., and Kundzewicz, Z. W. (2024). Emerging plagiarism in peer-review evaluation reports: a tip of the iceberg? Scientometrics, pages 1–10. find
  113. Plank, B. and van Dalen, R. (2019). CiteTracked: A longitudinal dataset of peer reviews and citations. In Proceedings of the 4th Joint Workshop on Bibliometric-enhanced Information Retrieval and Natural Language Processing for Digital Libraries (BIRNDL 2019) co-located with the 42nd International AC… find
  114. Pride, D. and Knoth, P. (2020). An authoritative approach to citation classification. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, JCDL ’20, page 337–340, New York, NY, USA. Association for Computing Machinery. find
  115. Priya, P., Firdaus, M., and Ekbal, A. (2024). Computational politeness in natural language processing: A survey. ACM Computing Surveys.
  116. ProPublica (2016). Machine bias. https://www.propublica.org/article/ machine-bias-risk-assessments-in-criminal-sentencing. Accessed: 2024-05-03. ↗
  117. Purkayastha, S., Lauscher, A., and Gurevych, I. (2023). Exploring Jiu-Jitsu argumentation for writing peer review rebuttals. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14479–14495. Association for Computational Linguistics. find
  118. Rastogi, C., Song, X., Jin, Z., Stelmakh, I., Daumé III, H., Zhang, K., and Shah, N. B. (2024). A randomized controlled trial on anonymizing reviewers to each other in peer review discussions. arXiv preprint arXiv:2403.01015. — [arXiv:2403.01015](https://arxiv.org/abs/2403.01015) find
  119. Rastogi, C., Stelmakh, I., Shen, X., Meila, M., Echenique, F., Chawla, S., and Shah, N. (2022). To ArXiv or not to ArXiv: A study quantifying pros and cons of posting preprints online. arXiv preprint arXiv:2203.17259. — [arXiv:2203.17259](https://arxiv.org/abs/2203.17259) find
  120. Régner, I., Thinus-Blanc, C., Netter, A., Schmader, T., and Huguet, P. (2019). Committees with implicit biases promote fewer women when they do not believe gender bias exists. Nature Human Behaviour, 3(11):1171–1179. find
  121. Robertson, Z. (2023). GPT-4 is slightly helpful for peer-review assistance: A pilot study. arXiv preprint arXiv:2307.05492. — [arXiv:2307.05492](https://arxiv.org/abs/2307.05492) find
  122. Rogers, A. and Augenstein, I. (2020). What can we do to improve peer review in NLP? In Cohn, T., He, Y., and Liu, Y., editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1256–1262. Association for Computational Linguistics. find
  123. Rogers, A., Baldwin, T., and Leins, K. (2021). ‘Just what do you think you’re doing, Dave?’ A checklist for responsible data use in NLP. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t., editors, Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4821–4833, Punta C… find
  124. Rogers, A., Boyd-Graber, J., and Okazaki, N. (2023a). find
  125. ACL 2023 policy on AI writing assistance. https://2023.aclweb.org/blog/ACL-2023-policy. Online; accessed 17-April-2024. ↗
  126. Rogers, A., Karpinska, M., Boyd-Graber, J., and Okazaki, N. (2023b). Program chairs’ report on peer review at ACL 2023. In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 40–… find
  127. Roos, M., Rothe, J., Rudolph, J., Scheuermann, B., and Stoyan, D. (2012). A statistical approach to calibrating the scores of biased reviewers: The linear vs. the nonlinear model. In Multidisciplinary Workshop on Advances in Preference Handling. find
  128. Ross, J. S., Gross, C. P., Desai, M. M., Hong, Y., Grant, A. O., Daniels, S. R., Hachinski, V. C., Gibbons, R. J., Gardner, T. J., and Krumholz, H. M. (2006). Effect of Blinded Peer Review on Abstract Acceptance. Journal of the American Medical Association, 295(14):1675–1680. find
  129. Ross-Hellauer, T. (2017). What is open peer review? A systematic review [version 2; peer review: 4 approved]. F1000Research, 6(588). find
  130. Santu, S. K. K., Sinha, S. K., Bansal, N., Knipper, A., Sarkar, S., Salvador, J., Mahajan, Y., Guttikonda, S., Akter, M., Freestone, M., and Jr, M. C. W. (2024). Prompting LLMs to compose meta-review drafts from peer-review narratives of scholarly manuscripts. arXiv preprint arXiv:2402.15589. — [arXiv:2402.15589](https://arxiv.org/abs/2402.15589) find
  131. Saveski, M., Jecmen, S., Shah, N., and Ugander, J. (2023). Counterfactual evaluation of peer-review assignment policies. In Oh, A., Neumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors, Advances in Neural Information Processing Systems, volume 36, pages 58765–58786. Curran Asso… find
  132. Schintler, L. A., McNeely, C. L., and Witte, J. (2023). A critical examination of the ethics of AI-mediated peer review. arXiv preprint arXiv:2309.12356. — [arXiv:2309.12356](https://arxiv.org/abs/2309.12356) find
  133. Schuff, H., Vanderlyn, L., Adel, H., and Vu, N. T. (2023). How to do human evaluation: A brief introduction to user studies in NLP. Natural Language Engineering, 29(5):1199–1222. find
  134. Shah, D. S., Schwartz, H. A., and Hovy, D. (2020). Predictive biases in naural language processing models: A conceptual framework and overview. In Jurafsky, D., Chai, J., Schluter, N., and Tetreault, J., editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics… find
  135. Shah, N. B. (2022). Challenges, experiments, and computational solutions in peer review. Communications of the ACM, 65(6):76–87. find
  136. Shah, N. B. (2023). The role of author identities in peer review. PLOS One, 18(6):e0286206. find
  137. Shah, N. B., Tabibian, B., Muandet, K., Guyon, I., and Von Luxburg, U. (2018). Design and analysis of the NIPS 2016 review process. Journal of Machine Learning Research, 19(49):1–34. find
  138. Shen, C., Cheng, L., Zhou, R., Bing, L., You, Y., and Si, L. (2022). MReD: A meta-review dataset for structure-controllable text generation. In Muresan, S., Nakov, P., and Villavicencio, A., editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 2521–2535, Dublin, Irelan… find
  139. Smith, R. (2010). Classical peer review: An empty gun. Breast Cancer Research, 12(4):S13. find
  140. Stelmakh, I., Rastogi, C., Shah, N. B., Singh, A., and Daumé III, H. (2023a). A large scale randomized controlled trial on herding in peer-review discussions. PLOS One, 18(7):e0287443. find
  141. Stelmakh, I., Shah, N., and Singh, A. (2021a). PeerReview4All: Fair and accurate reviewer assignment in peer review. Journal of Machine Learning Research, 22(163):1–66. find
  142. Stelmakh, I., Shah, N. B., and Singh, A. (2021b). Catch me if I can: Detecting strategic behaviour in peer assessment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4794–4802. find
  143. Stelmakh, I., Shah, N. B., Singh, A., and Daumé, H. (2021c). Prior and prejudice: The novice reviewers’ bias against resubmissions in conference peer review. Proceedings of the ACM on Human-Computer Interaction, 5(CSCW1). find
  144. Stelmakh, I., Shah, N. B., Singh, A., and Daumé III, H. (2021d). A novice-reviewer experiment to address scarcity of qualified reviewers in large conferences. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4785–4793. find
  145. Stelmakh, I., Wieting, J., Neubig, G., and Shah, N. B. (2023b). A gold standard dataset for the reviewer assignment problem. arXiv preprint arXiv:2303.16750. find
  146. Strauss, D., Gran-Ruaz, S., Osman, M., Williams, M. T., and Faber, S. C. (2023). Racism and censorship in the editorial and peer review process. Frontiers in Psychology, 14. find
  147. Sun, L., Chan, A., Chang, Y. S., and Dow, S. P. (2024). ReviewFlow: Intelligent scaffolding to support academic peer reviewing. arXiv preprint arXiv:2402.03530. — [arXiv:2402.03530](https://arxiv.org/abs/2402.03530) find
  148. Syed, S., Völske, M., Potthast, M., Lipka, N., Stein, B., and Schütze, H. (2018). Task proposal: The TL;DR challenge. In Krahmer, E., Gatt, A., and Goudbeek, M., editors, Proceedings of the 11th International Conference on Natural Language Generation, pages 318–321, Tilburg University, The Netherlan… find
  149. Teplitskiy, M., Ranub, H., Grayb, G. S., Meniettid, M., Guinan, E. C., and Lakhani, K. R. (2019). Social influence among experts: Field experimental evidence from peer review. find
  150. Teufel, S., Siddharthan, A., and Tidhar, D. (2006). Automatic classification of citation function. In Jurafsky, D. and Gaussier, E., editors, Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, pages 103–110, Sydney, Australia. Association for Computational Lingui…
  151. Thorn Jakobsen, T. and Rogers, A. (2022). What factors should paper-reviewer assignments rely on? community perspectives on issues and ideals in conference peer-review. In Carpuat, M., de Marneffe, M.-C., and Meza Ruiz, I. V., editors, Proceedings of the 2022 Conference of the North American Chapter… find
  152. Tomkins, A., Zhang, M., and Heavlin, W. D. (2017). Reviewer bias in single- versus double-blind peer review. Proceedings of the National Academy of Sciences, 114(48):12708–12713. van Rooyen, S., Black, N., and Godlee, F. (1999). Development of the review quality instrument (RQI) for assessing peer r… find
  153. Vervloet, M., Linn, A. J., van Weert, J. C., De Bakker, D. H., Bouvy, M. L., and Van Dijk, L. (2012). The effectiveness of interventions using electronic reminders to improve adherence to chronic medication: a systematic review of the literature. Journal of the American Medical Informatics Associati… find
  154. Vijaykumar, T. N. (2020). Potential organized fraud in ACM/IEEE computer architecture conferences. https://medium.com/@tnvijayk/ potential-organized-fraud-in-acm-ieee-computer-architecture-conferences-ccd61169370d. Online; accessed 17-April-2024. ↗
  155. Wadden, D., Lin, S., Lo, K., Wang, L. L., van Zuylen, M., Cohan, A., and Hajishirzi, H. (2020). Fact or fiction: Verifying scientific claims. In Webber, B., Cohn, T., He, Y., and Liu, Y., editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7…
  156. Wadden, D., Lo, K., Kuehl, B., Cohan, A., Beltagy, I., Wang, L. L., and Hajishirzi, H. (2022). SciFact-Open: Towards open-domain scientific claim verification. In Goldberg, Y., Kozareva, Z., and Zhang, Y., editors, Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4719–473…
  157. Wang, J. and Shah, N. B. (2019). Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 864–872. find
  158. Wang, J., Stelmakh, I., Wei, Y., and Shah, N. (2021). Debiasing evaluations that are biased by evaluations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 10120–10128. find
  159. Wang, Q., Downey, D., Ji, H., and Hope, T. (2024). Scimon: Scientific inspiration machines optimized for novelty. arXiv preprint arXiv:2305.14259.
  160. Wieting, J., Gimpel, K., Neubig, G., and Berg-Kirkpatrick, T. (2019). Simple and effective paraphrastic similarity from parallel translations. In Korhonen, A., Traum, D., and Màrquez, L., editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4602–460… find
  161. Wolfram, D., Wang, P., Hembree, A., and Park, H. (2020). Open peer review: promoting transparency in open science. Scientometrics, 125(2):1033–1051. find
  162. Xiao, Z., Zhang, S., Lai, V., and Liao, Q. V. (2023). Evaluating evaluation metrics: A framework for analyzing NLG evaluation metrics using measurement theory. In Bouamor, H., Pino, J., and Bali, K., editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pag… find
  163. Xu, S. and Wan, X. (2022). PosterBot: A system for generating posters of scientific papers with neural models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 13233–13235. find
  164. Yamamoto, S., Lauscher, A., Ponzetto, S. P., Glavaš, G., and Morishima, S. (2021). Visual summary identification from scientific publications via self-supervised learning. Frontiers in Research Metrics and Analytics, 6. find
  165. Zeng, Q., Sidhu, M., Chan, H. P., Wang, L., and Ji, H. (2023). Meta-review generation with checklist-guided iterative introspection. arXiv preprint arXiv:2305.14647. — [arXiv:2305.14647](https://arxiv.org/abs/2305.14647) find
  166. Zyska, D., Dycke, N., Buchmann, J., Kuznetsov, I., and Gurevych, I. (2023). CARE: Collaborative AI-assisted reading environment. In Bollegala, D., Huang, R., and Ritter, A., editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrati… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kuznetsov-2024-natural-language-processing-peer/