Elamur Research⌕
☰
home / papers / 2023 / NLPeer: A Unified Resource for the Computational Study of Peer Review

NLPeer: A Unified Resource for the Computational Study of Peer Review

Paper · 2023 · Nils Dycke, Ilia Kuznetsov and Iryna Gurevych

Summary

Computational research on peer review has been held back by three gaps. Most available review data comes from machine learning conferences on OpenReview.net and carries no explicit license. Studies rarely look beyond one research community. Each dataset also uses its own task-specific format. NLPeer addresses all three gaps.

The authors collect consent-based review data from the NLP/CL community (ARR-22, COLING-20) using the donation workflow of Dycke et al. (2022). They add a new crawl of the openly reviewed, multidisciplinary F1000Research platform (F1000-22) and the openly licensed ACL-17 and CONLL-16 portions of PeerRead. All five collections are mapped onto a single common data model. This model keeps the structure of each document intact, along with figures and other non-text content, successive paper versions and metadata, so that papers can be linked to their reviews.

The authors also supply baseline implementations for review score prediction, pragmatic labeling of reviews, and a new guided skimming task. Experiments reveal large performance differences across venues and communities, and complementarity between structured review forms and discourse labeling. The result is a legally reusable, cross-domain foundation for the field.

Key points

  • NLPeer contains 5,672 papers, 11,515 reviews and about 4.2M review tokens, spanning 2012–2022 and both blind and open reviewing systems.
  • F1000-22 is the largest component, with 4,949 papers and 10,418 reviews across fields such as medicine and science policy.
  • The NLP-domain parts are:
    • ARR-22 (476 papers, 684 reviews), new
    • COLING-20 (89 papers, 112 reviews), new and released under CC-BY-NC-SA 4.0
    • ACL-17 (136 papers) and CONLL-16 (22 papers), CC-BY data from PeerRead
  • Three assistance tasks are benchmarked: review score prediction, pragmatic labeling, and the newly proposed guided skimming for peer review.
  • The authors list several limitations:
    • The data is English-only, and arts and humanities are under-represented.
    • Donation-based collection may introduce bias.
    • Author profiling is explicitly excluded as a use.
    • Data and code are public via TUdatalib and GitHub (UKPLab/nlpeer).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Technische Universität DarmstadtDarmstadt · Germany109 works · 35,920 citations🔬Hessian Center for Artificial IntelligenceDarmstadt · Germany12 works · 529 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksHistory and Philosophy of Science · 53 works

Cite

Cite as Nils Dycke, Ilia Kuznetsov and Iryna Gurevych. 2023. NLPeer: A Unified Resource for the Computational Study of Peer Review. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.acl-long.277

@inproceedings{dycke2023nlpeer,
  title = {{NLPeer: A Unified Resource for the Computational Study of Peer Review}},
  author = {Nils Dycke, Ilia Kuznetsov and Iryna Gurevych},
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2023},
  doi = {10.18653/v1/2023.acl-long.277},
  url = {https://doi.org/10.18653/v1/2023.acl-long.277}
}

References

35 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2008–2023. Grobid. https://github.com/ kermitt2/grobid. ↗
  2. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3615–3620.
  3. Lutz Bornmann. 2011. Scientific peer review. Annual Review of Information Science and Technology, 45(1):197–245. find
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  5. Liying Cheng, Lidong Bing, Qian Yu, Wei Lu, and Luo Si. 2020. APE: Argument pair extraction from peer review and rebuttal via multi-task learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7000–7011, Online. Association for Computational… find
  6. Alexandra Chronopoulou, Matthew Peters, and Jesse Dodge. 2022. Efficient hierarchical domain adaptation for pretrained language models. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1336–1351,… find
  7. Nils Dycke, Ilia Kuznetsov, and Iryna Gurevych. 2022. Yes-yes-yes: Proactive data collection for ACL rolling review and beyond. In Findings of the Association for Computational Linguistics: Empirical Methods in Natural Language Processing 2022, pages 300–318, Abu Dhabi, United Arab Emirates. Associa… find
  8. Nils Dycke, Edwin Simpson, Ilia Kuznetsov, and Iryna Gurevych. 2021. Assisting decision making in scholarly peer review: A preference learning perspective. arXiv:2109.01190. — [arXiv:2109.01190](https://arxiv.org/abs/2109.01190) find
  9. Raymond Fok, Hita Kambhamettu, Luca Soldaini, Jonathan Bragg, Kyle Lo, Marti Hearst, Andrew Head, and Daniel S Weld. 2023. Scim: Intelligent skimming support for scientific papers. In Proceedings of the 28th International Conference on Intelligent User Interfaces, IUI ’23, page 476–490, New York, NY… find
  10. Tirthankar Ghosal, Sandeep Kumar, Prabhat Kumar Bharti, and Asif Ekbal. 2022a. Peer review analyze: A novel benchmark resource for computational analysis of peer reviews. PLOS ONE, 17(1):1–29. find
  11. Tirthankar Ghosal, Kamal Kaushik Varanasi, and Valia Kordoni. 2022b. HedgePeer: A dataset for uncertainty detection in peer reviews. In Proceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries, JCDL ’22, pages 1–5, New York, NY, USA. Association for Computing Machinery. find
  12. Tirthankar Ghosal, Rajeev Verma, Asif Ekbal, and Pushpak Bhattacharyya. 2019. DeepSentiPeer: Harnessing Sentiment in Review Texts to Recommend Peer Review Decisions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1120–1130, Florence, Italy. Associat… find
  13. Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2022. DEMix layers: Disentangling domains for modular language modeling. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies… find
  14. Xinyu Hua, Mitko Nikolov, Nikhil Badugu, and Lu Wang. 2019. Argument mining for understanding peer reviews. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2131–… find
  15. Rob Johnson, Anthony Watkinson, and Michael Mabe. 2018. The STM Report: An overview of scientific and scholarly publishing. International Association of Scientific, Technical and Medical Publishers, The Hague, Netherlands. 2020. Biobert: a pre-trained biomedical language representation model for bio…
  16. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
  17. Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021. On the stability of fine-tuning BERT: misconceptions, explanations, and strong baselines. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. find
  18. Barbara Plank. 2016. What to do about non-standard (or non-canonical) language in NLP. arXiv:1608.07836. — [arXiv:1608.07836](https://arxiv.org/abs/1608.07836) find
  19. Simon Price and Peter A Flach. 2017. Computational support for academic peer review: A perspective from artificial intelligence. Communications of the Association for Computing Machinery (ACM), 60(3):70–79. find
  20. Chenglei Qin and Chengzhi Zhang. 2022. Which structure of academic articles do referees pay more attention to?: perspective of peer review and full-text of academic articles. CoRR, abs/2209.01841. — [arXiv:2209.01841](https://arxiv.org/abs/2209.01841) find
  21. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (PeerRead): Collection, insights and NLP applications. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Comp…
  22. Anna Rogers and Isabelle Augenstein. 2020. What can we do to improve peer review in NLP? In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1256–1262, Online. ACL. find
  23. Neha Kennard, Tim O’Gorman, Rajarshi Das, Akshay Sharma, Chhandak Bagchi, Matthew Clinton, Pranay Kumar Yelugam, Hamed Zamani, and Andrew McCallum. 2022. DISAPERE: A dataset for discourse structure in peer review discussions. In Proceedings of the 2022 Conference of the North American Chapter of the… find
  24. Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing, Yang You, and Luo Si. 2022. MReD: A meta-review dataset for structure-controllable text generation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2521–2535, Dublin, Ireland. Association for Computational Linguistics. find
  25. Ilia Kuznetsov, Jan Buchmann, Max Eichler, and Iryna Gurevych. 2022. Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review. Computational Linguistics, 48(4):949–986.
  26. Carole J. Lee, Cassidy R. Sugimoto, Guo Zhang, and Blaise Cronin. 2013. Bias in peer review. Journal of the American Society for Information Science and Technology, 64(1):2–17. find
  27. Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. find
  28. Nihar B Shah. 2019. Principled Methods to Improve Peer Review. Available online at http://www.cs.cmu.edu/~nihars/publications/ survey_peerreview_niharshah.pdf (accessed on 22.01.2022). ↗
  29. Lukas Stappen, Georgios Rizos, Madina Hasan, Thomas Hain, and Björn W. Schuller. 2020. Uncertainty-aware machine support for paper reviewing on the interspeech 2019 submission corpus. In Interspeech 2020, 21st Annual Conference of the International Speech Communication Association, Virtual Event, Sh… find
  30. Ivan Stelmakh, Nihar B Shah, Aarti Singh, and Hal Daumé III. 2021. Prior and prejudice: The novice reviewers’ bias against resubmissions in conference peer review. Proceedings of the Association for Computing Machinery (ACM) on Human-Computer Interaction, 5:1–17. find
  31. Andrew Tomkins, Min Zhang, and William D. Heavlin. 2017. Reviewer bias in single- versus double-blind peer review. Proceedings of the National Academy of Sciences, 114(48):12708–12713. find
  32. Richard Walker and Pascal Rocha da Silva. 2015. Emerging trends in peer review—a survey. Frontiers in Neuroscience, 9. find
  33. Jingyan Wang and Nihar B. Shah. 2019. Your 2 is my 1, your 3 is my 9: Handling arbitrary miscalibrations in ratings. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS ’19, Montreal, QC, Canada, May 13-17, 2019, pages 864–872. International Foundat… find
  34. Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171–212. find
  35. Rongsheng Zhang, Yinhe Zheng, Xiaoxi Mao, and Minlie Huang. 2021. Unsupervised domain adaptation with adapter. arXiv:2111.00667. — [arXiv:2111.00667](https://arxiv.org/abs/2111.00667) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/dycke-2023-nlpeer-unified-resource-computational-study/