Elamur Research⌕
☰
home / papers / 2018 / A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

Summary

Quantitative research on scientific peer review has been difficult because reviews are usually visible only to journal editors and program chairs. This paper introduces PeerRead v1, the first openly released corpus of peer reviews intended for research. The authors drew on three sources:

  • opt-in drafts and reviews collected through Softconf for CoNLL 2016 and ACL 2017;
  • publicly posted reviews from NIPS 2013–2017 and from ICLR 2017 via OpenReview;
  • arXiv preprints from 2007–2017, automatically labeled as accepted or probably-rejected by matching them against top NLP, ML and AI venues.

In total, the resource contains about 14.7K papers with decisions and 10.7K expert reviews covering roughly 3K of them. The authors also manually annotated 1.3K ICLR reviews with aspect scores. They analyze trends in the reviews and propose two new tasks: predicting whether a paper is accepted, and predicting aspect scores from paper and review text. Simple baselines reduce acceptance-prediction error by 6–21% relative to a reject-all majority baseline. They also slightly beat a mean baseline for high-variance aspects. The dataset makes it possible to study review consistency and bias and to build reviewing aids.

Key points

  • Composition by section: NIPS 2013–2017 has 2,420 accepted papers with 9,152 reviews. ICLR 2017 has 427 papers with 1,304 reviews. ACL 2017 has 137 papers with 275 reviews. CoNLL 2016 has 22 papers with 39 reviews. arXiv 2007–2017 has 11,778 papers with no reviews (2,891 accepted, 8,887 probably-rejected).
  • arXiv acceptance labels: These combine the DBLP matching of Sutton and Gong (2017) with title matching at a relative Levenshtein distance below 0.1. In a sanity check of 364 papers whose arXiv comments mention acceptance, 352 had been labeled accepted. Only one of the remaining papers turned out to be a true false negative.
  • Data format and splits: Each section is split 0.9/0.05/0.05 into train, dev and test sets. Paper text is extracted with Science Parse and stored as json.
  • Annotation feasibility: A pilot on reviews with known scores showed that, using only the review text, annotators were able to recover roughly 50% of the aspect scores. Their scores differed from reviewers' gold scores by 0.51 on average on a 1–5 scale.
  • Analysis findings: The overall recommendation correlates strongly with recommending an oral presentation. Having an appendix is associated with a higher acceptance rate.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

104 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 101 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1149

@inproceedings{kang2018dataset,
  title = {{A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications}},
  author = {Kang, Dongyeop and Ammar, Waleed and Dalvi, Bhavana and Zuylen, Madeleine van and Kohlmeier, Sebastian and Hovy, Eduard and Schwartz, Roy},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1149},
  url = {https://doi.org/10.18653/v1/n18-1149}
}

References

24 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Vikas Ganjigunte Ashok, Song Feng, and Yejin Choi. 2013. Success with style: Using writing style to predict the success of novels. In Proc. of EMNLP. pages 1753–1764. find
  2. Vipin Balachandran. 2013. Reducing human effort and improving quality in peer code reviews using automatic static analysis and reviewer recommendation. In Proc. of ICSE. find
  3. Chandra Bhagavatula, Sergey Feldman, Russell Power, and Waleed Ammar. 2018. Content-based citation recommendation. In Proc. of NAACL. find
  4. Aliaksandr Birukou, Joseph R. Wakeling, Claudio Bartolini, Fabio Casati, Maurizio Marchese, Katsiaryna Mirylenka, Nardine Osman, Azzurra Ragone, Carles Sierra, and Aalam Wassef. 2011. Alternatives to peer review: Novel approaches for research evaluation. In Front. Comput. Neurosci.. find
  5. Andrea Bonaccorsi, Antonio Ferrara, and Marco Mal-garini. 2018. Journal ratings as predictors of article quality in arts, humanities, and social sciences: An analysis based on the italian research evaluation exercise. In The Evaluation of Research in Social Sciences and Humanities, Springer, pages 2… find
  6. Pali UK De Silva and Candace K Vance. 2017. Preserving the quality of scientific research: Peer review of research articles. In Scientific Scholarly Communication, Springer, pages 73–99. find
  7. Ferric C Fang, Anthony Bowen, and Arturo Casadevall. 2016. NIH peer review percentile scores are poorly predictive of grant productivity. eLife . find
  8. Sarah Greaves, Joanna Scott, Maxine Clarke, Linda Miller, Timo Hannay, Annette Thomas, and Philip Campbell. 2006. Nature’s trial of open peer review. Nature 444(971):10–1038. find
  9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation 9(8):1735– 1780. find
  10. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proc. of ACL-IJCNLP. volume 1, pages 1681– 1691. find our card
  11. John Langford and Mark Guzdial. 2015. The arbitrariness of reviews, and advice for school administrators. Communications of the ACM Blog 58(4):12– 13. find
  12. Seán G Roberts and Tessa Verhoef. 2016. Double-blind reviewing at evolang 11 reveals gender bias. Journal of Language Evolution 1(2):163–167. find
  13. Charles Sutton and Linan Gong. 2017. Popularity of arxiv.org within computer science. ArXiv:1710.05225. — [arXiv:1710.05225](https://arxiv.org/abs/1710.05225) find
  14. Tijmen Tieleman and Geoffrey Hinton. 2012. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning 4(2):26–31. find
  15. Andrew Tomkins, Min Zhang, and William D Heavlin. 2017. Single versus double blind reviewing at wsdm 2017. ArXiv:1702.00502. — [arXiv:1702.00502](https://arxiv.org/abs/1702.00502) find
  16. Oren Tsur and Ari Rappoport. 2009. Revrank: A fully unsupervised algorithm for selecting the most helpful book reviews. In Proc. of ICWSM. find
  17. E. Walsh, Michael W Rooney, Louis Appleby, and Greg Wilkinson. 2000. Open peer review: a randomised controlled trial. The British journal of psychiatry : the journal of mental science 176:47–51. find
  18. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proc. of NIPS. find our card
  19. Claire Le Goues, Yuriy Brun, Sven Apel, Emery Berger, Sarfraz Khurshid, and Yannis Smaragdakis. 2017. Effectiveness of anonymization in double-blind review. ArXiv:1709.01609. — [arXiv:1709.01609](https://arxiv.org/abs/1709.01609) find
  20. Vladimir I. Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Soviet physics doklady 10(8):707–710. find
  21. Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Édouard Duchesnay. 2011. Scikit-learn:… find
  22. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP). pages 1532–1543. http://www.aclweb.org/anthology/D14-1162. ↗
  23. Azzurra Ragone, Katsiaryna Mirylenka, Fabio Casati, and Maurizio Marchese. 2011. A quantitative analysis of peer review. In Proc. of ISSI. find
  24. Drummond Rennie. 2016. Make peer review scientific: thirty years on from the first congress on peer review, drummond rennie reflects on the improvements brought about by research into the process– and calls for more. Nature 535(7610):31–34. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kang-2018-dataset-peer-reviews-peerread-collection/