Elamur Research⌕
☰
home / papers / 2021 / Can We Automate Scientific Reviewing?

Can We Automate Scientific Reviewing?

Paper · 2021 · Weizhe Yuan, Pengfei Liu and Graham Neubig

Summary

Peer review demands expert effort, while the number of submissions keeps growing. Yuan, Liu and Neubig explore whether current NLP models can draft first-pass reviews of scientific papers. They begin by defining a good review: drawing on conference guidelines, they list desiderata (decisiveness, comprehensiveness, justification, accuracy, kindness) and link each to manual or automatic metrics. They then build ASAP-Review, a machine learning corpus of papers paired with reviews whose content is annotated by aspect (e.g. clarity, originality). Review writing is treated as aspect-based summarization. Their ReviewAdvisor systems handle long inputs with a two-stage design: salient content is extracted from the paper, then a BART-based generator writes the review. The generated reviews address more aspects than human ones and usually describe the paper's main idea correctly. However, opinions on other aspects tend to be unconstructive and often factually wrong, and the systems show measurable biases. The authors conclude that reviewing cannot be automated yet. Their released data, code and demo could still support tools that assist human reviewers.

Key points

  • Resources: the ASAP-Review dataset (ASpect-enhAnced Peer Review, ML domain, aspect-annotated reviews), code at github.com/neulab/ReviewAdvisor, and an online ReviewAdvisor demo.
  • Metrics include Recommendation Accuracy (RAcc = paper decision × review recommendation), aspect coverage/recall, informativeness, Summary Accuracy, aspect-level constructiveness, ROUGE and BERTScore. Only some of these can be computed automatically.
  • Papers usually exceed 5,000 words, which is beyond typical Transformer/BART input limits. This motivates the extract-then-generate pipeline. SciBERT-style encoders with Transformer decoders produced less fluent output than BART.
  • Bias analysis found that humans give native English speakers higher "Clarity" scores, a gap that the systems reduce. The systems judge "Originality" more harshly for non-native authors. Both humans and systems favour non-anonymous submissions already posted on arXiv.
  • The paper outlines eight future challenges, including long-document modeling, scientific-domain seq2seq pretraining, structural and external knowledge, open fine-grained review data, better PDF parsers, fairness, and factuality and calibration.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States598 works · 191,226 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksHistory and Philosophy of Science · 62 works

Cite

Cite as Weizhe Yuan, Pengfei Liu and Graham Neubig. 2021. Can We Automate Scientific Reviewing?. arXiv:2102.00176.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2102.00176 · arXiv 2102.00176

@misc{yuan2021can,
  title = {{Can We Automate Scientific Reviewing?}},
  author = {Weizhe Yuan, Pengfei Liu and Graham Neubig},
  year = {2021},
  doi = {10.48550/arxiv.2102.00176},
  eprint = {2102.00176},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2102.00176}
}

References

73 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Stefanos Angelidis and Mirella Lapata. 2018. Summarizing opinions: Aspect extraction meets sentiment prediction and they are both weakly supervised. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3675–3686, Brussels, Belgium. Association for Computat…
  2. Omer Anjum, Hongyu Gong, Suma Bhat, Wen-Mei Hwu, and JinJun Xiong. 2019. PaRe: A paper-reviewer matching approach using a common topic space. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce… find
  3. Tal August, Lauren Kim, Katharina Reinecke, and Noah A. Smith. 2020. Writing strategies for science communication: Data and computational analysis. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5327–5344, Online. Association for Computationa… find
  4. Alberto Bartoli, Andrea De Lorenzo, Eric Medvet, and Fabiano Tarlao. 2016. Your paper has been accepted, rejected, or whatever: Automatic generation of scientific paper reviews. In International Conference on Availability, Reliability, and Security, pages 19– 28. Springer. find
  5. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676.
  6. Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings.
  7. Lutz Bornmann and Rüdiger Mutz. 2015. Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references. Journal of the Association for Information Science and Technology, 66(11):2215–2222. find
  8. Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
  9. Souvic Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2020. Aspect-based sentiment analysis of scientific reviews. arXiv preprint arXiv:2006.03257. — [arXiv:2006.03257](https://arxiv.org/abs/2006.03257) find
  10. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 675–686.
  11. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, W. Chang, and Nazli Goharian. 2018a. A discourse-aware attention model for abstractive summarization of long documents. In NAACL-HLT.
  12. Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018b. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational…
  13. Arman Cohan and Nazli Goharian. 2017. Scientific article summarization using citation-context and article’s discourse structure. arXiv preprint arXiv:1704.06619. find
  14. Nicola De Bellis. 2009. Bibliometrics and citation analysis: from the science citation index to cyber-metrics. scarecrow press. find
  15. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  16. Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2020. Gsum: A general framework for guided neural abstractive summarization. arXiv preprint arXiv:2010.08014.
  17. Bradley Efron. 1992. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics, pages 569–593. Springer. find
  18. Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel Weld. 2020a. TLDR: Extreme summarization of scientific documents. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4766–4777, Online. Association for Computational Linguistics.
  19. Shai Erera, Michal Shmueli-Scheuer, Guy Feigenblat, O. Nakash, O. Boni, Haggai Roitman, Doron Cohen, B. Weiner, Y. Mass, Or Rivlin, G. Lev, Achiya Jerbi, Jonathan Herzig, Yufang Hou, Charles Jochim, Martin Gleize, F. Bonin, and D. Konopnicki. 2019. A summarization system for scientific documents. In… find
  20. Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel S. Weld. 2020b. Tldr: Extreme summarization of scientific documents. ArXiv, abs/2004.15011.
  21. Guy Feigenblat, Haggai Roitman, Odellia Boni, and David Konopnicki. 2017. Unsupervised query-focused multi-document summarization using the cross entropy method. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’17, page 961–96… find
  22. Steven Jecmen, Hanrui Zhang, Ryan Liu, Nihar B Shah, Vincent Conitzer, and Fei Fang. 2020. Mitigating manipulation in peer review via randomized reviewer assignments. arXiv preprint arXiv:2006.16437. — [arXiv:2006.16437](https://arxiv.org/abs/2006.16437) find
  23. Lea Frermann and Alexandre Klementiev. 2019. Inducing document structure for aspect-based summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6263–6273, Florence, Italy. Association for Computational Linguistics. find
  24. Tom Jefferson, Philip Alderson, Elizabeth Wager, and Frank Davidoff. 2002a. Effects of editorial peer review: a systematic review. Jama, 287(21):2784– 2786. find
  25. Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao. 2019. Does my rebuttal matter? insights from a major NLP conference. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (… find
  26. Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. 2018. Bottom-up abstractive summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4098–4109.
  27. Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. 2020. Wikiasp: A dataset for multi-domain aspect-based summarization. Transactions of the Association for Computational Linguistics (TACL).
  28. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770– 778.
  29. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1684–1692.
  30. Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, and Debasis Ganguly. 2019. Identification of tasks, datasets, evaluation metrics, and numeric scores for scientific leaderboards construction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages… find
  31. Jia-Bin Huang. 2018. Deep paper gestalt. preprint arXiv:1812.08775. arXiv Sarthak Jain, Madeleine van Zuylen, Hannaneh Hajishirzi, and Iz Beltagy. 2020. SciREX: A challenge dataset for document-level information extraction. In Proceedings of the 58th Annual Meeting of the Association for Computation…
  32. Tom Jefferson, Elizabeth Wager, and Frank Davidoff. 2002b. Measuring the quality of editorial peer review. Jama, 287(21):2786–2790. find
  33. Rahul Jha, Amjad Abu-Jbara, and Dragomir Radev. 2013. A system for summarizing scientific topics starting from keywords. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 572–577, Sofia, Bulgaria. Association for Computational… find
  34. Rahul Jha, Reed Coke, and Dragomir R. Radev. 2015a. Surveyor: A system for generating coherent survey articles for scientific topics. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, January 25-30, 2015, Austin, Texas, USA, pages 2167–2173. AAAI Press. find
  35. Rahul Jha, Catherine Finegan-Dollak, Ben King, Reed Coke, and Dragomir Radev. 2015b. Content models for survey generation: A factoid-based evaluation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Lang… find
  36. Jian Jin, Qian Geng, Qian Zhao, and Lixue Zhang. 2017. Integrating the trend of research interest for reviewer assignment. In Proceedings of the 26th International Conference on World Wide Web Companion, pages 1233–1241. find
  37. Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz. 2018. A dataset of peer reviews (peerread): Collection, insights and nlp applications. In Meeting of the North American Chapter of the Association for Computational Linguistics (NAAC…
  38. Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  39. John Langford and Mark Guzdial. 2015. The arbitrariness of reviews, and advice for school administrators. find
  40. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. ArXiv, abs/1910.13461.
  41. Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 150–157.
  42. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, Online. Association for Computational Linguistics.
  43. Feng Qiao, Lizhen Xu, and Xiaowei Han. 2018. Modularized and attention-based recurrent convolutional neural network for automatic academic paper aspect scoring. In International Conference on Web Information Systems and Applications, pages 68–76. Springer. find
  44. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  45. Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, and T. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. ArXiv, abs/1911.05507.
  46. Yi Luan, Luheng He, Mari Ostendorf, and Hannaneh Hajishirzi. 2018. Multi-task identification of entities, relations, and coreference for scientific knowledge graph construction. arXiv preprint arXiv:1808.09602.
  47. Anna Rogers and Isabelle Augenstein. 2020. What can we do to improve peer review in NLP? In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1256–1262, Online. Association for Computational Linguistics. find
  48. Emaad Manzoor and Nihar B. Shah. 2020. Uncovering latent biases in text: Method and application to peer review. find
  49. Reuven Y Rubinstein and Dirk P Kroese. 2013. The cross-entropy method: a unified approach to combinatorial optimization, Monte-Carlo simulation and machine learning. Springer Science & Business Media. find
  50. Saif Mohammad, Bonnie Dorr, Melissa Egan, Ahmed Hassan, Pradeep Muthukrishan, Vahed Qazvinian, Dragomir Radev, and David Zajic. 2009. Using citations to generate surveys of scientific paradigms. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter o… find
  51. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. ArXiv, abs/1611.04230.
  52. Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don’t give me the details, just the summary! Topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium.
  53. Jennifer Nguyen, Germán Sánchez-Hernández, Núria Agell, Xari Rovira, and Cecilio Angulo. 2018. A decision support tool using order weighted averaging for conference review assignment. Pattern Recognition Letters, 105:114–120. find
  54. Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive summarization. arXiv preprint arXiv:1705.04304.
  55. Vahed Qazvinian and Dragomir R. Radev. 2008. Scientific paper summarization using citation summary networks. In Proceedings of the 22nd International Conference on Computational Linguistics - Volume 1, COLING ’08, page 689–696, USA. Association for Computational Linguistics.
  56. R. Smith. 2006. Peer review: A flawed process at the heart of science and journals. Journal of the Royal Society of Medicine, 99:178 – 182. find
  57. Gabriel Stanovsky, Noah A. Smith, and Luke Zettlemoyer. 2019. Evaluating gender bias in machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1679–1684, Florence, Italy. Association for Computational Linguistics.
  58. Ivan Stelmakh, Nihar Shah, and Aarti Singh. 2019. On testing for biases in peer review. In Advances in Neural Information Processing Systems, pages 5286– 5296. find
  59. Sandeep Subramanian, Raymond Li, Jonathan Pilault, and Christopher Pal. 2019. On extractive and abstractive neural document summarization with transformer language models. arXiv preprint arXiv:1909.03186.
  60. Albert N Tabah. 1999. Literature dynamics: Studies on growth, diffusion, and epidemics. Annual review of information science and technology (ARIST), 34:249–86. find
  61. Andrew Tomkins, Min Zhang, and William D Heavlin. 2017. Reviewer bias in single-versus double-blind peer review. Proceedings of the National Academy of Sciences, 114(48):12708–12713. find
  62. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008.
  63. Carven Von Bearnensquash. 2010. Paper gestalt. Secret Proceedings of Computer Vision and Pattern Recognition (CVPR). find
  64. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association…
  65. Ke Wang and Xiaojun Wan. 2018. Sentiment analysis of peer review texts for scholarly papers. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 175– 184. find
  66. Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. 2020. Reviewrobot: Explainable paper review generation based on knowledge synthesis. In Proceedings of INLG. find
  67. Wen Xiao and Giuseppe Carenini. 2019. Extractive summarization of long documents by combining global and local context. ArXiv, abs/1909.08089.
  68. Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020. Automatic generation of citation texts in scholarly papers: A pilot study. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6181–6190, Online. Association for Computational Linguistics. find
  69. Wenting Xiong and Diane Litman. 2011. Automatically predicting peer-review helpfulness. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 502–507, Portland, Oregon, USA. Association for Computational Linguistics. find
  70. Michihiro Yasunaga, Jungo Kasai, Rui Zhang, A. R. Fabbri, Irene Li, D. Friedman, and Dragomir R. Radev. 2019a. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In AAAI. find
  71. Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R. Fabbri, Irene Li, Dan Friedman, and Dragomir R. Radev. 2019b. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In The Thirty-Third AAAI Conference on Artificial Intellige… find
  72. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv, pages arXiv– 1904. find
  73. Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. 2018. Gender bias in coreference resolution: Evaluation and debiasing methods. arXiv preprint arXiv:1804.06876.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yuan-2021-automate-scientific-reviewing/