Elamur Research⌕
☰
home / papers / 2024 / ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews

ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews

Paper · 2024 · Mike D'Arcy, Alexis Ross, Erin Bransom et al.

Summary

Turning reviewer feedback into concrete manuscript revisions requires interpretation and domain reasoning, and there has been little data for studying this in technical writing. The authors define two tasks. In comment-edit alignment, a model decides which paragraph-level edits respond to a given review comment. In edit generation, a model writes a revision from a comment and the original paper. To support both tasks they release ARIES, built from computer science papers, reviews, author responses and revised drafts on OpenReview. A large silver training set is created automatically, relying on the fact that authors often quote reviewers in their responses and describe changes in words close to the actual edits. A smaller expert-annotated test set complements it. Across ten baselines, alignment proves hard even for GPT-4, especially when a comment relates to its edit only indirectly. In generation, GPT-4 produces fluent, on-topic text but follows the literal wording of the feedback rather than its intent, and it includes fewer technical details than the authors' own edits. The resource offers a demanding testbed for LLM-based writing assistants.

Key points

  • ARIES contains about 3.9K review comments automatically linked to edits with 92% precision, plus a gold test set of 196 human-annotated comments with high inter-annotator agreement. The source pool started from 6,501 OpenReview papers that had a post-review revision.
  • On alignment, the best model (GPT-4) reaches only 27.0 micro-F1, while humans reach 70.7.
  • In 94% of cases, GPT-4 edits are highly compliant with the reviewer's request, versus 68% of real author edits. Real authors sometimes push back or argue the feedback is mistaken.
  • GPT-4 edits often paraphrase the comment (48% vs. 4% for real edits) and contain technical details less often (38% vs. 53%).
  • Limitations: the data covers English AI papers only, the experiments depend on proprietary LLMs, and the generation setup cannot access the authors' experimental results or code.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States304 works · 106,138 citations🎓Massachusetts Institute of TechnologyCambridge · United States243 works · 76,506 citations🎓Hebrew University of JerusalemJerusalem · Israel69 works · 9,594 citations🎓Northwestern UniversityEvanston · United States46 works · 2,850 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Arts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksLiterature and Literary Theory · 541 worksHistory and Philosophy of Science · 62 works

Cite

Cite as Mike D'Arcy, Alexis Ross, Erin Bransom, Bailey Kuehl, Jonathan Bragg, Tom Hope and Doug Downey. 2024. ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-long.377

@inproceedings{darcy2024aries,
  title = {{ARIES: A Corpus of Scientific Paper Edits Made in Response to Peer Reviews}},
  author = {Mike D'Arcy, Alexis Ross, Erin Bransom, Bailey Kuehl, Jonathan Bragg, Tom Hope and Doug Downey},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.377},
  url = {https://doi.org/10.18653/v1/2024.acl-long.377}
}

References

35 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2008–2023. Grobid. kermitt2/grobid. https://github.com/ ↗
  2. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. Marg: Multi-agent review generation for scientific papers.
  3. Wanyu Du, Zae Myung Kim, Vipul Raheja, Dhruv Kumar, and Dongyeop Kang. 2022a. Read, Revise, Repeat: A System Demonstration for Human-in-the-loop Iterative Text Revision. In Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), pages 96–108, Dublin, Ir… find
  4. Wanyu Du, Vipul Raheja, Dhruv Kumar, Zae Myung Kim, Melissa Lopez, and Dongyeop Kang. 2022b. Understanding Iterative Revision from Human-Written Text. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3573–3590, Dublin, Ireland.… find
  5. Felix Faltings, Michel Galley, Gerold Hintz, Chris Brockett, Chris Quirk, Jianfeng Gao, and Bill Dolan. 2021. Text Editing by Command. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5259–5274, O… find
  6. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing. _eprint: 2111.09543.
  7. Takumi Ito, Tatsuki Kuribayashi, Masatoshi Hidaka, Jun Suzuki, and Kentaro Inui. 2020. Langsmith: An Interactive Academic Text Revision System. arXiv:2010.04332 [cs]. ArXiv: 2010.04332. find
  8. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2126–2138, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  9. Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. 2020. Neural crf model for sentence alignment in text simplification. In Proceedings of the Association for Computational Linguistics (ACL).
  10. Eric Malmi, Sebastian Krause, Sascha Rothe, Daniil Mirylenka, and Aliaksei Severyn. 2019. Encode, Tag, Realize: High-Precision Text Editing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proces…
  11. Chao Jiang, Wei Xu, and Samuel Stevens. 2022. arXivEdits: Understanding the human revision process in scientific writing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9420–9435, Abu Dhabi, United Arab Emirates. Association for Computational Linguis… find
  12. Omid Kashefi, Tazin Afrin, Meghan Dale, Christopher Olshefski, Amanda Godley, Diane Litman, and Rebecca Hwa. 2022. ArgRewrite V.2: an Annotated Argumentative Revisions Corpus. Language Resources and Evaluation, 56(3):881–915. ArXiv:2206.01677 [cs]. — [arXiv:2206.01677](https://arxiv.org/abs/2206.01677) find
  13. Zae Myung Kim, Wanyu Du, Vipul Raheja, Dhruv Kumar, and Dongyeop Kang. 2022. Improving Iterative Text Revision by Learning Where to Edit from Other Revision Tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9986–9999, Abu Dhabi, United Arab Emira… find
  14. Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
  15. Karen Kukich. 1992. Techniques for automatically correcting words in text. ACM Computing Surveys, 24(4):377–439. find
  16. Ilia Kuznetsov, Jan Buchmann, Max Eichler, and Iryna Gurevych. 2022. Revise and Resubmit: An Intertextual Model of Text-based Collaboration in Peer Review. Computational Linguistics, 48(4):949–986.
  17. Yixin Liu, Budhaditya Deb, Milagro Teruel, Aaron Halfaker, Dragomir Radev, and Ahmed H. Awadallah. 2022. On Improving Summarization Factual Consistency from Natural Language Feedback. ArXiv:2212.09968 [cs]. find
  18. Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The Semantic Scholar Open Research Corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4969–4983, Online. Association for Computational Linguistics.
  19. Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. 2022. EdiT5: Semi-Autoregressive Text Editing with T5 Warm-Start. find
  20. Masato Mita, Keisuke Sakaguchi, Masato Hagiwara, Tomoya Mizumoto, Jun Suzuki, and Kentaro Inui. 2022. Towards Automated Document Revision: Grammatical Error Correction, Fluency Edits, and Beyond. ArXiv:2205.11484 [cs]. find
  21. OpenAI. 2023. Gpt-4 technical report.
  22. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Te-jani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and… — PyTorch: An Imperative Style, High-Performance Deep Learning Library
  23. Vipul Raheja, Dhruv Kumar, Ryan Koo, and Dongyeop Kang. 2023. CoEdIT: Text Editing by Task-Specific Instruction Tuning. ArXiv:2305.09857 [cs]. find
  24. Machel Reid and Graham Neubig. 2022. Learning to model editing processes. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3822–3832, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  25. Emily Reif, Daphne Ippolito, Ann Yuan, Andy Coenen, Chris Callison-Burch, and Jason Wei. 2022. A Recipe for Arbitrary Text Style Transfer with Large Language Models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 837– 848, D…
  26. Stephen Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4):333–389. Publisher: Now Publishers Inc. Hanover, MA, USA. find
  27. Timo Schick, Jane Dwivedi-Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel. 2022. PEER: A Collaborative Language Model. ArXiv:2208.11663 [cs].
  28. Amanpreet Singh, Mike D’Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. 2023. SciRepEval: A multi-format benchmark for scientific document representations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5548–5566, Singapore. Association for Compu… find
  29. Yu Wang, Yuelin Wang, Kai Dang, Jie Liu, and Zhuo Liu. 2021. A Comprehensive Survey of Grammatical Error Correction. ACM Transactions on Intelligent Systems and Technology, 12(5):65:1–65:51. find
  30. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing
  31. Michihiro Yasunaga, Jure Leskovec, and Percy Liang. 2022. LinkBERT: Pretraining Language Models with Document Links. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8003–8016, Dublin, Ireland. Association for Computational Lin…
  32. Ann Yuan, Andy Coenen, Emily Reif, and Daphne Ippolito. 2022. Wordcraft: Story Writing With Large Language Models. In 27th International Conference on Intelligent User Interfaces, IUI ’22, pages 841– 852, New York, NY, USA. Association for Computing Machinery. find
  33. Fan Zhang, Homa B. Hashemi, Rebecca Hwa, and Diane Litman. 2017. A Corpus of Annotated Revisions for Studying Argumentative Writing. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1568–1578, Vancouver, Canada. Association for… find
  34. Radim Řehůřek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks, pages 45–50, Valletta, Malta. ELRA. find
  35. A To perform initial alignment of drafts, we create a mapping m(j) → i from a paragraph t j in the target document to paragraph s i in the source document. For each pair, we score the similarity as sim(i, j) = bigram(s i , t j ) − \|m(j − 1) + 1 − i\| \|S\| find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/arcy-2024-aries-corpus-scientific-paper-edits/