Elamur Research⌕
☰
home / papers / 2024 / MARG: Multi-Agent Review Generation for Scientific Papers

MARG: Multi-Agent Review Generation for Scientific Papers

Paper · 2024 · Mike D'Arcy, Tom Hope, Larry Birnbaum et al.

Summary

Large language models struggle to critique full scientific papers: these texts are long and technical, and they often exceed or underuse a model's context window. The authors present MARG, which turns GPT-4 into a team of cooperating agents. The paper is split at paragraph boundaries into chunks, and each worker agent holds one chunk. A leader agent coordinates the work by broadcasting messages and collecting replies, so the whole text stays available without truncation or summarization. The strongest variant, MARG-S, adds expert agents for three comment types (experiments, clarity, impact). A further multi-agent pass refines each draft comment or discards it. In a user study, MARG-S produced more than twice as many useful comments per paper as a single-agent GPT-4 baseline, and its comments were far less generic. It also beat the GPT-4 method of Liang et al. (2023). However, most generated comments were still judged poor, and the system is costly. LLM-generated peer feedback is therefore promising but not yet reliable.

Key points

  • MARG uses three agent roles: a leader, workers holding paper chunks, and optional experts. The leader exchanges natural-language messages with the others. Figures and tables are not included in the input. Code is available at github.com/allenai/marg-reviewer.
  • In the user study, MARG-S averaged 3.7 good comments per paper. The single-agent baseline averaged 1.7 (a 2.2x gain), and the Liang et al. (2023) approach averaged 0.3.
  • The share of generic comments fell from 60% to 29%, and 71% of MARG-S comments were rated specific. In automatic alignment-based evaluation, MARG-S gained 6.1 recall points over the strongest baseline.
  • Weaknesses remain:
    • Across all methods, most comments were rated bad, and 38–48% were judged highly inaccurate.
    • The refinement stage missed invalid comments in 47% of analyzed cases, often because information was lost in PDF parsing or ignored by the model.
  • Proposed future work includes cheaper models for simpler sub-tasks, use of background literature, and compressing inter-agent messages to handle longer inputs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States304 works · 106,138 citations🎓Hebrew University of JerusalemJerusalem · Israel69 works · 9,594 citations🎓Northwestern UniversityEvanston · United States46 works · 2,850 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 worksHistory and Philosophy of Science · 62 works

Cite

Cite as Mike D'Arcy, Tom Hope, Larry Birnbaum and Doug Downey. 2024. MARG: Multi-Agent Review Generation for Scientific Papers. arXiv:2401.04259.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.04259 · arXiv 2401.04259

@misc{darcy2024marg,
  title = {{MARG: Multi-Agent Review Generation for Scientific Papers}},
  author = {Mike D'Arcy, Tom Hope, Larry Birnbaum and Doug Downey},
  year = {2024},
  doi = {10.48550/arxiv.2401.04259},
  eprint = {2401.04259},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.04259}
}

References

39 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zafar Ali, Pavlos Kefalas, Khan Muhammad, Bahadar Ali, and Muhammad Imran. 2020. Deep learning in citation recommendation models survey. Expert Systems with Applications, 162:113790. find
  2. Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. 2023. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. ArXiv:2308.14508 [cs].
  3. Setio Basuki and Masatoshi Tsuchiya. 2022. The Quality Assist: A Technology-Assisted Peer Review Based on Citation Functions to Predict the Paper Quality. IEEE Access, 10:126815–126831. Conference Name: IEEE Access. find
  4. Douglas Bates, Martin Maechler, Ben Bolker, Steven Walker, Rune Haubo Bojesen Christensen, Henrik Singmann, Bin Dai, Fabian Scheipl, Gabor Grothendieck, Peter Green, and others. 2009. Package ‘lme4’. URL http://lme4.r-forge. r-project. org. ↗
  5. Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The Long-Document Transformer.
  6. Karim Benharrak, Tim Zindulka, Florian Lehmann, Hendrik Heuer, and Daniel Buschek. 2023. Writer-Defined AI Personas for On-Demand Feedback Generation. ArXiv:2309.10433 [cs]. — [arXiv:2309.10433](https://arxiv.org/abs/2309.10433) find
  7. Prabhat Kumar Bharti, Tirthankar Ghosal, Mayank Agarwal, and Asif Ekbal. 2023. PEERRec: An AI-based approach to automatically generate recommendations and predict decisions in peer review. International Journal on Digital Libraries. find
  8. Rune Haubo Bojesen Christensen. 2015. Package ‘ordinal’. Stand, 19(2016). Publisher: Citeseer. find
  9. Mike D’Arcy, Alexis Ross, Erin Bransom, Bailey Kuehl, Jonathan Bragg, Tom Hope, and Doug Downey. 2023. Aries: A corpus of scientific paper edits made in response to peer reviews.
  10. Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2023. Improving Factuality and Reasoning in Language Models through Multiagent Debate. ArXiv:2305.14325 [cs].
  11. Nell K Duke and P David Pearson. 2009. Effective practices for developing reading comprehension. Journal of education, 189(1-2):107–122. Publisher: SAGE Publications Sage CA: Los Angeles, CA. find
  12. Linda G. Fielding and And Others. 1990. How Discussion Questions Influence Children’s Story Understanding. Technical Report No. 490. Technical report. ERIC Number: ED314724. find
  13. Raymond Fok and Daniel S Weld. 2023. What can’t large language models do? the future of ai-assisted academic writing. In In2Writing Workshop at CHI 2023. find
  14. Frederic Gmeiner and Nur Yildirim. 2023. Dimensions for Designing LLM-based Writing Support. In In2Writing Workshop at CHI 2023. find
  15. Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. 2023. MetaGPT: Meta Programming for Multi-Agent Collaborative Framework. ArXiv:2308.00352 [cs].
  16. Maor Ivgi, Uri Shaham, and Jonathan Berant. 2023. Efficient Long-Text Understanding with Short-Text Models. Transactions of the Association for Computational Linguistics, 11:284–299.
  17. A. U. Kalnins, K. Halm, and M. Castillo. 2015. Screening for Self-Plagiarism in a Subspecialty-versus-General Imaging Journal Using iThenticate. American Journal of Neuroradiology, 36(6):1034–1038. Publisher: American Journal of Neuroradiology Section: Editorial Perspectives. find
  18. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The Efficient Transformer.
  19. Kayvan Kousha and Mike Thelwall. 2023. Artificial intelligence to support publishing and peer review: A summary and review. Learned Publishing, n/a(n/a). _eprint: https://onlinelibrary.wiley.com/doi/ pdf/10.1002/leap.1570. — [doi:10.1002/leap.1570](https://doi.org/10.1002/leap.1570) ↗
  20. Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023. CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society. ArXiv:2303.17760 [cs].
  21. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Smith, Yian Yin, Daniel McFarland, and James Zou. 2023. Can large language models provide useful feedback on research papers? A large-scale empirical analysis. ArXiv:2310.01783 [cs].
  22. Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. Lost in the Middle: How Language Models Use Long Contexts. ArXiv:2307.03172 [cs].
  23. Ryan Liu and Nihar B. Shah. 2023. ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewing. ArXiv:2306.00622 [cs].
  24. Cerstin Mahlow. 2023. Writing Tools: Looking Back to Look Ahead. In In2Writing Workshop at CHI 2023. ArXiv:2303.17894 [cs]. — [arXiv:2303.17894](https://arxiv.org/abs/2303.17894) find
  25. Michèle B. Nuijten and Joshua R. Polanin. 2020. “statcheck”: Automatically detect statistical reporting inconsistencies to increase reproducibility of meta-analyses. Research Synthesis Methods, 11(5):574– 579. find
  26. OpenAI. 2023. Gpt-4 technical report.
  27. Afshin Oroojlooy and Davood Hajinezhad. 2022. A review of cooperative multi-agent deep reinforcement learning. Applied Intelligence, 53(11):13677–13722. find
  28. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents: Interactive Simulacra of Human Behavior. ArXiv:2304.03442 [cs]. find
  29. Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. 2023. Retrieval meets Long Context Large Language Models. ArXiv:2310.03025 [cs].
  30. Guanghui Qin, Yukun Feng, and Benjamin Van Durme. 2023. The NLP Task Effectiveness of Long-Range Transformers. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 3774–3790, Dubrovnik, Croatia. Association for Computational Linguistic… find
  31. Zejiang Shen, Tal August, Pao Siangliulue, Kyle Lo, Jonathan Bragg, Jeff Hammerbacher, Doug Downey, Joseph Chee Chang, and David Sontag. 2023. Beyond summarization: Designing ai support for real-world expository writing tasks. In In2Writing Workshop at CHI 2023. find
  32. Weizhe Yuan and Pengfei Liu. 2022. KID-Review: Knowledge-Guided Scientific Review Generation with Oracle Pre-Training. In Proceedings of the First MiniCon Conference. find
  33. Qingyue Wang, Liang Ding, Yanan Cao, Zhiliang Tian, Shi Wang, Dacheng Tao, and Li Guo. 2023a. Recursively Summarizing Enables Long-Term Dialogue Memory in Large Language Models. ArXiv:2308.15022 [cs]. — [arXiv:2308.15022](https://arxiv.org/abs/2308.15022) find
  34. Qingyun Wang, Qi Zeng, Lifu Huang, Kevin Knight, Heng Ji, and Nazneen Fatema Rajani. 2020a. ReviewRobot: Explainable Paper Review Generation based on Knowledge Synthesis. arXiv:2010.06119 [cs]. ArXiv: 2010.06119. find
  35. Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020b. Linformer: Self-Attention with Linear Complexity. ArXiv:2006.04768 [cs, stat]. find
  36. Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. 2023b. Unleashing Cognitive Synergy in Large Language Models: A Task-Solving Agent through Multi-Persona Self-Collaboration. ArXiv:2307.05300 [cs].
  37. Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. 2021. Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms. In Kyriakos G. Vamvoudakis, Yan Wan, Frank L. Lewis, and Derya Cansever, editors, Handbook of Reinforcement Learning and Control, Studies in Systems, Decision and… find
  38. Matthew Welsh, Daniel Navarro, and Steve Begg. 2011. Number Preference, Precision and Implicit Confidence. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 33. Issue: 33. find
  39. Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. 2021. Recursively Summarizing Books with Human Feedback. ArXiv:2109.10862 [cs].

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/arcy-2024-marg-multi-agent-review/