Elamur Research⌕
☰
home / papers / 2025 / Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews

Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews

Paper · 2025 · Hyungyu Shin, Jingyu Tang, Yoonjoo Lee et al.

Summary

LLMs can now draft peer reviews, but existing evaluations rely on surface overlap (BLEU, ROUGE) or content qualities such as specificity. These checks do not reveal whether a model praises and criticizes the same things human experts do. The authors propose a focus-level evaluation framework. It represents a review's focus as a normalized distribution of strength and weakness points over predefined facets, along two dimensions: target (what is discussed, e.g., problem, method, experiment) and aspect (which criterion applies, e.g., validity, clarity, novelty). They derived the facets from AI conference guidelines and earlier review-analysis work. An automatic LLM annotator labels each review point, and the resulting distributions are compared with those from ICLR expert reviews on OpenReview. They tested eight off-the-shelf LLMs, the MARG system and a fine-tuned gpt-4o. The untuned models overemphasize technical validity and largely neglect novelty when listing weaknesses, and their agreement with experts on individual points is low. Fine-tuning brings the focus distributions closest to human ones. The work offers an interpretable diagnostic of LLM review blind spots and a basis for training more balanced reviewers.

Key points

  • Benchmark data: 676 ICLR papers (2021–2024) were drawn from 18,407 collected OpenReview submissions. From these, 3,657 expert strengths and weaknesses were extracted, mainly from meta-reviews enriched with details from individual reviews.
  • Facet scheme and annotator: The scheme has 7 target facets and 5 aspect facets. The automatic annotator agreed well with humans (Cohen's kappa 0.81 for target, 0.79 for aspect).
  • Models evaluated: 8 LLMs (4 GPT, 2 Llama, 2 DeepSeek), plus MARG and a gpt-4o fine-tuned on the dataset. Even the best model matched human target/aspect labels per point with an F1 of only 0.373.
  • Different strengths across models: The fine-tuned gpt-4o had the most human-like focus distribution, while Llama-405B scored highest on text similarity. This argues for evaluating reviews from several angles.
  • Released resource: The authors released papers, expert reviews, annotated expert points, reviews from 8 LLMs, and 43,042 LLM-generated strengths and weaknesses with target/aspect labels (hosted on figshare).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee and Juho Kim. 2025. Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1805

@inproceedings{shin2025mind,
  title = {{Mind the Blind Spots: A Focus-Level Evaluation Framework for LLM Reviews}},
  author = {Hyungyu Shin, Jingyu Tang, Yoonjoo Lee, Nayoung Kim, Hyunseung Lim, Ji Yong Cho, Hwajung Hong, Moontae Lee and Juho Kim},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1805},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1805}
}

References

34 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, and Zheng Wen. 2025. Multi-objective preference optimization: Improving human alignment of generative models. Preprint, arXiv:2505.10892. — [arXiv:2505.10892](https://arxiv.org/abs/2505.10892) find
  2. Nicolas Bougie and Narimasa Watanabe. 2024. Generative adversarial reviews: When llms become the critic. ArXiv, abs/2412.10415. — [arXiv:2412.10415](https://arxiv.org/abs/2412.10415) find
  3. Souvic Chakraborty, Pawan Goyal, and Animesh Mukherjee. 2020. Aspect-based sentiment analysis of scientific reviews. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020, pages 207–216. find
  4. Eric Chamoun, Michael Schlichktrull, and Andreas Vlachos. 2024. Automated focused feedback generation for scientific writing assistance. arXiv preprint arXiv:2405.20477. find
  5. G. Choudhary, Natwar Modani, and Nitish Maurya. 2022. React: A review comment dataset for actionability (and more). ArXiv, abs/2210.00443. — [arXiv:2210.00443](https://arxiv.org/abs/2210.00443) find
  6. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1):37–46. find
  7. Mike D’Arcy, Tom Hope, Larry Birnbaum, and Doug Downey. 2024. Marg: Multi-agent review generation for scientific papers. arXiv preprint arXiv:2401.04259. find
  8. Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, et al. 2024. Llms assist nlp researchers: Critique paper (meta-) reviewing. arXiv preprint arXiv:2406.16253.
  9. Xian Gao, Jiacheng Ruan, Jingsheng Gao, Ting Liu, and Yuzhuo Fu. 2025. Reviewagents: Bridging the gap between human and ai-generated paper reviews. ArXiv, abs/2503.08506. — [arXiv:2503.08506](https://arxiv.org/abs/2503.08506) find
  10. Zhaolin Gao, Kianté Brantley, and Thorsten Joachims. 2024. Reviewer2: Optimizing review generation through prompt generation. Preprint, arXiv:2402.10886.
  11. Tirthankar Ghosal, Sandeep Kumar, Prabhat Kumar Bharti, and Asif Ekbal. 2022. Peer review analyze: A novel benchmark resource for computational analysis of peer reviews. Plos one, 17(1):e0259238. find
  12. Hugo Horta and Jisun Jung. 2024. The crisis of peer review: Part of the evolution of science. Higher Education Quarterly, page e12511. find
  13. Eftekhar Hossain, Sanjeev Kumar Sinha, Naman Bansal, Alex Knipper, Souvika Sarkar, John Salvador, Yash Mahajan, Sri Guttikonda, Mousumi Akter, Md. Mahadi Hassan, Matthew Freestone, Matthew C. Williams Jr., Dongji Feng, and Santu Karmaker. 2025. Llms as meta-reviewers’ assistants: A case study. Prepr… find
  14. Mohammad Hosseini and Serge P.J.M. Horbach. 2023. Fighting reviewer fatigue or amplifying bias? considerations and recommendations for use of chatgpt and other large language models in scholarly peer review. Research Integrity and Peer Review, 8. find
  15. Johann Füller Julian Just, Thomas Ströhle and Katja Hutter. 2024. Ai-based novelty detection in crowdsourced idea spaces. Innovation, 26(3):359–386. find
  16. Giuseppe Russo Latona, Manoel Horta Ribeiro, Tim R Davidson, Veniamin Veselovsky, and Robert West. 2024. The ai review lottery: Widespread ai-assisted peer reviews boost paper scores and acceptance rates. arXiv preprint arXiv:2405.02150. find
  17. Miao Li, Eduard H. Hovy, and Jey Han Lau. 2023. Summarizing multiple documents with conversational structure for meta-review generation. In Conference on Empirical Methods in Natural Language Processing. find
  18. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022. Holistic evaluation of language models. arXiv preprint arXiv:2211.09110. find
  19. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. 2024. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 1(8):AIoa2400196. find
  20. Ethan Lin, Zhiyuan Peng, and Yi Fang. 2024. Evaluating and enhancing large language models for novelty assessment in scholarly publications. ArXiv, abs/2409.16605. find
  21. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob N. Foerster, Jeff Clune, and David Ha. 2024. The ai scientist: Towards fully automated open-ended scientific discovery. ArXiv, abs/2408.06292.
  22. Sheng Lu, Ilia Kuznetsov, and Iryna Gurevych. 2025. Identifying aspects in peer reviews. arXiv preprint arXiv:2504.06910. find
  23. Mehrnaz Mostafapour, Jacqueline H Fortier, Karen Pacheco, Heather Murray, and Gary Garber. 2024. Evaluating literature reviews conducted by humans versus chatgpt: Comparative study. Jmir ai, 3:e56537. find
  24. Lakshmi Ramachandran, Edward F Gehringer, and Ravi K Yadav. 2017. Automated assessment of the quality of peer reviews using natural language processing techniques. International Journal of Artificial Intelligence in Education, 27(3):534–581. find
  25. Zachary Robertson. 2023. Gpt4 is slightly helpful for peer-review assistance: A pilot study. ArXiv, abs/2307.05492. — [arXiv:2307.05492](https://arxiv.org/abs/2307.05492) find
  26. Ahmed Saad, Nathan Jenko, Sisith Ariyaratne, Nick Birch, Karthikeyan P Iyengar, Arthur Mark Davies, Raju Vaishya, and Rajesh Botchu. 2024. Exploring the potential of chatgpt in the peer review process: an observational study. Diabetes & Metabolic Syndrome: Clinical Research & Reviews, 18(2):102946. find
  27. Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615.
  28. Lu Sun, Stone Tao, Junjie Hu, and Steven P. Dow. 2024. Metawriter: Exploring the potential and perils of ai writing support in scientific peer review. Proceedings of the ACM on Human-Computer Interaction, 8:1 – 32. find
  29. Carolina Tropini, B Brett Finlay, Mark Nichter, Melissa K Melby, Jessica L Metcalf, Maria Gloria Dominguez-Bello, Liping Zhao, Margaret J McFall-Ngai, Naama Geva-Zatorsky, Katherine R Amato, et al. 2023. Time to rethink academic publishing: the peer reviewer crisis. find
  30. Keith Tyser, Ben Segev, Gaston Longhitano, Xin-Yu Zhang, Zachary Meeks, Jason Lee, Uday Garg, Nicholas Belsten, Avi Shporer, Madeleine Udell, et al. 2024. Ai-driven review systems: evaluating llms in scalable and bias-aware academic reviews. arXiv preprint arXiv:2408.10365. find
  31. Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, and Jianshu Chen. 2024. Rewards-in-context: Multi-objective alignment of foundation models with dynamic preference adjustment. arXiv preprint arXiv:2402.10207. find
  32. Rui Ye, Xianghe Pang, Jingyi Chai, Jiaao Chen, Zhen fei Yin, Zhen Xiang, Xiaowen Dong, Jing Shao, and Siheng Chen. 2024. Are we there yet? revealing the risks of utilizing large language models in scholarly peer review. ArXiv, abs/2412.01708. find
  33. Weizhe Yuan, Pengfei Liu, and Graham Neubig. 2022. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171–212. find
  34. Ruiyang Zhou, Lu Chen, and Kai Yu. 2024. Is llm a reliable reviewer? a comprehensive evaluation of llm on automatic paper reviewing tasks. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 9340–9351.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/shin-2025-mind-blind-spots-focus-level/