Elamur Research⌕
☰
home / papers / 2025 / GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation

GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation

Paper · 2025 · Tao Feng, Yihang Sun and Jiaxuan You

Summary

Using LLMs to judge research ideas is fragile. Their scores shift noticeably when the prompt is slightly reworded, they lean toward favourable verdicts, and they often miss factual errors buried inside an otherwise plausible idea.

The authors propose GraphEval, a lightweight framework that restructures this task. A small prompted LLM first splits each idea into short, self-contained viewpoints. These become nodes in a shared viewpoint-graph, with edges derived from LLM relation extraction and/or BERT similarity. Two predictors run on this graph:

  • GraphEval-LP needs no training. It spreads known quality labels to unlabelled nodes along weighted edges.
  • GraphEval-GNN is a compact graph neural network trained to predict labels. It also uses temporal node features and is trained on deliberately constructed plagiarised ideas labelled as weak, so it learns to penalise unoriginal work.

Across two datasets, GraphEval improves F1 by at least 14% over prompting and fine-tuning baselines, while keeping compute and API costs low. It can also detect plagiarised ideas. The work shows that breaking ideas into connected viewpoints makes automated review cheaper and more robust.

Key points

  • Baselines include prompted LLMs (7B and 72B), Finetuned-Bert, and prompt pipelines from Research Agent and AI Scientist. The code is available at github.com/ulab-uiuc/GraphEval.
  • On the Fact Verification dataset, GraphEval-GNN reaches 85.00% accuracy and 84.00% macro F1. GraphEval-LP scores 82.40% F1, Finetuned-Bert 68.64%, and the 72B prompted LLM 56.33%.
  • When ideas are evaluated across different years, GraphEval-GNN achieves 51.32% macro F1, compared with 31.57% for Finetuned-Bert and 11.76% for the 72B LLM. On ASAP-Review, the margin is smaller: 32.20% versus 29.86% for Finetuned-Bert.
  • In a comparison of lightweight graph models, the GNN variant (44.80% F1) clearly beats GraphEval-SGC (27.3%) and GraphEval-LightGCN (26.70%). It also outperforms a "Hybrid" relation-extraction alternative (25.46%).
  • Extracted viewpoints stay faithful to their source ideas. Entity-level factual consistency F1 is 0.9314 on ICLR Papers and 0.9232 on AI Researcher.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksInformation Systems · 1,888 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 4,145 works

Cite

Cite as Tao Feng, Yihang Sun and Jiaxuan You. 2025. GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation. arXiv:2503.12600.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2503.12600 · arXiv 2503.12600

@misc{feng2025grapheval,
  title = {{GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation}},
  author = {Tao Feng, Yihang Sun and Jiaxuan You},
  year = {2025},
  doi = {10.48550/arxiv.2503.12600},
  eprint = {2503.12600},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2503.12600}
}

References

78 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Qwen2 technical report. 2024. find
  2. Jinheon Baek, Sujay Kumar Jauhar, Silviu Cucerzan, and Sung Ju Hwang. Researchagent: Iterative research idea generation over scientific literature with large language models. arXiv preprint arXiv:2404.07738, 2024.
  3. J Jason Bell, Christian Pescher, Gerard J Tellis, and Johann Füller. Can ai help in ideation? a theory-based model for idea screening in crowdsourcing contests. Marketing Science, 43(1):54–72, 2024. find
  4. Tom B Brown. Language models are few-shot learners. arXiv preprint arXiv:2005.14165, 2020.
  5. Salvatore Carta, Alessandro Giuliani, Leonardo Piano, Alessandro Sebastian Podda, Livio Pompianu, and Sandro Gabriele Tiddia. Iterative zero-shot llm prompting for knowledge graph construction. arXiv preprint arXiv:2307.01128, 2023. — [arXiv:2307.01128](https://arxiv.org/abs/2307.01128) find
  6. Rosemary A Cowell, Morgan D Barense, and Patrick S Sadil. A roadmap for understanding memory: Decomposing cognitive processes into operations and representations. Eneuro, 6(4), 2019. find
  7. Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.
  8. P Kingma Diederik. Adam: A method for stochastic optimization. (No Title), 2014.
  9. Katinka Dijkstra, Anita Eerland, Josjan Zijlmans, and Lysanne S Post. Embodied cognition, abstract concepts, and the benefits of new technology for implicit body manipulation. Frontiers in psychology, 5:757, 2014. find
  10. Jiangshu Du, Yibo Wang, Wenting Zhao, Zhongfen Deng, Shuaiqi Liu, Renze Lou, Henry Peng Zou, Pranav Narayanan Venkit, Nan Zhang, Mukund Srinath, et al. Llms assist nlp researchers: Critique paper (meta-) reviewing. arXiv preprint arXiv:2406.16253, 2024.
  11. Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: A graph rag approach to query-focused summarization, 2024. URL https://arxiv.org/abs/2404.16130. ↗
  12. Federico Errica, Giuseppe Siracusano, Davide Sanvito, and Roberto Bifulco. What did i do wrong? quantifying llms’ sensitivity and consistency to prompt engineering. arXiv preprint arXiv:2406.12334, 2024.
  13. Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan. Human-like summarization evaluation with chatgpt. arXiv preprint arXiv:2304.02554, 2023.
  14. Akash Ghosh, Arkadeep Acharya, Raghav Jain, Sriparna Saha, Aman Chadha, and Setu Sinha. Clipsyntel: clip and llm synergy for multimodal question summarization in healthcare. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 22031–22039, 2024. find
  15. Akshay Goel, Almog Gueta, Omry Gilon, Chang Liu, Sofia Erell, Lan Huong Nguyen, Xiaohong Hao, Bolous Jaber, Shashir Reddy, Rupesh Kartha, et al. Llms accelerate annotation for medical information extraction. In Machine Learning for Health (ML4H), pp. 82–100. PMLR, 2023. find
  16. Ayoub Harnoune, Maryem Rhanoui, Mounia Mikram, Siham Yousfi, Zineb Elkaimbillah, and Bouchra El Asri. Bert based clinical knowledge extraction for biomedical knowledge graph construction and analysis. Computer Methods and Programs in Biomedicine Update, 1:100042, 2021. find
  17. Justin C Hayes and David JM Kraemer. Grounded understanding of abstract concepts: The case of stem learning. Cognitive research: principles and implications, 2:1–15, 2017. find
  18. Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pp. 639–648, 2020. find
  19. Yanli Huang. How to represent abstract concepts? from the perspective of conceptual metaphor theory. J Hum Psychol, 1(2):27–37, 2020. find
  20. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. arXiv preprint arXiv:2112.09118, 2021.
  21. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroi… find
  22. Hanlei Jin, Yang Zhang, Dan Meng, Jun Wang, and Jinghua Tan. A comprehensive survey on process-oriented automatic text summarization with exploration of llm-based methods. arXiv preprint arXiv:2403.02901, 2024. — [arXiv:2403.02901](https://arxiv.org/abs/2403.02901) find
  23. Dawulie Jinensibieke, Mieradilijiang Maimaiti, Wentao Xiao, Yuanhang Zheng, and Xiangbo Wang. How good are llms at relation extraction under low-resource scenario? comprehensive evaluation. arXiv preprint arXiv:2406.11162, 2024. — [arXiv:2406.11162](https://arxiv.org/abs/2406.11162) find
  24. Julian Just, Thomas Ströhle, Johann Füller, and Katja Hutter. Ai-based novelty detection in crowdsourced idea spaces. Innovation, 26(3):359–386, 2024. find
  25. Omid Khatin-Zadeh and Danyal Farsani. The understanding of abstract concepts: a perspective from distributed models of conceptual representation. Discover Psychology, 2(1):34, 2022. find
  26. Markus Knauff and Ann G Wolf. Complex cognition: the science of human reasoning, problem-solving, and decision-making, 2010. find
  27. Litton J Kurisinkel and Nancy F Chen. Llm based multi-document summarization exploiting main-event biased monotone submodular content extraction. arXiv preprint arXiv:2310.03414, 2023. — [arXiv:2310.03414](https://arxiv.org/abs/2310.03414) find
  28. LangChain. Langchain graphs. https://python.langchain.com/docs/use_cases/ graph/, 2024. ↗
  29. Baorui Li, Yi Wang, Kesheng Wang, and Jinghui Yang. Application of cnn deep learning in product design evaluation. In Advanced Manufacturing and Automation VIII 8, pp. 517–526. Springer, 2019. find
  30. Xiaonan Li, Kai Lv, Hang Yan, Tianyang Lin, Wei Zhu, Yuan Ni, Guotong Xie, Xiaoling Wang, and Xipeng Qiu. Unified demonstration retriever for in-context learning. arXiv preprint arXiv:2305.04320, 2023.
  31. Weixin Liang, Yuhui Zhang, Hancheng Cao, Binglu Wang, Daisy Yi Ding, Xinyu Yang, Kailas Vodrahalli, Siyu He, Daniel Scott Smith, Yian Yin, et al. Can large language models provide useful feedback on research papers? a large-scale empirical analysis. NEJM AI, 1(8):AIoa2400196, 2024.
  32. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let’s verify step by step. arXiv preprint arXiv:2305.20050, 2023.
  33. Jialiang Lin, Jiaxin Song, Zhangping Zhou, Yidong Chen, and Xiaodong Shi. Automated scholarly paper review: concepts, technologies, and challenges. Information fusion, 98:101830, 2023a. find
  34. Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih, and Xilun Chen. How to train your dragon: Diverse augmentation towards generalizable dense retrieval. arXiv preprint arXiv:2302.07452, 2023b.
  35. LlamaIndex. Llamaindex knowledge graph index. https://docs.llamaindex.ai/en/ stable/examples/index_structs/knowledge_graph/KnowledgeGraphDemo. html, 2024. ↗
  36. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292, 2024.
  37. Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. Chatgpt as a factual inconsistency evaluator for text summarization. arXiv preprint arXiv:2303.15621, 2023.
  38. Jinjie Mai, Jun Chen, Guocheng Qian, Mohamed Elhoseiny, Bernard Ghanem, et al. Llm as a robotic brain: Unifying egocentric memory and control. 2023. find
  39. Christopher D Manning. Foundations of statistical natural language processing. The MIT Press, 1999. find
  40. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. arXiv preprint arXiv:2305.14251, 2023.
  41. Feng Nan, Ramesh Nallapati, Zhiguo Wang, Cicero Nogueira dos Santos, Henghui Zhu, Dejiao Zhang, Kathleen McKeown, and Bing Xiang. Entity-level factual consistency of abstractive text summarization. arXiv preprint arXiv:2102.09130, 2021.
  42. Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Anton Tsitsulin, Mehran Kazemi, Rami Al-Rfou, and Jonathan Halcrow. Let your graph do the talking: Encoding structured data for llms. arXiv preprint arXiv:2402.05862, 2024. — [arXiv:2402.05862](https://arxiv.org/abs/2402.05862) find
  43. Marco Pota, Mirko Ventura, Hamido Fujita, and Massimo Esposito. Multilingual evaluation of pre-processing for bert-based sentiment analysis of tweets. Expert Systems with Applications, 181: 115119, 2021. find
  44. Rukhma Qasim, Waqas Haider Bangyal, Mohammed A Alqarni, and Abdulwahab Ali Almazroi. A fine-tuned bert-based transfer learning approach for text classification. Journal of healthcare engineering, 2022(1):3498123, 2022. find
  45. Usha Nandini Raghavan, Réka Albert, and Soundar Kumara. Near linear time algorithm to detect community structures in large-scale networks. Physical Review E—Statistical, Nonlinear, and Soft Matter Physics, 76(3):036106, 2007. find
  46. Lance J Rips, Edward E Smith, and Douglas L Medin. Concepts and categories: Memory, meaning, and metaphysics. The Oxford handbook of thinking and reasoning, 1:177–209, 2012. find
  47. A Rush. A neural attention model for abstractive sentence summarization. arXiv Preprint, CoRR, abs/1509.00685, 2015.
  48. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910.01108, 2019.
  49. Hannah Sansford, Nicholas Richardson, Hermina Petric Maretic, and Juba Nait Saada. Grapheval: A knowledge-graph based llm hallucination evaluation framework. arXiv preprint arXiv:2407.10793, 2024. — [arXiv:2407.10793](https://arxiv.org/abs/2407.10793) find
  50. Shubhra Kanti Karmaker Santu, Sanjeev Kumar Sinha, Naman Bansal, Alex Knipper, Souvika Sarkar, John Salvador, Yash Mahajan, Sri Guttikonda, Mousumi Akter, Matthew Freestone, et al. Prompting llms to compose meta-review drafts from peer-review narratives of scholarly manuscripts. arXiv preprint arXiv… — [arXiv:2402.15589](https://arxiv.org/abs/2402.15589) find
  51. Wenbo Shang, Xuliang Zhu, and Xin Huang. Path-llm: A shortest-path-based llm learning for unified graph representation. arXiv preprint arXiv:2408.05456, 2024. — [arXiv:2408.05456](https://arxiv.org/abs/2408.05456) find
  52. Shreya Shankar, JD Zamfirescu-Pereira, Björn Hartmann, Aditya G Parameswaran, and Ian Arawjo. Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences. arXiv preprint arXiv:2404.12272, 2024. — [arXiv:2404.12272](https://arxiv.org/abs/2404.12272) find
  53. Guangsi Shi, Xiaofeng Deng, Linhao Luo, Lijuan Xia, Lei Bao, Bei Ye, Fei Du, Shirui Pan, and Yuxiao Li. Llm-powered explanations: Unraveling recommendations through subgraph reasoning. arXiv preprint arXiv:2406.15859, 2024. — [arXiv:2406.15859](https://arxiv.org/abs/2406.15859) find
  54. Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
  55. Chenglei Si, Diyi Yang, and Tatsunori Hashimoto. Can llms generate novel research ideas? a large-scale human study with 100+ nlp researchers. arXiv preprint arXiv:2409.04109, 2024.
  56. Dominik Siemon. Let the computer evaluate your idea: evaluation apprehension in human-computer collaboration. Behaviour & Information Technology, 42(5):459–477, 2023. find
  57. John Stamper, Ruiwei Xiao, and Xinying Hou. Enhancing llm-based feedback: Insights from intelligent tutoring systems and the learning sciences. In International Conference on Artificial Intelligence in Education, pp. 32–43. Springer, 2024. find
  58. Lina Sun and Sitan Li. Cnn-based evaluation method of academic innovation effect of american research universities. In 2021 IEEE International Conference on Industrial Application of Artificial Intelligence (IAAI), pp. 355–360. find
  59. Aye Thida. Bert-based dialogue evaluation methods with ruber framework. In Advances in Artificial Intelligence: Selected Papers from the Annual Conference of Japanese Society of Artificial Intelligence (JSAI 2020), volume 1357, pp. 133. find
  60. Milena Trajanoska, Riste Stojanov, and Dimitar Trajanov. Enhancing knowledge graph construction using large language models. arXiv preprint arXiv:2305.04676, 2023. — [arXiv:2305.04676](https://arxiv.org/abs/2305.04676) find
  61. Thanyalak Ubonsiri. AI-GENERATED EVALUATION: THE INFLUENCE OF CHATGPT EVAL-UATION ON INDIVIDUALS’DECISIONS IN THE IDEA EVALUATION PHASE. PhD thesis, Leopold-Franzens-Universität Innsbruck, 2024. find
  62. Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao. Human-llm collaborative annotation through effective verification of llm labels. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1–21, 2024. find
  63. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022.
  64. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824–24837, 2022.
  65. Kangda Wei, Aayush Gautam, and Ruihong Huang. Are llms good annotators for discourse-level event relation extraction? arXiv preprint arXiv:2407.19568, 2024. find
  66. Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning, pp. 6861–6871. PMLR, 2019. find
  67. Yi Xu, Bo Xue, Shuqian Sheng, Cheng Deng, Jiaxin Ding, Zanwei Shen, Luoyi Fu, Xinbing Wang, and Chenghu Zhou. Good idea or not, representation of llm could tell. arXiv preprint arXiv:2409.13712, 2024. — [arXiv:2409.13712](https://arxiv.org/abs/2409.13712) find
  68. Rui Yang, Boming Yang, Sixun Ouyang, Tianwei She, Aosong Feng, Yuang Jiang, Freddy Lecue, Jinghui Lu, and Irene Li. Graphusion: Leveraging large language models for scientific knowledge graph fusion and construction in nlp education, 2024. URL https://arxiv.org/abs/ 2407.10794. ↗
  69. Jia-Yu Yao, Kun-Peng Ning, Zhen-Hui Liu, Mu-Nan Ning, and Li Yuan. Llm lies: Hallucinations are not bugs, but features as adversarial examples. arXiv preprint arXiv:2310.01469, 2023a. — [arXiv:2310.01469](https://arxiv.org/abs/2310.01469) find
  70. Liang Yao, Jiazhen Peng, Chengsheng Mao, and Yuan Luo. Exploring large language models for knowledge graph completion. arXiv preprint arXiv:2308.13916, 2023b. — [arXiv:2308.13916](https://arxiv.org/abs/2308.13916) find
  71. Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36, 2024. find
  72. Weizhe Yuan, Pengfei Liu, and Graham Neubig. Can we automate scientific reviewing? Journal of Artificial Intelligence Research, 75:171–212, 2022.
  73. Collin Zhang, John X Morris, and Vitaly Shmatikov. Extracting prompts by inverting llm outputs. arXiv preprint arXiv:2405.15012, 2024a. find
  74. Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xiaochuang Han, Tianxing He, and Yulia Tsvetkov. Can llm graph reasoning generalize beyond pattern memorization? arXiv preprint arXiv:2406.15992, 2024b. find
  75. Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren’s song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219, 2023.
  76. Zhi-Wu Zhang, Xiao-Yuan Jing, and Tie-Jian Wang. Label propagation based semi-supervised learning for software defect prediction. Automated Software Engineering, 24:47–69, 2017. find
  77. Shen Zheng, Jie Huang, and Kevin Chen-Chuan Chang. Why does chatgpt fall short in providing truthful answers? arXiv preprint arXiv:2304.10513, 2023. — [arXiv:2304.10513](https://arxiv.org/abs/2304.10513) find
  78. Yuqi Zhu, Xiaohan Wang, Jing Chen, Shuofei Qiao, Yixin Ou, Yunzhi Yao, Shumin Deng, Huajun Chen, and Ningyu Zhang. Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities. World Wide Web, 27(5):58, 2024. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/feng-2025-grapheval-lightweight-graph-based-llm/