Elamur Research⌕
☰
home / papers / 2026 / InsightEval: An Expert-Curated Benchmark for Assessing Insight Discovery in LLM-Driven Data Agents

InsightEval: An Expert-Curated Benchmark for Assessing Insight Discovery in LLM-Driven Data Agents

Paper · 2026 · Zhenghao Zhu, Yuanfeng Song, Xing Chen et al.

Summary

LLM-based data agents are increasingly used to extract insights from tables, but few benchmarks measure this ability well. The authors audited InsightBench, the main existing benchmark, by running Agent Poirot with GPT-4o on 50 random samples. They found vague goals, questions that refer to columns missing from the tables, undefined insight types, insights wrongly claiming the data is insufficient, duplicated insights, and an evaluation that looks only at recall.

They then set out quality criteria for an insight benchmark and built InsightEval with a four-stage pipeline: goal refinement, question verification and generation, answering and insight generation, and summarization. The pipeline combines automated checks with expert review. They also introduced Insight F1, which combines precision and recall, and a novelty metric that rewards valid insights not present in the annotations.

Experiments compared single-model agents with multi-agent frameworks. Insight F1 agreed more closely with human expert scores than recall alone did. The results also show that agents favour precision over broad exploration. InsightEval therefore offers a more reliable way to evaluate automated insight discovery.

Key points

  • InsightEval contains 1000 insights across six types, including two new categories, Evaluative and Exploratory.
  • In the InsightBench audit, the five error types (E1–E5) occurred at rates of 57.0%, 32.8%, 20.8%, 18.3% and 14.5%. Ambiguous goals (E1) were the most frequent problem.
  • The evaluated systems were DeepResearch Agent and Pandas Agent (both GPT-4o), plus Agent Poirot with GPT-4o, DeepSeek-V3 and Claude-3.7-Sonnet backbones. Scoring used ROUGE-1, G-Eval and human expert ratings.
  • Multi-agent systems beat single-model baselines:
    • Agent Poirot (Claude-3.7-Sonnet) achieved the best Insight F1 under G-Eval.
    • Agent Poirot (DeepSeek-V3) achieved the best Summary score.
  • On the novelty metric, the Claude-3.7-Sonnet agent reached 76.2%, and the DeepSeek-V3 agent showed the largest relative gain over its original novelty score (13.3%). The average score rose from 0.585 (original) to 0.634 (new).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Hong Kong University of Science and TechnologyHong Kong · China142 works · 25,136 citations🏢ByteDanceBeijing · China41 works · 4,180 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 worksDecision Sciences · 209 worksInformation Systems and Management · 41 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 works

Cite

Cite as Zhenghao Zhu, Yuanfeng Song, Xing Chen, Chengzhong Liu, Cui Yakun, Caleb Chen Cao, Sirui Han and Yike Guo. 2026. InsightEval: An Expert-Curated Benchmark for Assessing Insight Discovery in LLM-Driven Data Agents. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1729

@inproceedings{zhu2026insighteval,
  title = {{InsightEval: An Expert-Curated Benchmark for Assessing Insight Discovery in LLM-Driven Data Agents}},
  author = {Zhenghao Zhu, Yuanfeng Song, Xing Chen, Chengzhong Liu, Cui Yakun, Caleb Chen Cao, Sirui Han and Yike Guo},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1729},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1729}
}

References

33 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. AI Anthropic. 2025. Claude 3.7 sonnet and claude code. find
  2. Shubham Chandel, Colin B Clement, Guillermo Serrato, and Neel Sundaresan. 2022. Training and evaluating a jupyter notebook data science assistant. arXiv preprint arXiv:2201.12901. — [arXiv:2201.12901](https://arxiv.org/abs/2201.12901) find
  3. Nan Chen, Yuge Zhang, Jiahang Xu, Kan Ren, and Yuqing Yang. 2024. Viseval: A benchmark for data visualization in the era of large language models. IEEE Transactions on Visualization and Computer Graphics. find
  4. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX…
  5. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, and 1 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2412.19437. find
  6. Mingwen Dong, Nischal Ashok Kumar, Yiqun Hu, Anuj Chauhan, Chung-Wei Hang, Shuaichen Chang, Lin Pan, Wuwei Lan, Henghui Zhu, Jiarong Jiang, and 1 others. 2025. Practiq: A practical conversational text-to-sql dataset with ambiguous and unanswerable queries. In Proceedings of the 2025 Conference of th…
  7. Xueyu Hu, Ziyu Zhao, Shuang Wei, Ziwei Chai, Qianli Ma, Guoyin Wang, Xuwu Wang, Jing Su, Jingjing Xu, Ming Zhu, and 1 others. 2024. Infiagent-dabench: evaluating agents on data analysis tasks. In Proceedings of the 41st International Conference on Machine Learning, pages 19544–19572. find
  8. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettlemoyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. Ds-1000: A natural and reliable benchmark for data science code generation. In International Conference on Machine Learning, pages 18319–18345. PMLR. find
  9. LangChain. 2024. Pandas dataframe. find
  10. Gyubok Lee, Hyeonji Hwang, Seongsu Bae, Yeonsu Kwon, Woncheol Shin, Seongjun Yang, Minjoon Seo, Jong-Yeup Kim, and Edward Choi. 2022. Ehrsql: A practical text-to-sql benchmark for electronic health records. Advances in Neural Information Processing Systems, 35:15589–15601. find
  11. Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin SU, ZHAOQING SUO, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, and 1 others. 2024. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. In The Thirteenth International Conference on Learning Repre… find
  12. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
  13. Xinyu Liu, Shuyu Shen, Boyan Li, Nan Tang, and Yuyu Luo. 2025. Nl2sql-bugs: A benchmark for detecting semantic errors in nl2sql translation. In Proceedings of the 31th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. find
  14. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
  15. Jinwei Lu, Yuanfeng Song, Zhiqian Qin, Haodi Zhang, Chen Zhang, and Raymond Chi-Wing Wong. 2025. Bridging the gap: Enabling natural language queries for nosql databases through text-to-nosql translation. arXiv preprint arXiv:2502.11201. — [arXiv:2502.11201](https://arxiv.org/abs/2502.11201) find
  16. Jinwei Lu, Yuanfeng Song, Chen Zhang, and Raymond Chi-Wing Wong. 2026. Multivis-agent: A multi-agent framework with logic rules for reliable and comprehensive cross-modal data visualization. Proc. ACM Manag. Data, 4(1). find
  17. Pingchuan Ma, Rui Ding, Shuai Wang, Shi Han, and Dongmei Zhang. 2023. InsightPilot: An LLM-empowered automated data exploration system. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 346–352, Singapore. Association for Computat… find
  18. Bodhisattwa Prasad Majumder, Harshit Surana, Dhruv Agarwal, Bhavana Dalvi Mishra, Abhijeetsingh Meena, Aryan Prakhar, Tirth Vora, Tushar Khot, Ashish Sabharwal, and Peter Clark. 2025. Discoverybench: Towards data-driven discovery with large language models. In The Thirteenth International Conference… find
  19. OpenAI. 2024. Gpt-4o system card. arXiv:2410.21276.
  20. Preprint, OpenAI. 2025. Openai o3-mini system card. find
  21. Geliang Ouyang, Jingyao Chen, Zhihe Nie, Yi Gui, Yao Wan, Hongyu Zhang, and Dongping Chen. 2025. nvAgent: Automated data visualization from natural language via collaborative agent workflow. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa… find
  22. Alberto Sánchez Pérez, Alaa Boukhary, Paolo Papotti, Luis Castejón Lozano, and Adam Elwood. 2025. An llm-based approach for insight generation in data analysis. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Langua… find
  23. Zhiqian Qin, Yuanfeng Song, Jinwei Lu, Yuanwei Song, Shuaimin Li, and Chen Jason Zhang. 2025. Multi-tend: A multilingual benchmark for natural language to nosql query translation. In Findings of the Association for Computational Linguistics: ACL 2025, pages 24632–24657. find
  24. Gaurav Sahu, Abhay Puri, Juan A Rodriguez, Amirhossein Abaskohi, Mohammad Chegini, Alexandre Drouin, Perouz Taslakian, Valentina Zantedeschi, Alexandre Lacoste, David Vazquez, and 1 others. 2025. Insightbench: Evaluating business analytics agents through multi-step insight generation. In The Thirtee… find
  25. Xiangru Tang, Daniel Shao, Jiwoong Sohn, Jiapeng Chen, Jiayi Zhang, Jinyu Xiang, Fang Wu, Yilun Zhao, Chenglin Wu, Wenqi Shi, and 1 others. 2025. Medagentsbench: Benchmarking thinking models and agent frameworks for complex medical reasoning. arXiv preprint arXiv:2503.07459. 2025. Dagent: A relation… — [arXiv:2503.07459](https://arxiv.org/abs/2503.07459) find
  26. Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, Zhiyuan Liu, Xiaodong Shi, and Maosong Sun. 2024. MatPlotAgent: Method and evaluation for LLM-based agentic scientific data visualization. In Findings of the Association for Computationa… find
  27. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
  28. Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, and 1 others. 2023. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420. find
  29. Pengcheng Yin, Wen-Ding Li, Kefan Xiao, Abhishek Rao, Yeming Wen, Kensen Shi, Joshua Howland, Paige Bailey, Michele Catasta, Henryk Michalewski, Oleksandr Polozov, and Charles Sutton. 2023. Natural language to code generation in interactive data science notebooks. In Proceedings of the 61st Annual M… find
  30. Wentao Zhang, Liang Zeng, Yuzhen Xiao, Yongcong Li, Ce Cui, Yilei Zhao, Rui Hu, Yang Liu, Yahui Zhou, and Bo An. 2025. Agentorchestra: Orchestrating hierarchical multi-agent intelligence with the tool-environment-agent(tea) protocol. Preprint, arXiv:2506.12508. — [arXiv:2506.12508](https://arxiv.org/abs/2506.12508) find
  31. Luoxuan Weng, Xingbo Wang, Junyu Lu, Yingchaojie Feng, Yihan Liu, Haozhe Feng, Danqing Huang, and Wei Chen. 2025. Insightlens: Augmenting llm-powered data analysis with interactive insight management and navigation. IEEE Transactions on Visualization and Computer Graphics. find
  32. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, and 1 others. 2024. Autogen: Enabling next-gen llm applications via multi-agent conversations. In First Conference on Language Modeling. find
  33. Wenyi Xu, Yuren Mao, Xiaolu Zhang, Chao Zhang, Xuemei Dong, Mengfei Zhang, and Yunjun Gao. A.1 Category and Difficulty • Incident Management: Tracks and analyzes operational or safety incidents to enable rapid response and root-cause investigation. • Asset Management: Manages IT hardware lifecycle—p… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2026-insighteval-expert-curated-benchmark-assessing/