Elamur Research⌕
☰
home / papers / 2024 / SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation

SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation

Paper · 2024 · Zeyao Ma, Bohan Zhang, Jing Zhang et al.

Summary

SpreadsheetBench tackles a weakness in how LLM-based spreadsheet assistants are evaluated. Earlier benchmarks rely on synthetic or crowdsourced instructions, simple single-table sheets, and a single check per task. The authors collected solved questions from four Excel help forums (ExcelForum, Chandoo, MrExcel, ExcelGuru). They filtered them for relevance, feasibility and representativeness, then rewrote them into instructions with the help of GPT-4 and human checking. The attached spreadsheets are messy, as real ones tend to be: several tables on one sheet, irregular headers, free text and formatting such as colours.

Each task requires a code solution. Scoring follows an online-judge approach: a solution counts as correct only if it handles several test spreadsheets that share a layout but contain different values. This penalises answers that fit one file by accident.

The authors tested TableQA models, open general and code LLMs, proprietary models and spreadsheet agents. Even the strongest system, GPT-4o, stays far behind human solvers. Models trained for code beat general models of similar openness. Multi-round interaction with execution feedback helps most models but not GPT-4o. The benchmark gives a realistic and robust target for future spreadsheet agents.

Key points

  • The dataset has 912 real forum instructions and 2,729 test-case spreadsheets, about three per instruction. The instructions span 10 manipulation categories and are split into cell-level and sheet-level tasks.
  • The spreadsheets are complex: 35.7% contain multiple tables and 42.7% contain non-standard relational tables. Some tables exceed 100 columns or 20,000 rows.
  • Model scores range from 0.05% to 23.65%, and some methods score 0%. GPT-4o reaches roughly 18% overall under the lenient criterion, while humans reach about 71%.
  • Requiring a pass on all test cases (hard restriction) lowers scores compared with the soft criterion. Performance also drops on large sheets, sheets with multiple tables, and sheets with irregular structure.
  • Code models such as DeepseekCoder beat Llama-3 (70B). Multi-round prompting boosts weaker models up to about tenfold. Adding more preview rows (10 instead of 5) brings little gain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksSoftware · 92 works

Cite

Cite as Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang and Jie Tang. 2024. SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation. arXiv:2406.14991.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2406.14991 · arXiv 2406.14991

@misc{ma2024spreadsheetbench,
  title = {{SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation}},
  author = {Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang and Jie Tang},
  year = {2024},
  doi = {10.48550/arxiv.2406.14991},
  eprint = {2406.14991},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2406.14991}
}

References

42 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. S. Rahman, K. Mack, M. Bendre, R. Zhang, K. Karahalios, and A. Parameswaran, “Benchmarking spreadsheet systems,” in Proceedings of the 2020 acm sigmod international conference on management of data, pp. 1589–1599, 2020. find
  2. M. Jackson and M. Staunton, Advanced modelling in finance using Excel and VBA. John Wiley & Sons, 2006. find
  3. W. L. Winston, Marketing analytics: Data-driven techniques with Microsoft Excel. John Wiley & Sons, 2014. find
  4. S. Gulwani, “Automating string processing in spreadsheets using input-output examples,” in Proceedings of the 38th annual ACM SIGPLAN-SIGACT symposium on Principles of programming languages, ACM, 2011. find
  5. S.-C. Cheung, W. Chen, Y. Liu, and C. Xu, “Custodes: automatic spreadsheet cell clustering and smell detection using strong and weak features,” in Proceedings of the 38th International Conference on Software Engineering, pp. 464–475, 2016. find
  6. S. Gulwani, “Automating string processing in spreadsheets using input-output examples,” ACM Sigplan Notices, vol. 46, no. 1, pp. 317–330, 2011. find
  7. S. Gulwani, W. R. Harris, and R. Singh, “Spreadsheet data manipulation using examples,” Communications of the ACM, vol. 55, no. 8, pp. 97–105, 2012. find
  8. X. Chen, P. Maniatis, R. Singh, C. Sutton, H. Dai, M. Lin, and D. Zhou, “Spreadsheetcoder: Formula prediction from semi-structured context,” in International Conference on Machine Learning, pp. 1661–1672, PMLR, 2021. find
  9. S. Chen, Y. He, W. Cui, J. Fan, S. Ge, H. Zhang, D. Zhang, and S. Chaudhuri, “Auto-formula: Recommend formulas in spreadsheets using contrastive learning for table representations,” Proceedings of the ACM on Management of Data, vol. 2, no. 3, pp. 1–27, 2024. find
  10. J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al., “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774, 2023.
  11. H. Li, J. Su, Y. Chen, Q. Li, and Z.-X. ZHANG, “Sheetcopilot: Bringing software productivity to the next level through large language models,” Advances in Neural Information Processing Systems, vol. 36, 2024. find
  12. Y. Chen, Y. Yuan, Z. Zhang, Y. Zheng, J. Liu, F. Ni, and J. Hao, “Sheetagent: A generalist agent for spreadsheet reasoning and manipulation via large language models,” arXiv preprint arXiv:2403.03636, 2024. — [arXiv:2403.03636](https://arxiv.org/abs/2403.03636) find
  13. P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experience: Evaluating the usability of code generation tools powered by large language models,” in Chi conference on human factors in computing systems extended abstracts, pp. 1–7, 2022. find
  14. J. Payan, S. Mishra, M. Singh, C. Negreanu, C. Poelitz, C. Baral, S. Roy, R. Chakravarthy, B. Van Durme, and E. Nouri, “Instructexcel: A benchmark for natural language instruction in excel,” in Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 4026– 4043, 2023. find
  15. Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi, “Self-instruct: Aligning language models with self-generated instructions,” in The 61st Annual Meeting Of The Association For Computational Linguistics, 2023. find
  16. P. Pasupat and P. Liang, “Compositional semantic parsing on semi-structured tables,” in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1470–1480, 2015.
  17. L. Nan, C. Hsieh, Z. Mao, X. V. Lin, N. Verma, R. Zhang, W. Kryściński, H. Schoelkopf, R. Kong, X. Tang, et al., “Fetaqa: Free-form table question answering,” Transactions of the Association for Computational Linguistics, vol. 10, pp. 35–49, 2022.
  18. V. Zhong, C. Xiong, and R. Socher, “Seq2sql: Generating structured queries from natural language using reinforcement learning,” arXiv preprint arXiv:1709.00103, 2017.
  19. T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, et al., “Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task,” in Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing,…
  20. Y. Katsis, S. Chemmengath, V. Kumar, S. Bharadwaj, M. Canim, M. Glass, A. Gliozzo, F. Pan, J. Sen, K. Sankaranarayanan, et al., “Ait-qa: Question answering dataset over complex tables in the airline industry,” in Proceedings of the 2022 Conference of the North American Chapter of the Association for… find
  21. S. Wasik, M. Antczak, J. Badura, A. Laskowski, and T. Sternal, “A survey on online judge systems and their applications,” ACM Computing Surveys (CSUR), vol. 51, no. 1, pp. 1–34, 2018. find
  22. A. Elangovan, J. He, and K. Verspoor, “Memorization vs. generalization: Quantifying data leakage in nlp performance evaluation,” in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 1325–1335, 2021. find
  23. Y. Lai, C. Li, Y. Wang, T. Zhang, R. Zhong, L. Zettlemoyer, W.-t. Yih, D. Fried, S. Wang, and T. Yu, “Ds-1000: A natural and reliable benchmark for data science code generation,” in International Conference on Machine Learning, pp. 18319–18345, PMLR, 2023. find
  24. F. Zhu, W. Lei, Y. Huang, C. Wang, S. Zhang, J. Lv, F. Feng, and T.-S. Chua, “Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance,” in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Con…
  25. W. Chen, M.-W. Chang, E. Schlinger, W. Wang, and W. W. Cohen, “Open question answering over tables and text,” arXiv preprint arXiv:2010.10439, 2020. find
  26. Z. Cheng, H. Dong, Z. Wang, R. Jia, J. Guo, Y. Gao, S. Han, J.-G. Lou, and D. Zhang, “Hitab: A hierarchical table dataset for question answering and natural language generation,” in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1… find
  27. Y. Zhao, Z. Qi, L. Nan, L. J. Flores, and D. Radev, “Loft: Enhancing faithfulness and diversity for table-to-text generation via logic form control,” in Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 554–561, 2023. find
  28. Y. Zhao, H. Zhang, S. Si, L. Nan, X. Tang, and A. Cohan, “Investigating table-to-text generation capabilities of large language models in real-world information seeking scenarios,” in Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pp. 160–175,… find
  29. X. He, M. Zhou, X. Xu, X. Ma, R. Ding, L. Du, Y. Gao, R. Jia, X. Chen, S. Han, et al., “Text2analysis: A benchmark of table question answering with advanced data analysis and unclear queries,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, pp. 18206–18215, 2024. find
  30. J. Li, N. Huo, Y. Gao, J. Shi, Y. Zhao, G. Qu, Y. Wu, C. Ma, J.-G. Lou, and R. Cheng, “Tapilot-crossing: Benchmarking and evolving llms towards interactive data analysis agents,” arXiv preprint arXiv:2403.05307, 2024. — [arXiv:2403.05307](https://arxiv.org/abs/2403.05307) find
  31. X. Hu, Z. Zhao, S. Wei, Z. Chai, G. Wang, X. Wang, J. Su, J. Xu, M. Zhu, Y. Cheng, et al., “Infiagent-dabench: Evaluating agents on data analysis tasks,” arXiv preprint arXiv:2401.05507, 2024. — [arXiv:2401.05507](https://arxiv.org/abs/2401.05507) find
  32. W. Zhou, M. Mesgar, H. Adel, and A. Friedrich, “Freb-tqa: A fine-grained robustness evaluation benchmark for table question answering,” arXiv preprint arXiv:2404.18585, 2024. find
  33. G. Cormack, I. Munro, T. Vasiga, and G. Kemkes, “Structure, scoring and purpose of computing competitions,” Informatics in education, vol. 5, no. 1, pp. 15–36, 2006. find
  34. Q. Liu, B. Chen, J. Guo, M. Ziyadi, Z. Lin, W. Chen, and J.-G. Lou, “TAPEX: Table pre-training via learning a neural SQL executor,” in International Conference on Learning Representations, 2022.
  35. J. Herzig, P. K. Nowak, T. Mueller, F. Piccinno, and J. Eisenschlos, “Tapas: Weakly supervised table parsing via pre-training,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4320–4333, 2020.
  36. Z. Cheng, T. Xie, P. Shi, C. Li, R. Nadkarni, Y. Hu, C. Xiong, D. Radev, M. Ostendorf, L. Zettlemoyer, N. A. Smith, and T. Yu, “Binding language models in symbolic languages,” ICLR, vol. abs/2210.02875, 2023.
  37. J. Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L. Ji, M. Li, J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang, W. Wang, S. Wu, B. Xu, J. Xu, A. Yang, H. Yang, J. Yang, S. Yang, Y. Yao, B.… — Qwen Technical Report
  38. D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y. Wu, Y. Li, et al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024. find
  39. A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bamford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressand, et al., “Mixtral of experts,” arXiv preprint arXiv:2401.04088, 2024. find
  40. AI@Meta, “Llama 3 model card,” 2024. find
  41. S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. R. Narasimhan, and Y. Cao, “React: Synergizing reasoning and acting in language models,” in The Eleventh International Conference on Learning Representations, 2022.
  42. T. Zheng, G. Zhang, T. Shen, X. Liu, B. Y. Lin, J. Fu, W. Chen, and X. Yue, “Opencodeinterpreter: Integrating code generation with execution and refinement,” arXiv preprint arXiv:2402.14658, 2024. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ma-2024-spreadsheetbench-towards-challenging-real-world/