Elamur Research⌕
☰
home / papers / 2024 / NL2Formula: Generating Spreadsheet Formulas from Natural Language Queries

NL2Formula: Generating Spreadsheet Formulas from Natural Language Queries

Paper · 2024 · Wei Zhao, Zhitao Hou, Siyuan Wu et al.

Summary

Writing spreadsheet formulas by hand is tedious and error-prone for many users. This paper proposes NL2Formula, a new benchmark task: given a natural-language question and a spreadsheet table, produce an executable Microsoft Excel formula grounded in that table. The task resembles Text2SQL, but it is harder in two ways. Spreadsheet layouts are loosely structured, and formulas must point to exact cell addresses rather than just column names.

Labelled data did not exist, so the authors built a dataset by converting SQL queries from WikiSQL and Spider into Excel formulas with hand-crafted grammar-mapping rules. They added manually collected examples that follow predefined rules. The resulting corpus pairs 70,799 queries with formulas over 21,670 tables.

For modelling, they present fCoder, a T5-based sequence-to-sequence system. It outperformed an adapted ForTaP baseline, and the authors also compared it with GPT-3.5 (text-davinci-003). An error analysis shows that most failures come from locating or extracting the wrong evidence. The work offers the first standardized testbed for natural-language interfaces to spreadsheets.

Key points

  • Dataset size: 70,799 query–formula pairs over 21,670 tables, covering 37 Excel formula function types. Formulas fall into two categories: Analysis Query (function-based) and Calculation (arithmetic over cells).
  • Data sources: SQL from WikiSQL (80,654 instances) and Spider (10,181 queries) was converted using rules such as MAX → MAXIFS, or multi-condition queries → FILTER combined with aggregates. Tables were limited to 3–20 rows and 3–10 columns.
  • Main result: fCoder (T5-based) reached 70.6% Exact Matching Accuracy and 77.1% execution-based accuracy in Microsoft Excel. It beat ForTaP, which builds on TUTA.
  • Error types: wrong evidence, missing evidence, wrong intent inference, and wrong calculation.
  • Limitations and availability: The table structure is fairly fixed, and some functions are not covered (e.g., FIND, string functions). Input is capped at 512, and custom lambda functions cannot be executed. Code and data are available at github.com/timetub/NL2Formula.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States403 works · 162,838 citations🎓Huazhong University of Science and TechnologyWuhan · China64 works · 8,427 citations🎓UNSW SydneySydney · Australia40 works · 1,630 citations🎓Chongqing UniversityChongqing · China15 works · 329 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksSoftware · 150 works

Cite

Cite as Wei Zhao, Zhitao Hou, Siyuan Wu, Yan Gao, Haoyu Dong, Yao Wan, Hongyu Zhang, Yulei Sui and Haidong Zhang. 2024. NL2Formula: Generating Spreadsheet Formulas from Natural Language Queries. Findings of the Association for Computational Linguistics: EACL 2024.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.findings-eacl.158 · arXiv 2402.14853

@inproceedings{zhao2024nl2formula,
  title = {{NL2Formula: Generating Spreadsheet Formulas from Natural Language Queries}},
  author = {Wei Zhao, Zhitao Hou, Siyuan Wu, Yan Gao, Haoyu Dong, Yao Wan, Hongyu Zhang, Yulei Sui and Haidong Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2024},
  year = {2024},
  doi = {10.18653/v1/2024.findings-eacl.158},
  eprint = {2402.14853},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2024.findings-eacl.158}
}

References

35 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Berant and P. Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  3. Harrison Chase. 2022. LangChain. find
  4. Srini Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Annual Meeting of the Association for Computational Linguistics. find
  5. Jean Kaddour, Joshua Harris, Maximilian Mozes, Herbie Bradley, Roberta Raileanu, and Robert McHardy. 2023. Challenges and applications of large language models.
  6. Ashwin Kalyan, Abhishek Mohta, Oleksandr Polozov, Dhruv Batra, Prateek Jain, and Sumit Gulwani. 2018. Neural-guided deductive search for real-time program synthesis from examples. In ICLR. find
  7. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020. Tabfact: A large-scale dataset for table-based fact verification. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. Op…
  8. F. Li and H. V. Jagadish. 2014. Constructing an interactive natural language interface for relational databases. Proceedings of the Vldb Endowment, 8(1):73–84. find
  9. Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, and Denny Zhou. 2021. Spreadsheetcoder: Formula prediction from semi-structured context. In International Conference on Machine Learning, pages 1661–1672. PMLR.
  10. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  11. Zhoujun Cheng, Haoyu Dong, Fan Cheng, Ran Jia, Pengfei Wu, Shi Han, and Dongmei Zhang. 2021. Fortap: Using formulae for numerical-reasoning-aware table pretraining. In Proceedings of the Association for Computational Linguistics. find
  12. Shing-Chi Cheung, Wanjun Chen, Yepang Liu, and Chang Xu. 2016. Custodes: automatic spreadsheet cell clustering and smell detection using strong and weak features. In Proceedings of the 38th International Conference on Software Engineering, pages 464–475. find
  13. OpenAI. ChatGPT plugins. https://openai.com/blog/chatgpt-plugins. 2023. ↗
  14. Emilio Parisotto, Abdel-rahman Mohamed, Rishabh Singh, Lihong Li, Dengyong Zhou, and Pushmeet Kohli. 2017. Neuro-symbolic program synthesis. In International Conference on Leaning Representations.
  15. A. M. Popescu, O. Etzioni, and H. Kautz. 2003. Towards a theory of natural language interfaces to databases. International Conference on Intelligent User Interfaces. find
  16. L. Dong and M. Lapata. 2016. Language to logical form with neural attention. Office for Official Publications of the European Communities,.
  17. P. J. Price. 1990. Evaluation of spoken language systems: the atis domain. In Proceedings of the third DARPA Speech and Natural Language Workshop. find
  18. Sumit Gulwani. 2011. Automating string processing in spreadsheets using input-output examples. ACM Sigplan Notices, 46(1):317–330.
  19. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67.
  20. Sumit Gulwani, William R Harris, and Rishabh Singh. 2012. Spreadsheet data manipulation using examples. Communications of the ACM, 55(8):97–105. find
  21. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320–4333, Online. Association for Compu…
  22. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. find
  23. S. Reddy, M. Lapata, and M. Steedman. 2014. Large-scale semantic parsing without question-answer pairs. Transactions of the Association for Computational Linguistics, 2(1):377–392. find
  24. L. R. Tang and R. J. Mooney. 2001. Using multiple clause constructors in inductive logic programming for semantic parsing. Springer, Berlin, Heidelberg. find
  25. Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. 2021. Tuta: tree-based transformers for generally structured table pre-training. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1780–1790. find
  26. Y. W. Wong and R. J. Mooney. 2007. Learning synchronous grammars for semantic parsing with lambda calculus. In Acl, Meeting of the Association for Computational Linguistics, June, Prague, Czech Republic.
  27. Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. Sqlizer: query synthesis from natural language. Proceedings of the ACM on Programming Languages, 1(OOPSLA):1–26. find
  28. Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413–8426.
  29. Tao Yu, Rui Zhang, Heyang Er, Suyi Li, Eric Xue, Bo Pang, Victoria Lin, Yi Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, and Dragomir Radev. 2019. Cosql: A conversational text-to-sql challenge…
  30. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn-ing Yao, Shanelle Roman, et al. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of EMNLP.
  31. J. M. Zelle and R. J. Mooney. 1996. Learning to parse database queries using inductive logic programming. AAAI Press.
  32. L. S. Zettlemoyer and M. Collins. 2012. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Conference on Uncertainty in Artificial Intelligence.
  33. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of l…
  34. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103.
  35. Papers), pages 3277–3287, Online. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhao-2024-nl2formula-generating-spreadsheet-formulas-natural/