Elamur Research⌕
☰
home / papers / 2022 / FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining

FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining

Paper · 2022 · Zhoujun Cheng, Haoyu Dong, Ran Jia et al.

Summary

Tables hold large amounts of numerical data, yet models still struggle to reason about it. They need to work out what a given number means from its headers, and how numbers relate to each other through calculations. Manually labelling such reasoning is costly and error-prone. FORTAP addresses this by using spreadsheet formulas from the web as free, naturally occurring supervision for table pretraining. Formulas show which cells feed into a computation and which operations combine them. Drawing on these formulas, the authors design two self-supervised tasks, NRP (Numerical Reference Prediction) along with NCP (Numerical Calculation Prediction). They add a masked language modelling objective over formula tokens. FORTAP uses TUTA as its encoder, chosen for its tree-based handling of spreadsheet layouts, but the objectives can work with any table encoder. FORTAP sets new state-of-the-art results on formula prediction (Enron), table question answering (HiTab), and cell type classification (DeEx). This suggests real spreadsheet formulas are a strong, scalable signal for teaching numerical reasoning over semi-structured tables.

Key points

  • Pretraining corpus: the same corpus as TUTA, crawled from 1.75M websites.
    • 13.5M spreadsheet files were filtered down to 4.5M tables.
    • Formulas were parsed with XLParser and linearised in prefix AST order, as in SpreadsheetCoder, yielding 10.8M formulas.
    • Cross-sheet, array, and user-defined-function formulas were removed.
  • Pretraining objectives:
    • NRP classifies whether a formula cell's header refers to another header.
    • NCP predicts which operator or function links the referenced cells.
    • Formula MLM recovers masked formula tokens.
  • Formula prediction (Enron): 55.8% top-1 accuracy, beating TUTA (48.5%), TaPEx (43.2%), and SpreadsheetCoder (40.4%).
    • In ablations, NRP mainly improves range accuracy and NCP mainly improves sketch accuracy.
  • HiTab QA: +6.3% over the previous SOTA.
    • The largest gains were on complex cell selection (+8.0%) and arithmetic questions (+7.3%).
  • DeEx cell type classification: +6.6% on the derived type and +3.2% overall Macro-F1 compared with TUTA.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Fudan UniversityShanghai · China213 works · 22,907 citations🎓Shanghai Jiao Tong UniversityShanghai · China232 works · 20,305 citations🏢Microsoft Research AsiaBeijing · China86 works · 61,617 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksArts and Humanities · 17,959 worksLanguage and Linguistics · 17,789 worksComputer Vision and Pattern Recognition · 3,621 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 4,145 worksReasoning · 1,948 works

Cite

Cite as Zhoujun Cheng, Haoyu Dong, Ran Jia, Pengfei Wu, Shi Han, Fan Cheng and Dongmei Zhang. 2022. FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining. Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.acl-long.82 · arXiv 2109.07323

@inproceedings{cheng2022fortap,
  title = {{FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining}},
  author = {Zhoujun Cheng, Haoyu Dong, Ran Jia, Pengfei Wu, Shi Han, Fan Cheng and Dongmei Zhang},
  booktitle = {Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2022},
  doi = {10.18653/v1/2022.acl-long.82},
  eprint = {2109.07323},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2022.acl-long.82}
}

References

44 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Efthimia Aivaloglou, David Hoepelman, and Felienne Hermans. A grammar for spreadsheet formulas evaluated on two large datasets. In 2015 IEEE 15th International Working Conference on Source Code Analysis and Manipulation (SCAM), pages 121–130. find
  2. Daniel Andor, Luheng He, Kenton Lee, and Emily Pitler. Giving bert a calculator: Finding operations and arguments with reading comprehension. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Proce…
  3. Titus Barik, Kevin Lubick, Justin Smith, John Slankas, and Emerson Murphy-Hill. Fuse: a reproducible, extendable, internet-scale corpus of spreadsheets. In 2015 IEEE/ACM 12th Working Conference on Mining Software Repositories, pages 486–489. find
  4. Chris Chambers and Martin Erwig. Dimension inference in spreadsheets. In 2008 IEEE Symposium on Visual Languages and Human-Centric Computing, pages 123–130. find
  5. Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Wang, and William W Cohen. Open question answering over tables and text. arXiv preprint arXiv:2010.10439, 2020.
  6. Xinyun Chen, Petros Maniatis, Rishabh Singh, Charles Sutton, Hanjun Dai, Max Lin, and Denny Zhou. Spreadsheetcoder: Formula prediction from semi-structured context. In International Conference on Machine Learning, pages 1661–1672. PMLR, 2021.
  7. Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. arXiv preprint arXiv:2109.00122, 2021.
  8. Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian-Guang Lou, and Dongmei Zhang. Hitab: A hierarchical table dataset for question answering and natural language generation. arXiv preprint arXiv:2108.06712, 2021.
  9. Shing-Chi Cheung, Wanjun Chen, Yepang Liu, and Chang Xu. Custodes: automatic spreadsheet cell clustering and smell detection using strong and weak features. In Proceedings of the 38th International Conference on Software Engineering, pages 464–475, 2016. find
  10. Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. Structure-grounded pretraining for text-to-sql. arXiv preprint arXiv:2010.12773, 2020.
  11. Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. Turl: Table understanding through representation learning. arXiv preprint arXiv:2006.14806, 2020. — [arXiv:2006.14806](https://arxiv.org/abs/2006.14806) find
  12. Xiang Deng, Ahmed Hassan, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. Structure-grounded pretraining for text-to-sql. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 13…
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volum…
  14. Li Dong and Mirella Lapata. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 731–742, Melbourne, Australia, July 2018. Association for Computational Linguistics. find
  15. Haoyu Dong, Shijie Liu, Zhouyu Fu, Shi Han, and Dongmei Zhang. Semantic structure extraction for spreadsheet tables with a multi-task learning architecture. In Workshop on Document Intelligence at NeurIPS 2019, 2019. find
  16. Haoyu Dong, Shijie Liu, Shi Han, Zhouyu Fu, and Dongmei Zhang. Tablesense: Spreadsheet table detection with convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 69–76, 2019.
  17. Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hum…
  18. Julian Martin Eisenschlos, Syrine Krichene, and Thomas Müller. Understanding tables with intermediate pre-training. arXiv preprint arXiv:2010.00571, 2020.
  19. Marc Fisher and Gregg Rothermel. The euses spreadsheet corpus: a shared resource for supporting experimentation with spreadsheet dependability mechanisms. In Proceedings of the first workshop on End-user software engineering, pages 1–5, 2005. find
  20. Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 946–958, 2020.
  21. Majid Ghasemi Gol, Jay Pujara, and Pedro Szekely. Tabular cell classification using pre-trained cell embeddings. In 2019 IEEE International Conference on Data Mining (ICDM), pages 230–239. find
  22. IEEE Computer Society, 2019. find
  23. Julius Gonsior, Josephine Rehak, Maik Thiele, Elvis Koci, Michael Günther, and Wolfgang Lehner. Active learning for spreadsheet cell classification. In EDBT/ICDT Workshops, 2020. find
  24. Felienne Hermans and Emerson Murphy-Hill. Enron’s spreadsheets and related emails: A dataset and analysis. In 2015 IEEE/ACM 37th IEEE International Conference on Software Engineering.
  25. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Mueller, Francesco Piccinno, and Julian Eisenschlos. Tapas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320–4333, 2020.
  26. Hiroshi Iida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. Tabbie: Pretrained representations of tabular data. arXiv preprint arXiv:2105.02584, 2021.
  27. Zhihua Jin, Xin Jiang, Xingbo Wang, Qun Liu, Yong Wang, Xiaozhe Ren, and Huamin Qu. Numgpt: Improving numeracy ability of generative pre-trained models. arXiv preprint arXiv:2109.03137, 2021. — [arXiv:2109.03137](https://arxiv.org/abs/2109.03137) find
  28. Elvis Koci, Maik Thiele, Josephine Rehak, Oscar Romero, and Wolfgang Lehner. Deco: A dataset of annotated spreadsheets for layout and table recognition. In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1280– 1285. find
  29. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. Chen Liang, Mohammad Norouzi, Jonathan Berant, Quoc V Le,…
  30. Qian Liu, Bei Chen, Jiaqi Guo, Zeqi Lin, and Jian-guang Lou. Tapex: Table pre-training via learning a neural sql executor. arXiv preprint arXiv:2107.07653, 2021.
  31. Nafise Sadat Moosavi, Andreas Rücklé, Dan Roth, and Iryna Gurevych. Learning to reason for text generation from scientific tables. arXiv preprint arXiv:2104.08296, 2021. find
  32. Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–1480, 2…
  33. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026–8037, 2019.
  34. Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. Tabert: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413–8426, 2020.
  35. Ori Yoran, Alon Talmor, and Jonathan Berant. Turning tables: Generating examples from semi-structured tables for endowing language models with reasoning skills. arXiv preprint arXiv:2107.07261, 2021. find
  36. Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. Grappa: Grammar-augmented pre-training for table semantic parsing. arXiv preprint arXiv:2009.13845, 2020. find
  37. Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. arXiv preprint arXiv:2105.07624, 2021.
  38. Shuai Peng, Ke Yuan, Liangcai Gao, and Zhi Tang. Mathbert: A pre-trained model for mathematical formula understanding. arXiv preprint arXiv:2105.00377, 2021. — [arXiv:2105.00377](https://arxiv.org/abs/2105.00377) find
  39. Basel Shbita, Arunkumar Rajendran, Jay Pujara, and Craig A Knoblock. Parsing, representing and transforming units of measure. Modeling the World’s Systems, 2019. find
  40. Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, and Hiroya Takamura. Towards table-to-text generation with numerical reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics, pages 1451–1465, 2021. find
  41. Nan Tang, Ju Fan, Fangyi Li, Jianhong Tu, Xiaoyong Du, Guoliang Li, Sam Madden, and Mourad Ouzzani. Rpt: Relational pre-trained transformer is almost all you need towards democratizing data preparation. arXiv preprint arXiv:2012.02469, 2020. — [arXiv:2012.02469](https://arxiv.org/abs/2012.02469) find
  42. Avijit Thawani, Jay Pujara, and Filip Ilievski. Numeracy enhances the literacy of language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6960– 6967, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Ling… find
  43. Fei Wang, Kexuan Sun, Muhao Chen, Jay Pujara, and Pedro Szekely. Retrieving complex tables with multi-granular graph representation learning. arXiv preprint arXiv:2105.01736, 2021. — [arXiv:2105.01736](https://arxiv.org/abs/2105.01736) find
  44. Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. Tuta: Tree-based transformers for generally structured table pre-training. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1780–1790, 2021. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/cheng-2022-fortap-formulas-numerical-reasoning-aware/