Elamur Research⌕
☰
home / papers / 2023 / Transformers for Tabular Data Representation: A Survey of Models and Applications

Transformers for Tabular Data Representation: A Survey of Models and Applications

Paper · 2023 · Gilbert Badaro, Mohammed Saeed and Paolo Papotti

Summary

Transformer language models have changed how free text is encoded, and a growing body of work adapts them to tables, which hold a large amount of knowledge. These efforts make different assumptions and target different tasks, so they are hard to compare. This TACL survey sorts them using one shared machine-learning pipeline with five stages:

  • pre-training and fine-tuning data
  • input processing, mainly flattening 2-D tables into token sequences together with their textual context
  • architectural changes and pre-training objectives in the transformer encoder
  • downstream task models
  • the resulting token-, row-, column- or table-level representations

The authors compare systems at each stage. They show where tabular structure (rows, attributes, headers, hierarchies) requires departures from text-only transformers. They also explain how pre-trained tabular models are fine-tuned for six application families. The paper introduces no new model. Its contribution is a common reference frame for placing new work. It also argues that progress is held back by the lack of shared intrinsic benchmarks, by single-table assumptions, by bias and by computational cost.

Key points

  • Six downstream task families: verifying claims against tables (TFC), answering questions over tables (QA), mapping utterances to formal queries (SP), finding relevant tables (TR), inferring table metadata (TMP) and filling in table content (TCP).
  • Three table kinds: vertical relational tables, horizontal entity tables, and spreadsheets/matrix tables. Only a few systems take whole databases as input, namely GTR, GRAPPA and DTR.
  • Pre-training corpora vary by model:
    • TAPAS uses only Wikipedia Tables.
    • TaBERT combines Wikipedia Tables with WDC.
    • TUTA adds spreadsheets to those two sources.
    • MMR draws on NQ, OTT-QA and WikiSQL.
    • The authors recommend mixing sources to broaden coverage.
  • Open problems identified:
    • no common benchmark for intrinsic evaluation of representations
    • weak support for queries involving joins or multiple tables
    • inherited data bias
    • inefficient attention on structured input
    • the carbon footprint of training
  • Venue: published in TACL vol. 11 (2023), pp. 227–249, by authors at EURECOM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓EURECOMValbonne · France3 works · 1,074 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Gilbert Badaro, Mohammed Saeed and Paolo Papotti. 2023. Transformers for Tabular Data Representation: A Survey of Models and Applications. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00544

@article{badaro2023transformers,
  title = {{Transformers for Tabular Data Representation: A Survey of Models and Applications}},
  author = {Gilbert Badaro, Mohammed Saeed and Paolo Papotti},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.1162/tacl\_a\_00544},
  url = {https://doi.org/10.1162/tacl_a_00544}
}

References

105 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. FEVEROUS: Fact extraction and VERification over unstructured and structured information. In Thirty-fifth Conference on Neural Information Processing…
  2. Gilbert Badaro, Hussein Jundi, Hazem Hajj, and Wassim El-Hajj. 2018b. EmoWordNet: Automatic expansion of emotion lexicon using English Wordnet. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 86–93. https://doi.org/10.18653/v1/S18-2009 ↗
  3. Gilbert Badaro, Hussein Jundi, Hazem Hajj, Wassim El-Hajj, and Nizar Habash. 2018c. ArSEL: A large scale Arabic sentiment and emotion lexicon. OSACT, 326. find
  4. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15.
  5. Gilbert Badaro and Paolo Papotti. 2022. Transformers for tabular data representation: A tutorial on models and applications. Proceedings of the VLDB Endowment, 15(12):3746–3749. https://doi.org/10 .14778/3554821.3554890 find
  6. Gilbert Badaro, Ramy Baly, Hazem Hajj, Wassim El-Hajj, Khaled Bashir Shaban, Nizar Habash, Ahmad Al-Sallab, and Ali Hamdi. 2019. A survey of opinion mining in Arabic: A comprehensive system perspective covering challenges and advances in tools, resources, models, applications, and visualizations. AC… — [doi:10.1145/3295662](https://doi.org/10.1145/3295662) find
  7. Amir Bakarov. 2018. A survey of word embeddings evaluation methods. arXiv preprint arXiv:1801.09536v1. find
  8. Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415–1425. https://doi.org/10.3115/v1/P14-1133 ↗
  9. Gilbert Badaro, Ramy Baly, Hazem Hajj, Nizar Habash, and Wassim El-Hajj. 2014. A large scale Arabic sentiment lexicon for arabic opinion mining. In Proceedings of the EMNLP Chandra Sekhar Bhagavatula, Thanapon Noraset, and Doug Downey. 2015. TabEL: Entity linking in web tables. In International Sema…
  10. Shikha Bordia and Samuel R. Bowman. 2019. Identifying and reducing gender bias in word-level language models. In 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2019-Student Research Workshop, SRW 2019, pages 7–15…
  11. Wenhu Chen, Ming-Wei Chang, Eva Schlinger, William Yang Wang, and William W. Cohen. 2021. Open question answering over tables and text. In International Conference on Learning Representations. find
  12. Wenhu Chen, Jianshu Chen, Yu Su, Zhiyu Chen, and William Yang Wang. 2020a. Logical natural language generation from open-domain tables. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7929–7942. https://doi.org/10.18653/v1/2020.acl-main.708 ↗
  13. Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci. 2021. Deep neural networks and tabular data: A survey. arXiv preprint arXiv:2110.01889v3. https://doi.org/10.1109/TNNLS.2022.3229161 — [doi:10.1109/tnnls.2022.3229161](https://doi.org/10.1109/tnnls.2022.3229161) ↗
  14. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020b. Tab-Fact: A large-scale dataset for table-based fact verification. In International Conference on Learning Representations. find
  15. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter,… — Language Models are Few-Shot Learners
  16. Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang, and William Yang Wang. 2020c. HybridQA: A dataset of multi-hop question answering over tabular and textual data. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1026–1036, Online. Association for Computation…
  17. Zhiyu Chen, Wenhu Chen, Hanwen Zha, Xiyou Zhou, Yunkai Zhang, Sairam Sundaresan, and William Yang Wang. 2020d. Logic2Text: High-fidelity natural language generation from logical forms. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2096–2111. https://doi.org/10 .1865… find
  18. Riccardo Cappuzzo, Paolo Papotti, and Saravanan Thirumuruganathan. 2020. Creating embeddings of heterogeneous relational datasets for data integration tasks. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data, SIGMOD ’20, pages 1335–1349, New York, NY, USA. Associat… — [doi:10.1145/3318464.3389742](https://doi.org/10.1145/3318464.3389742) find
  19. Zhiyu Chen, Harini Eavani, Wenhu Chen, Yinyin Liu, and William Yang Wang. 2020e. Few-shot nlg with pre-trained language model. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 183–190. https://doi.org/10.18653/v1/2020.acl-main.18 ↗
  20. Yoan Chabot, Pierre Monnin, Frédéric Deuzé, Viet-Phi Huynh, Thomas Labbé, Jixiong Liu, and Raphaël Troncy. 2021. A framework for automatically interpreting tabular data at orange. In Proceedings of the 20th International Semantic Web Conference. find
  21. Zhoujun Cheng, Haoyu Dong, Ran Jia, Pengfei Wu, Shi Han, Fan Cheng, and Dongmei Zhang. 2022. FORTAP: Using formulas for numerical-reasoning-aware table pretraining. find
  22. Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020. Pre-training tasks for embedding-based In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1150–1166. https://doi.org/10.18653/v1/2022 .acl-long.82 — [doi:10.18653/v1/2022](https://doi.org/10.18653/v1/2022) ↗
  23. Radostin Cholakov and Todor Kolev. 2021. Transformers predicting the future. Applying attention in next-frame and time series forecasting. arXiv preprint arXiv:2108.08224v1. — [arXiv:2108.08224](https://arxiv.org/abs/2108.08224) find
  24. Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, David Belanger, Lucy Colwell, and Adrian Weller. 2020. Masked language modeling for proteins via linearly scalable long-context transformers. arXiv preprint arXiv:2006.035… — [arXiv:2006.03555](https://arxiv.org/abs/2006.03555) find
  25. (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics. find
  26. Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. 2022. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.06904v2. https://doi.org /10.21203/rs… — [doi:10.21203/rs.3.rs-1553541/v1](https://doi.org/10.21203/rs.3.rs-1553541/v1) ↗
  27. Haoyu Dong, Zhoujun Cheng, Xinyi He, Mengyu Zhou, Anda Zhou, Fan Zhou, Ao Liu, Shi Han, and Dongmei Zhang. 2022. Table pre-training: A survey on model architectures, pre-training objectives, and downstream tasks. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelli… find
  28. Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pre-training text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26–30, 2020. OpenReview.net. find
  29. Haoyu Dong, Shijie Liu, Zhouyu Fu, Shi Han, and Dongmei Zhang. 2019. Semantic structure extraction for spreadsheet tables with a multi-task learning architecture. In Workshop on Document Intelligence at NeurIPS 2019. find
  30. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. 2013. Recognizing Textual Entailment: Models and Applications. Synthesis Lectures on Human Language Technologies. Morgan and Claypool publishers. https://doi.org/10.1007/978-3-031 -02151-0 ↗
  31. Yihe Dong, Jean-Baptiste Cordonnier, and Andreas Loukas. 2021. Attention is not all you need: Pure attention loses rank doubly exponentially with depth. In International Conference on Machine Learning, pages 2793–2803. PMLR. find
  32. Sabyasachi Dash, Sushil Kumar Shakyawar, Mohit Sharma, and Sandeep Kaushik. 2019. Big data in healthcare: Management, analysis and future prospects. Journal of Big Data, 6(1):1–25. https://doi.org/10.1186/s40537-019-0217-0 ↗
  33. Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. 2020. TURL: Table understanding through representation learning. Proceedings of the VLDB Endowment, 14(3):307–319. https://doi.org/10.14778/3430915.3430921 — [doi:10.14778/3430915.3430921](https://doi.org/10.14778/3430915.3430921) ↗
  34. Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2020. An image is worth 16x16 words: Transformers for image recognition at scale. In Intern…
  35. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,…
  36. Lun Du, Fei Gao, Xu Chen, Ran Jia, Junshan Wang, Jiang Zhang, Shi Han, and Dongmei Zhang. 2021. TabularNet: A neural network architecture for understanding semantic structures of tabular data. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 322–331. find
  37. Jonathan Herzig, Thomas Mueller, Syrine Krichene, and Julian Eisenschlos. 2021. Open domain question answering over tables via dense retrieval. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 512…
  38. Julian Eisenschlos, Maharshi Gor, Thomas Mueller, and William Cohen. 2021. MATE: Multi-view attention for table transformer efficiency. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7606–7619. https://doi.org/10 .18653/v1/2021.emnlp-main.600 find
  39. Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Mueller, Francesco Piccinno, and Julian Eisenschlos. 2020. TaPas: Weakly supervised table parsing via pre-training. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4320–4333. https://doi.org/10 .18653/v1…
  40. Majid Ghasemi-Gol and Pedro A. Szekely. 2018. TabVec: Table vectors for classification of web tables. CoRR, abs/1802.06290. — [arXiv:1802.06290](https://arxiv.org/abs/1802.06290) find
  41. Michael Glass, Mustafa Canim, Alfio Gliozzo, Saneem Chemmengath, Vishwajeet Kumar, Rishav Chakravarti, Avirup Sil, Feifei Pan, Samarth Bharadwaj, and Nicolas Rodolfo Fauceglia. 2021. Capturing row and column semantics in transformer based question answering over tables. In Proceedings of the 2021 Co… find
  42. Kevin Hu, Snehalkumar’Neil’S Gaikwad, Madelon Hulsebos, Michiel A. Bakker, Emanuel Zgraggen, César Hidalgo, Tim Kraska, Guoliang Li, Arvind Satyanarayan, and Çağatay Demiralp. 2019. VizNet: Towards a large-scale visualization learning and benchmarking repository. In Proceedings of the 2019 CHI Co… find
  43. Heng Gong, Yawei Sun, Xiaocheng Feng, Bing Qin, Wei Bi, Xiaojiang Liu, and Ting Liu. 2020. TableGPT: Few-shot table-to-text generation with table structure reconstruction and content matching. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1978–1988. https://… find
  44. Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019. ClinicalBERT: Modeling clinical notes and predicting hospital readmission. arXiv preprint arXiv:1904.05342v3.
  45. Yuan Gong, Yu-An Chung, and James Glass. 2021. AST: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778v3. https://doi.org/10.21437/Interspeech.2021-698 ↗
  46. Hiroshi Iida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. 2021. TABBIE: Pretrained representations of tabular data. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3446–3456. https://doi.org/10…
  47. Sujay Kumar Jauhar, Peter Turney, and Eduard Hovy. 2016. TabMCQ: A dataset of general knowledge tables and multiple-choice questions. arXiv preprint arXiv:1602.03960v1. — [arXiv:1602.03960](https://arxiv.org/abs/1602.03960) find
  48. Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. 2021. Revisiting deep learning models for tabular data. Advances in Neural Information Processing Systems, 34:18932–18943. find
  49. Georgios Karagiannis, Mohammed Saeed, Paolo Papotti, and Immanuel Trummer. 2020. Scrutinizer: A mixed-initiative approach to large-scale, data-driven claim verification. Proceedings of the VLDB Endowment, 13(11):2508–2521. https://doi.org/10 .14778/3407790.3407841 find
  50. Poonam Gupta and Vishal Gupta. 2012. A survey of text question answering techniques. International Journal of Computer Applications, 53(4). https://doi.org/10.5120/8406-2030 ↗
  51. Marcin Kaszkiel and Justin Zobel. 1997. Passage retrieval revisited. In SIGIR, pages 178–185. ACM. https://doi.org/10.1145/278459.258561 — [doi:10.1145/278459.258561](https://doi.org/10.1145/278459.258561) ↗
  52. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistic…
  53. Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. 2020. Transformers are RNNs: Fast autoregressive transformers with linear attention. In International Conference on Machine Learning, pages 5156–5165. PMLR. find
  54. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In International Conference on Learning Representations.
  55. George Katsogiannis-Meimarakis and Georgia Koutrika. 2021. A deep dive into deep learning approaches for text-to-SQL systems. In Proceedings of the 2021 International Conference on Management of Data, pages 2846–2851. https://doi.org/10 .1145/3448016.3457543 find
  56. Oliver Lehmberg, Dominique Ritze, Robert Meusel, and Christian Bizer. 2016. A large public corpus of web tables containing time and context metadata. In Proceedings of the 25th International Conference Companion on World Wide Web, pages 75–76. https://doi.org/10.1145/2872518.2889386 — [doi:10.1145/2872518.2889386](https://doi.org/10.1145/2872518.2889386) ↗
  57. Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. 2021. Transformers in vision: A survey. arXiv preprint arXiv:2101.01169. — [arXiv:2101.01169](https://arxiv.org/abs/2101.01169) find
  58. Junyi Li, Tianyi Tang, Wayne Xin Zhao, Zhicheng Wei, Nicholas Jing Yuan, and Ji-Rong Wen. 2021. Few-shot knowledge graph-to-text generation with pretrained language models. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1558–1568. find
  59. Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In International Conference on Learning Representations.
  60. Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019. Text generation from knowledge graphs with graph transformers. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi…
  61. Yuliang Li, Jinfeng Li, Yoshihiko Suhara, AnHai Doan, and Wang-Chiew Tan. 2020. Deep entity matching with pre-trained language models. Proceedings of the VLDB Endowment, 14(1):50–60. https://doi.org/10 .14778/3421424.3421431 find
  62. Qian Liu, Bei Chen, Jiaqi Guo, Zeqi Lin, and Jian-guang Lou. 2021a. TAPEX: Table pre-training via learning a neural SQL executor. arXiv preprint arXiv:2107.07653v3.
  63. Daniel Z. Korman, Eric Mack, Jacob Jett, and Allen H. Renear. 2018. Defining textual entailment. Journal of the Association for Information Science and Technology, 69(6):763–772. https://doi.org/10.1002/asi.24007 — [doi:10.1002/asi.24007](https://doi.org/10.1002/asi.24007) ↗
  64. Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. 2021b. Mitigating political bias in language models through reinforced calibration. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI, pages 14857–14866. AAAI Press. https://doi.org/10.1609/aaai.v35i17 .… ↗
  65. Bogdan Kostić, Julian Risch, and Timo Möller. 2021. Multi-modal retrieval of tables and texts using tri-encoder models. In Proceedings of the 3rd Workshop on Machine Reading for Question Answering, pages 82–91, Punta Cana, Dominican Republic. Association for Computational Linguistics. https://doi.… ↗
  66. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692. ing (EMNLP), pages 1173–1186. https://doi.org/10.18653/v1/2020.emnlp-main.89 ↗
  67. Moin Nadeem, Anna Bethke, and Siva Reddy. 2021. StereoSet: Measuring stereotypical bias in pretrained language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long…
  68. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1470–1…
  69. Preslav Nakov, David Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barrón-Cedeño, Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. 2021. Automated fact-checking for assisting human fact-checkers. In Proceedings of the Thirtieth International Joint Conference on Artificial Int… — [doi:10.24963/ijcai.2021](https://doi.org/10.24963/ijcai.2021) find
  70. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9. find
  71. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1–67.
  72. J. Neeraja, Vivek Gupta, and Vivek Srikumar. 2021. Incorporating external knowledge to enhance tabular reasoning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics, pages 2799–2809, Online. Association for Computational Linguistics.… find
  73. Wullianallur Raghupathi and Viju Raghupathi. 2014. Big data analytics in healthcare: Promise and potential. Health Information Science and Systems, 2(1):1–10. https://doi.org/10.1186/2047-2501-2-3, PubMed: 25825667 — [doi:10.1186/2047-2501-2-3](https://doi.org/10.1186/2047-2501-2-3) ↗
  74. Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902–4912, Online. Association for Computational Linguistic… find
  75. Feifei Pan, Mustafa Canim, Michael Glass, Alfio Gliozzo, and Peter Fox. 2021. CLTR: An end-to-end, transformer-based system for cell-level table retrieval and table question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internation… find
  76. Stephen E. Robertson, Steve Walker, Susan Jones, Micheline M. Hancock-Beaulieu, and Mike Gatford. 1995. Okapi at TREC-3. Nist Special Publication Sp, 109:109.
  77. Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process-Roy Schwartz, Jesse Dodge, Noah A. Smith, and…
  78. Sofia Serrano and Noah A. Smith. 2019. Is attention interpretable? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2931–2951, Florence, Italy. Association for Computational Linguistics. https://doi.org/10 .18653/v1/P19-1282
  79. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, pages 3104–3112, Cambridge, MA, USA. MIT Press.
  80. Ravid Shwartz-Ziv and Amitai Armon. 2021. Tabular data: Deep learning is not all you need. In 8th ICML Workshop on Automated Machine Learning (AutoML). https://doi.org/10.1016/j.inffus.2021.11.011 — [doi:10.1016/j.inffus.2021.11.011](https://doi.org/10.1016/j.inffus.2021.11.011) ↗
  81. Nan Tang, Ju Fan, Fangyi Li, Jianhong Tu, Xiaoyong Du, Guoliang Li, Samuel Madden, and Mourad Ouzzani. 2021. RPT: Relational pre-trained transformer is almost all you need towards democratizing data preparation. Proceedings of the VLDB Endowment, 14(8):1254–1261. https://doi.org/10 .14778/3457390.34… find
  82. Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein. 2021. SAINT: Improved neural networks for tabular data via row attention and contrastive pre-training. arXiv preprint arXiv:2106.01342v1. — [arXiv:2106.01342](https://arxiv.org/abs/2106.01342) find
  83. Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2020. Efficient transformers: A survey. CoRR, abs/2009.06732.
  84. Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2020. Energy and policy considerations for modern deep learning research. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, pages 13693–13696. AAAI Press. https://doi.org/10.1609/aaai.v34i09 .7123 ↗
  85. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: A large-scale dataset for fact extraction and verification. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, pages 809–819.…
  86. Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, and Hiroya Takamura. 2021. Towards table-to-text generation with numerical reasoning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference o… find
  87. James Thorne, Majid Yazdani, Marzieh Saeidi, Fabrizio Silvestri, Sebastian Riedel, and Alon Halevy. 2021. Database reasoning over text. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processin… find
  88. Yoshihiko Suhara, Jinfeng Li, Yuliang Li, Dan Zhang, Çağatay Demiralp, Chen Chen, and Wang-Chiew Tan. 2022. Annotating columns with pre-trained language models. In Proceedings of the 2022 International Conference on Management of Data, pages 1493–1503. https://doi.org/10 .1145/3514221.3517906 find
  89. Roger P. G. Van Gompel and Martin J. Pickering. 2007. Syntactic parsing. In The Oxford handbook of psycholinguistics, pages 289–307. https://doi.org/10.1093/oxfordhb/9780198568971.013.0017 — [doi:10.1093/oxfordhb](https://doi.org/10.1093/oxfordhb) ↗
  90. Yibo Sun, Zhao Yan, Duyu Tang, Nan Duan, and Bing Qin. 2019. Content-based table retrieval for web queries. Neurocomputing, 349:183–189. https://doi.org/10 .1016/j.neucom.2018.10.033 find
  91. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. preprint arXiv:2205.09843v1. https://doi .org/10.186…
  92. Enzo Veltri, Gilbert Badaro, Mohammed Saeed, and Paolo Papotti. 2023. Data ambiguity profiling for the generation of training examples. In 39th IEEE International Conference on Data Engineering, ICDE 2023, Anaheim, California, USA, April 3–7, 2023. IEEE. find
  93. Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zett… find
  94. Enzo Veltri, Donatello Santoro, Gilbert Badaro, Mohammed Saeed, and Paolo Papotti. 2022. Pythia: Unsupervised generation of ambiguous textual claims from relational data. In Proceedings of the 2022 International Conference on Management of Data, SIGMOD ’22, pages 2409–2412, New York, NY, USA. Associ… — [doi:10.1145/3514221.3520164](https://doi.org/10.1145/3514221.3520164) find
  95. Jingfeng Yang, Aditya Gupta, Shyam Upadhyay, Luheng He, Rahul Goel, and Shachi Paul. 2022. TableFormer: Robust transformer modeling for table-text encoding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 528–537. https://doi.… ↗
  96. Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. 2020. Investigating gender bias in language models using causal mediation analysis. Advances in Neural Information Processing Systems, 33:12388–12401. find
  97. Xiaoyu Yang and Xiaodan Zhu. 2021. Exploring decomposition for table-based fact verification. In EMNLP, pages 1045–1052, Punta Cana, Dominican Republic. Association for Computational Linguistics. https://doi.org/10 .18653/v1/2021.findings-emnlp.90 find
  98. Fei Wang, Kexuan Sun, Muhao Chen, Jay Pujara, and Pedro Szekely. 2021a. Retrieving complex tables with multi-granular graph representation learning. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SI-GIR ’21, pages 1472–1482, New Yo… find
  99. Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020. TaBERT: Pretraining for joint understanding of textual and tabular data. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8413–8426, Online. Association for Computational Linguisti…
  100. Sinong Wang, Belinda Z. Li, Madian Khabsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.04768v3. find
  101. Tao Yu, Chien-Sheng Wu, Xi Victoria Lin, Bailin Wang, Yi Chern Tan, Xinyi Yang, Dragomir Radev, Richard Socher, and Caiming Xiong. 2021. GraPPa: Grammar-augmented pre-training for table semantic parsing. In International Conference on Learning Representations. find
  102. Zhiruo Wang, Haoyu Dong, Ran Jia, Jia Li, Zhiyi Fu, Shi Han, and Dongmei Zhang. 2021b. TUTA: Tree-based transformers for generally structured table pre-training. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pages 1780–1790. https://doi.org/10 .1145/3447548.3… find
  103. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing Zhiruo Wang, Zhengbao Jiang, Eric Nyberg, and Gra…
  104. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103v7.
  105. Shuo Zhang and Krisztian Balog. 2017. En-tiTables: Smart assistance for entity-focused tables. In Proceedings of the 40th Interna- find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/badaro-2023-transformers-tabular-data-representation-survey/