Elamur Research⌕
☰
home / papers / 2019 / A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization

A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization

Paper · 2019 · Wonseok Hwang, Jinyeong Yim, Seunghyun Park et al.

Summary

The paper addresses NL2SQL on WikiSQL. The task is to turn an English question about a single Wikipedia table into an executable SQL query, using only the table's headers. The authors present SQLova, which pairs BERT with a decoder constrained by SQL syntax.

In the encoder, the question and every column header go into BERT as one sequence, separated by [SEP] tokens. Segment ids distinguish question tokens from header tokens, so the resulting word vectors reflect the table schema. On top sits an NL2SQL layer inspired by SQLNet. It fills six slots independently: select column, aggregation, number of conditions, where columns, operators and values. It uses column attention, and condition values are predicted as start/end spans in the question.

SQLova reaches 83.6% logical form and 89.6% execution accuracy on the test set, beating the prior best system by 8.2 and 2.5 points. In contrast, simply attaching a seq2seq decoder to BERT performs poorly. An error analysis and a crowdsourced human baseline indicate that WikiSQL is nearly saturated, which is relevant for designing future NL2SQL benchmarks.

Key points

  • Dataset: WikiSQL has 80,654 questions over 24,241 tables. Models see headers only, not cell contents.
  • Results: Test accuracy is 83.6% (logical form) and 89.6% (execution), gains of +8.2 and +2.5 over the previous state of the art. BERT with a plain seq2seq decoder underperforms, so architecture design around BERT matters.
  • Dev-set error analysis: Of 1,533 logical-form mismatches, 100 were sampled.
    • 26 cannot be answered from the given information: headers lack needed info, several valid queries exist, nested SQL is required, or the question is ambiguous.
    • Among the 74 answerable cases, 49 have faulty ground truth, 45 of them in the aggregation operator. SQLova gets 41 of those 49 right.
  • Human baseline: 246 Amazon Mechanical Turk workers answered 1,551 test examples with 88.3% accuracy. SQLova scores 86.8% on the same examples without execution-guided decoding and 91.0% with it.
  • Release: Code and human evaluation data are at github.com/naver/sqlova (Clova AI, NAVER; NeurIPS 2019).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢NaverSeongnam-si · South Korea20 works · 12,146 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Networks and Communications · 100 works

Cite

Cite as Wonseok Hwang, Jinyeong Yim, Seunghyun Park and Minjoon Seo. 2019. A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization. arXiv:1902.01069.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1902.01069 · arXiv 1902.01069

@misc{hwang2019comprehensive,
  title = {{A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization}},
  author = {Wonseok Hwang, Jinyeong Yim, Seunghyun Park and Minjoon Seo},
  year = {2019},
  doi = {10.48550/arxiv.1902.01069},
  eprint = {1902.01069},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1902.01069}
}

References

21 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. NAACL, abs/1810.04805.
  2. When measuring human performance, errors in ground truth are manually corrected by experts (us). find
  3. Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 33–43, Berlin, Germany. Association for Computational Linguistics.
  4. Li Dong and Mirella Lapata. 2018. Coarse-to-fine decoding for neural semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 731–742, Melbourne, Australia. Association for Computational Linguistics. find
  5. Po-Sen Huang, Chenglong Wang, Rishabh Singh, Wen tau Yih, and Xiaodong He. 2018. Natural language to structured query generation via meta-learning. In NAACL. find
  6. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963–973, Vancouver, Canada… find
  7. Robin Jia and Percy Liang. 2016. Data recombination for neural semantic parsing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12–22, Berlin, Germany. Association for Computational Linguistics. find
  8. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60.
  9. Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. 2018. The natural language decathlon: Multitask learning as question answering. arXiv preprint arXiv:1806.08730.
  10. Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolog…
  11. Tianze Shi, Kedar Tatwawadi, Kaushik Chakrabarti, Yi Mao, Oleksandr Polozov, and Weizhu Chen. 2018. Incsql: Training incremental text-to-sql parsers with non-deterministic oracles. CoRR, abs/1809.05054. — [arXiv:1809.05054](https://arxiv.org/abs/1809.05054) find
  12. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Proces…
  13. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692–2700. Curran Associates, Inc. find
  14. Chenglong Wang, Marc Brockschmidt, and Rishabh Singh. 2017. Pointing out SQL queries from text. Technical Report MSR-TR-2017-45, Microsoft. find
  15. Chenglong Wang, Po-Sen Huang, Alex Polozov, Marc Brockschmidt, and Rishabh Singh. 2018a. Execution-guided neural program decoding. In ICML workshop on Neural Abstract Machines and Program Induction v2 (NAMPI). find
  16. Wenlu Wang, Yingtao Tian, Hongyu Xiong, Haixun Wang, and Wei-Shinn Ku. 2018b. A transfer-learnable natural language interface for databases. CoRR, abs/1809.02649. — [arXiv:1809.02649](https://arxiv.org/abs/1809.02649) find
  17. Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, N… find
  18. Xiaojun Xu, Chang Liu, and Dawn Song. 2017. Sqlnet: Generating structured queries from natural language without reinforcement learning. CoRR, abs/1711.04436.
  19. Pengcheng Yin and Graham Neubig. 2018. TRANX: A transition-based neural abstract syntax parser for semantic parsing and code generation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 7–12, Brussels, Belgium. Association for Co… find
  20. Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. 2018. TypeSQL: Knowledge-based type-aware neural text-to-SQL generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Pa…
  21. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/hwang-2019-comprehensive-exploration-wikisql-table-aware/