Elamur Research⌕
☰
home / papers / 2019 / Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing

Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing

Summary

Text-to-SQL parsers have usually been trained and tested on one fixed database or on single-table setups, so they rarely needed to understand how a schema is organized. The SPIDER benchmark changes this: test questions target complex databases never seen during training. Bogin, Berant and Gardner argue that, in this zero-shot setting, the links between tables and columns (especially foreign/primary keys) are what tell a parser whether a join is needed. They convert each schema into a graph whose nodes are tables and columns and whose typed edges mark table–column membership and key relations. A gated GNN then computes node representations, weighted by how relevant each item appears to be for the current question. These structure-aware vectors are fed into the grammar-based encoder-decoder of Krishnamurthy et al. (2017). On the decoder side, a self-attention component favors schema items related to those already generated. On the SPIDER test set, accuracy rises to 39.4%, roughly double the previous state of the art. The largest gains come on questions that involve several tables.

Key points

  • Data and setup: experiments use SPIDER (7,000 / 1,034 / 2,147 train/dev/test examples). The base parser uses an AllenNLP SQL grammar covering 98.3% of SPIDER queries, and the code is released at github.com/benbogin/spider-schema-gnn.
  • Main results: test accuracy is 39.4%, compared with 33.8% for the same parser without the GNN and 19.7% for the prior best system (SyntaxSQLNet). On dev, multi-table accuracy climbs from 14.6% to 26.8%.
  • Ablations (dev set):
    • Dropping the decoder self-attention costs about 2 points overall and 6.5 points on multi-table questions.
    • Using the GNN only through self-attention gives 35.9%.
    • Removing the relevance score gives 37.0%.
  • Relevance headroom: an oracle relevance score raises dev accuracy to 54.3%, which suggests better schema-item relevance estimation is a promising direction.
  • Join quality: malformed joins (a table joined with itself, or tables with no key connection) fall from 83.4% of join-containing predictions without the GNN to 15.6% with it.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics Code & programming languages Semantic parsing & meaning representation

 Institutions

🔬Allen Institute for Artificial IntelligenceSeattle · United States169 works · 65,355 citations🎓Tel Aviv UniversityTel Aviv · Israel54 works · 7,966 citations

Cite

Cite as Ben Bogin, Jonathan Berant and Matt Gardner. 2019. Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p19-1448

@inproceedings{bogin2019representing,
  title = {{Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing}},
  author = {Ben Bogin, Jonathan Berant and Matt Gardner},
  booktitle = {Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  doi = {10.18653/v1/p19-1448},
  url = {https://doi.org/10.18653/v1/p19-1448}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Bahdanau, K. Cho, and Y. Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR). our card
  2. J. Cheng, S. Reddy, V. Saraswat, and M. Lapata. 2017. Learning structured natural language representations for semantic parsing. In Association for Computational Linguistics (ACL). find
  3. K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1724–1734. find
  4. N. De Cao, W. Aziz, and I. Titov. 2019. Question answering by reasoning across documents with graph convolutional networks. In North American Association for Computational Linguistics (NAACL). find
  5. C. Finegan-Dollak, J. K. Kummerfeld, L. Zhang, K. Ramanathan, S. Sadasivam, R. Zhang, and D. Radev. 2018. Improving text-to-sql evaluation methodology. In Association for Computational Linguistics (ACL). our card
  6. M. Gardner, J. Grus, M. Neumann, O. Tafjord, P. Dasigi, N. Liu, M. Peters, M. Schmitz, and L. Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. arXiv preprint arXiv:1803.07640. our card
  7. S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735– 1780. find
  8. S. Iyer, I. Konstas, A. Cheung, J. Krishnamurthy, and L. Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Association for Computational Linguistics (ACL). find
  9. J. Krishnamurthy, P. Dasigi, and M. Gardner. 2017. Neural semantic parsing with type constraints for semi-structured tables. In Empirical Methods in Natural Language Processing (EMNLP). find
  10. Y. Li, D. Tarlow, M. Brockschmidt, and R. Zemel. 2016. Gated graph sequence neural networks. In International Conference on Learning Representations (ICLR). find
  11. K. Lin, B. Bogin, M. Neumann, J. Berant, and M. Gardner. 2019. Grammar-based neural text-to-sql generation. arXiv preprint arXiv:1905.13326. — [arXiv:1905.13326](https://arxiv.org/abs/1905.13326) find
  12. M. Rabinovich, M. Stern, and D. Klein. 2017. Abstract syntax networks for code generation and semantic parsing. In Association for Computational Linguistics (ACL). find
  13. D. Sorokin and I. Gurevych. 2018. Modeling semantics with gated graph neural networks for knowledge base question answering. In International Conference on Computational Linguistics (COLING). find
  14. C. Xiao, M. Dymetman, and C. Gardent. 2016. Sequence-based structured prediction for semantic parsing. In Association for Computational Linguistics (ACL). find
  15. P. Yin and G. Neubig. 2017. A syntactic neural model for general-purpose code generation. In Association for Computational Linguistics (ACL), pages 440– 450. our card
  16. T. Yu, M. Yasunaga, K. Yang, R. Zhang, D. Wang, Z. Li, and D. Radev. 2018a. SyntaxSQLNet: Syntax tree networks for complex and cross-domaintext-to-SQL task. In Empirical Methods in Natural Language Processing (EMNLP). find
  17. T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev. 2018b. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. In Empirical Methods in Natural Language Processing (EMNLP). our card
  18. M. Zelle and R. J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In Association for the Advancement of Artificial Intelligence (AAAI), pages 1050–1055. our card
  19. L. S. Zettlemoyer and M. Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Uncertainty in Artificial Intelligence (UAI), pages 658– 666. our card
  20. V. Zhong, C. Xiong, and R. Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bogin-2019-representing-schema-structure-graph-neural/