Elamur Research⌕
☰
home / papers / 2021 / RYANSQL: Recursively Applying Sketch-based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databases

RYANSQL: Recursively Applying Sketch-based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databases

Paper · 2021 · DongHyun Choi, Myeong Cheol Shin, EungGyun Kim et al.

Summary

RYANSQL addresses Text-to-SQL: turning a natural-language question into an SQL query when the target database was not seen during training. Earlier benchmarks such as WikiSQL allowed only single tables and flat queries. RYANSQL targets the harder Spider benchmark, whose queries can contain nesting, multi-table JOINs, GROUPBY, ORDERBY and HAVING.

The authors introduce Statement Position Code (SPC). SPC labels where each sub-query sits, which turns a nested query into a set of flat SELECT statements. The model starts from the outermost statement and generates each sub-statement recursively. A sketch-based slot-filling decoder fills the slots of every SELECT. The encoder combines several components:

  • GloVe and character embeddings
  • dense CNNs
  • attention-based alignment of question tokens with columns and tables

Two simple input manipulation methods add further gains, and BERT can replace the base embeddings.

On Spider, RYANSQL reaches 58.2% test exact-match accuracy, and the RYANSQL v2 variant reaches 60.6%. This made v2 the best system that does not use database content, and third overall as of April 2020. The work shows that structured, recursive slot filling handles complex cross-domain SQL generation well.

Key points

  • SPC elements are NONE, UNION, INTERSECT, EXCEPT, WHERE, HAVING and PARALLEL. Generation begins at [NONE] and recurses until every SPC has its own SELECT statement.
  • Among systems that do not use database content, RYANSQL and RYANSQL v2 improve on the previous best, SLSQL, by 2.5 and 4.9 points of test exact-match accuracy.
  • On CSpider (Chinese), RYANSQL with Multilingual BERT scores 41.3% on dev and 34.7% on test.
  • The authors examined 345 dev-set failures of RYANSQL + BERT and could categorize 195 of them:
    • wrong column selection: 34.9%
    • wrong number of tables: 25.2%
    • wrong number of conditions: 11.3%
  • Accuracy depends on word overlap between the question and the schema names in the query, rising from 32.5% at the lowest overlap to 80.5% at the highest. Accuracy is weakly negatively correlated with the number of foreign keys (Pearson ρ = −0.22). Code is available at github.com/kakaoenterprise/RYANSQL.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sungkyunkwan UniversitySuwon · South Korea2 works · 173 citations

Also on the byline: Kakao Enterprise.

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 worksComputer Networks and Communications · 19 works

🧭 Topics 2Research subjects and languages the work is about.Semantic parsing & meaning representation · 135 worksCode & programming languages · 55 works

Cite

Cite as DongHyun Choi, Myeong Cheol Shin, EungGyun Kim and Dong Ryeol Shin. 2021. RYANSQL: Recursively Applying Sketch-based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databases. Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/coli_a_00403

@article{choi2021ryansql,
  title = {{RYANSQL: Recursively Applying Sketch-based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databases}},
  author = {DongHyun Choi, Myeong Cheol Shin, EungGyun Kim and Dong Ryeol Shin},
  journal = {Computational Linguistics},
  year = {2021},
  doi = {10.1162/coli\_a\_00403},
  url = {https://doi.org/10.1162/coli_a_00403}
}

References

30 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abadi, Martı́n, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenb… find
  2. Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. Computing Research Repository, arXiv:1607.06450.
  3. Beck, Thorsten, Asli Demirguc-Kunt, and Ross Levine. 2000. A new database on the structure and development of the financial sector. The World Bank Economic Review, 14(3):597–605. https://doi.org/10.1093/wber/14.3.597 Bogin, Ben, Matt Gardner, and Jonathan Berant. 2019. Global reasoning over databas… ↗
  4. Clark, Kevin, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. Electra: Pre-training text encoders as discriminators rather than generators. Computing Research Repository, arXiv:2003.10555. find
  5. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies…
  6. Minneapolis, MN. Dong, Li and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 33–43. Berlin. https://doi.org/10.18653/v1/P16-1004 Dong, Li and Mirella Lapata. 2018. Coarse-to-fine… ↗
  7. Guo, Tong and Huilin Gao. 2019. Content enhanced BERT-based text-to-SQL generation. Computing Research Repository, arXiv:1910.07179. — [arXiv:1910.07179](https://arxiv.org/abs/1910.07179) find
  8. He, Pengcheng, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. 2018. X Reinforce schema representation with context. Computing Research Repository, arXiv:1808.073837. — [arXiv:1808.07383](https://arxiv.org/abs/1808.07383) find
  9. Hillestad, Richard, James Bigelow, Anthony Bower, Federico Girosi, Robin Meili, Richard Scoville, and Roger Taylor. 2005. Can electronic medical record systems transform health care? Potential health benefits, savings, and costs. Health Affairs. 24(5):1103–1117. https://doi.org /10.1377/hlthaff.24.5… — [doi:10.1377/hlthaff.24.5.1103](https://doi.org/10.1377/hlthaff.24.5.1103) ↗
  10. Stockholm. Hwang, Wonseok, Jinyeong Yim, Seunghyun Park, and Minjoon Seo. 2019. A comprehensive exploration on WikiSQL with table-aware word contextualization. Computing Research Repository, arXiv:1902.01069. find
  11. Lee, Dongjun. 2019. Clause-wise and recursive decoding for complex and cross-domain text-to-SQL generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6047–6… find
  12. Lin, Kevin, Ben Bogin, Mark Neumann, Jonathan Berant, and Matt Gardner. 2019. Grammar-based neural text-to-SQL generation. Computing Research Repository, arXiv:1905.13326. — [arXiv:1905.13326](https://arxiv.org/abs/1905.13326) find
  13. Liu, Yinhan, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. Computing Research Repository. arXiv:1907.11692. find
  14. Lyu, Qin, Kaushik Chakrabarti, Shobhit Hathi, Souvik Kundu, Jianwen Zhang, and Zheng Chen. 2020. Hybrid ranking network for text-to-SQL, Microsoft Dynamics 365 AI. find
  15. Min, Qingkai, Yuefeng Shi, and Yue Zhang. 2019. A pilot study for Chinese SQL semantic parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3652–3658. Hong Ko… find
  16. Mou, Lili, Rui Men, Ge Li, Yan Xu, Lu Zhang, Rui Yan, and Zhi Jin. 2016. Natural language inference by tree-based convolution and heuristic matching. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, pages 130–136. Berlin. https://doi.org /10.18653/v1/P16-20… ↗
  17. Srivastava, Nitish, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1):1929–1958. find
  18. Srivastava, Rupesh K., Klaus Greff, and Jrgen Schmidhuber. 2015. Training very deep networks. Advances in Neural Information Processing Systems, pages 2377–2385. find
  19. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems, 30:5998–6008.
  20. Vinyals, Oriol, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. Advances in Neural Information Processing Systems, pages 2692–2700. find
  21. Wang, Bailin, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2020. RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7567–7578. Online. https://doi .or…
  22. Xu, Xiaojun, Chang Liu, and Dawn Song. 2017. SQLnet: Generating structured queries from natural language without reinforcement learning. Computing Research Repository, arXiv:1711.04436. find
  23. Yavuz, Semih, Izzeddin Gur, Yu Su, and Xifeng Yan. 2018. What it takes to achieve 100% condition accuracy on WikiSQL. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1702–1711. Brussels. https://doi.org/10.18653/v1/D18-1197 ↗
  24. Yoon, Deunsol, Dongbok Lee, and Sangkeun Lee. 2018. Dynamic self-attention: Computing attention over words dynamically for sentence embedding. Computing Research Repository, arXiv:1808.073837. — [arXiv:1808.07383](https://arxiv.org/abs/1808.07383) find
  25. Yu, Tao, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev. 2018a. TypeSQL: Knowledge-based type-aware neural text-to-SQL generation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 588–594.
  26. New Orleans, LA. Yu, Tao, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir Radev. 2018b. SyntaxSQLnet: Syntax tree networks for complex and cross-domain text-to-SQL task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1653… find
  27. Yu, Tao, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018c. Spider: A large-scale human-labeled data set for complex and cross-domain semantic parsing and text-to-SQL task. In Proceedings of the 20…
  28. Zelle, John Marvin and Raymond Joseph Mooney. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the Thirteenth National Conference On Artificial Intelligence, volume 2, pages 1050–1055.
  29. Portland, OR. Zhang, Rui, Tao Yu, Heyang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019. Editing-based SQL query generation for cross-domain context-dependent questions. In Proceedings of the 2019 Conference on Empirical Methods in Na…
  30. Hong Kong. Zhong, Victor, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. Computing Research Repository, arXiv:1709.00103. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/choi-2021-ryansql-recursively-applying-sketch-based/