Elamur Research⌕
☰
home / papers / 2017 / Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning

Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning

Paper · 2017 · Victor Zhong, Caiming Xiong and Richard Socher

Summary

The paper addresses natural language interfaces to relational databases. The authors aim to let a person ask a question in ordinary language and obtain the corresponding SQL query without knowing the query language. To this end, they propose the Seq2SQL model. It relies on the typical structure of SQL and splits generation into three steps: selecting the aggregation operation, selecting the column for SELECT, and constructing the WHERE conditions. The first two steps are trained with cross entropy. The model learns the WHERE conditions using policy gradient: the generated query is executed against the database, and its result serves as the reward. As a result, the model is not penalized if it writes equivalent conditions in a different order. The second contribution is the WikiSQL corpus. It contains questions about tables from Wikipedia, paraphrased by people via crowdsourcing, together with the corresponding queries. Along with the corpus, the authors release an engine for executing these queries. On WikiSQL, Seq2SQL clearly outperforms the best semantic parser available at the time. The work is significant because it provided a large benchmark dataset for the text-to-SQL task. It also showed that reinforcement learning combined with the structural constraints of SQL yields better results.

Key points

  • WikiSQL contains 80,654 "question + SQL query" examples based on 24,241 tables from Wikipedia. This is roughly an order of magnitude more than previous semantic parsing datasets with logical forms. The tables are released in JSON format and as an SQL database.
  • How the data were collected. For each selected table, 6 queries were generated automatically. The aggregation could be empty or COUNT, and for numeric columns also MIN or MAX. The conditions allow the = operator, and for numbers also > and <. The template questions were paraphrased by Amazon Mechanical Turk workers. Each paraphrase was verified by two more people. Only paraphrases whose character-level edit distance from the template question exceeds 10 were kept.
  • Results. Seq2SQL achieves an execution accuracy of 59.4% versus 35.9% for the Seq2Seq parser of Dong & Lapata (2016). Logical form accuracy rises from 23.4% to 48.3%. The augmented pointer network, as an intermediate baseline, reaches 53.3% execution accuracy.
  • Architecture. The encoder is a two-layer bidirectional LSTM. Its input consists of the column names, the SQL vocabulary, and the question itself, separated by special tokens. The aggregation is determined by a classifier, and the SELECT column is chosen by a pointer mechanism. The WHERE conditions are generated by a pointer decoder trained jointly with RL on a mixed objective function.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Networks and Communications · 64 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Machine learning · 51 works

Cite

Cite as Victor Zhong, Caiming Xiong and Richard Socher. 2017. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning. arXiv:1709.00103.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1709.00103 · arXiv 1709.00103

@misc{zhong2017seq2sql,
  title = {{Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning}},
  author = {Victor Zhong, Caiming Xiong and Richard Socher},
  year = {2017},
  doi = {10.48550/arxiv.1709.00103},
  eprint = {1709.00103},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1709.00103}
}

References

42 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. I. Androutsopoulos, G.D. Ritchie, and P. Thanisch. Natural language interfaces to databases - an introduction. 1995. find
  2. Yoav Artzi and Luke S. Zettlemoyer. Bootstrapping semantic parsers from conversations. In EMNLP, 2011. find
  3. Yoav Artzi and Luke S. Zettlemoyer. Weakly supervised learning of semantic parsers for mapping instructions to actions. TACL, 1:49–62, 2013.
  4. Thorsten Beck, Asli Demirgüç-Kunt, and Ross Levine. A new database on the structure and development of the financial sector. The World Bank Economic Review, 14(3):597–605, 2000. find
  5. Irwan Bello, Hieu Pham, Quoc V. Le, Mohammad Norouzi, and Samy Bengio. Neural combinatorial optimization with reinforcement learning. ICLR, 2017. find
  6. Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on freebase from question-answer pairs. In EMNLP, 2013.
  7. Chandra Bhagavatula, Thanapon Noraset, and Doug Downey. Methods for exploring and mining tables on wikipedia. In IDEA@KDD, 2013. find
  8. Qingqing Cai and Alexander Yates. Large-scale semantic parsing via schema matching and lexicon extension. In ACL, 2013. find
  9. Li Dong and Mirella Lapata. Language to logical form with neural attention. ACL, 2016.
  10. Alessandra Giordani and Alessandro Moschitti. Translating questions to SQL queries with generative parsers discriminatively reranked. In COLING, 2012. find
  11. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O. K. Li. Incorporating copying mechanism in sequence-to-sequence learning. ACL, 2016.
  12. Kelvin Guu, Panupong Pasupat, Evan Zheran Liu, and Percy Liang. From language to programs: Bridging reinforcement learning and maximum marginal likelihood. In ACL, 2017.
  13. Kazuma Hashimoto, Caiming Xiong, Yoshimasa Tsuruoka, and Richard Socher. A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks. arXiv, cs.CL 1611.01587, 2016.
  14. Richard Hillestad, James Bigelow, Anthony Bower, Federico Girosi, Robin Meili, Richard Scoville, and Roger Taylor. Can electronic medical record systems transform health care? potential health benefits, savings, and costs. Health affairs, 24(5):1103–1117, 2005. find
  15. Sepp Hochreiter and Jurgen Schmidhuber. Long short-term memory. Neural computation, 1997. find
  16. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. Learning a neural semantic parser from user feedback. In ACL, 2017. find
  17. Diederik P. Kingma and Jimmy Ba. abs/1412.6980, 2014.
  18. Adam: A method for stochastic optimization. Chen Liang, Jonathan Berant, Quoc V. Le, Ken Forbus, and Ni Lao. Neural symbolic machines: Learning semantic parsers on freebase with weak supervision. In ACL, 2017. find
  19. Percy Liang, Michael I. Jordan, and Dan Klein. Learning dependency-based compositional semantics. Computational Linguistics, 39:389–446, 2011. find
  20. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pp. 55–60, 2014.
  21. Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. ICLR, 2017. arXiv, Lili Mou, Zhengdong Lu, Hang Li, and Zhi Jin. Coupling distributed and symbolic execution for natural language queries. In ICML, 2017.
  22. Arvind Neelakantan, Quoc V. Le, Martı́n Abadi, Andrew McCallum, and Dario Amodei. Learning a natural language interface with neural programmer. In ICLR, 2017. find
  23. Eric WT Ngai, Li Xiu, and Dorothy CK Chau. Application of data mining techniques in customer relationship management: A literature review and classification. Expert systems with applications, 36(2):2592–2602, 2009. find
  24. Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. In ACL, 2015.
  25. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. Glove: Global vectors for word representation. In EMNLP, 2014.
  26. Ana-Maria Popescu, Oren Etzioni, and Henry Kautz. Towards a theory of natural language interfaces to databases. In Proceedings of the 8th International Conference on Intelligent User Interfaces, pp. 149–157. ACM, 2003. find
  27. Patti J. Price. Evaluation of spoken language systems: The ATIS domain. 1990. find
  28. Siva Reddy, Mirella Lapata, and Mark Steedman. Large-scale semantic parsing without question-answer pairs. TACL, 2:377–392, 2014. find
  29. John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. Gradient estimation using stochastic computation graphs. In NIPS, 2015. find
  30. Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. ICLR, 2017.
  31. Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15:1929–1958, 2014. find
  32. Janez Starc and Dunja Mladenic. Joint learning of ontology and semantic parser from text. Intelligent Data Analysis, 21:19–38, 2017. find
  33. Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems, pp. 1057–1063, 2000. find
  34. Lappoon R. Tang and Raymond J. Mooney. Using multiple clause constructors in inductive logic programming for semantic parsing. In ECML, 2001. find
  35. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In NIPS, 2015. find
  36. Yushi Wang, Jonathan Berant, and Percy Liang. Building a semantic parser overnight. In ACL, 2015. find
  37. Yuk Wah Wong and Raymond J. Mooney. Learning synchronous grammars for semantic parsing with lambda calculus. In ACL, 2007.
  38. Caiming Xiong, Victor Zhong, and Richard Socher. Dynamic coattention networks for question answering. ICLR, 2017.
  39. Pengcheng Yin, Zhengdong Lu, Hang Li, and Ben Kao. Neural enquirer: Learning to query tables. In ACL, 2016. find
  40. John M. Zelle and Raymond J. Mooney. Learning to parse database queries using inductive logic programming. In AAAI/IAAI, Vol. 2, 1996.
  41. Luke S. Zettlemoyer and Michael Collins. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In Uncertainty in Artificial Intelligence, 2005.
  42. Luke S. Zettlemoyer and Michael Collins. Online learning of relaxed ccg grammars for parsing to logical form. In EMNLP-CoNLL, 2007. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhong-2017-seq2sql-generating-structured-queries-natural/