Elamur Research⌕
☰
home / papers / 2020 / DuSQL: A Large-Scale and Pragmatic Chinese Text-to-SQL Dataset

DuSQL: A Large-Scale and Pragmatic Chinese Text-to-SQL Dataset

Summary

Most text-to-SQL resources (ATIS, WikiSQL, Spider) are English. Their SQL queries are designed around syntactic coverage, not around what users actually ask. DuSQL fills this gap with a cross-domain Chinese text-to-SQL dataset of 200 databases, 813 tables and 23,797 question/SQL pairs.

The authors first manually categorised real user questions from Baidu search, customer-service bots and data-analysis bots. They found that queries requiring arithmetic over rows or columns are frequent in practice but almost absent from existing corpora.

To build the dataset, they assembled multi-table databases from web tables, mainly Baidu Baike, linking the tables through entity-based foreign keys. SQL queries were then generated automatically from a grammar constrained by each database. Each query was turned into a pseudo question, which annotators rewrote as natural Chinese. Uncertain pairs were flagged for human review.

Three open-source parsers were evaluated as baselines. An extended IRNet (IRNetExt), which adds calculation rules and value prediction, scored far above the original IRNet. Even so, overall accuracy remains modest, so DuSQL is a challenging and realistic benchmark.

Key points

  • Scale and comparison: DuSQL contains 200 databases, 813 tables and 23,797 question/SQL pairs. This is more than double Spider's 10,181 pairs over 200 databases.
  • Real-world query analysis: The authors analysed 1,000 Baidu search questions, 1,500 customer-service questions and 500 data-analysis questions. Each was sorted into one of five types: matching, sorting, clustering, calculation and other.
    • Calculation questions make up 8% of customer-service queries.
    • They dominate the data-analysis queries.
  • Calculation subtypes: Three kinds are distinguished: arithmetic across rows, arithmetic across columns, and arithmetic involving a fixed constant.
  • Construction method: The pipeline uses human-computer collaboration, similar in spirit to Overnight but targeting SQL.
    • SQL queries are generated from a grammar.
    • Pseudo questions are produced by traversing each query in execution order.
    • Annotators paraphrase the pseudo questions, and low-confidence pairs are checked manually.
  • Baseline results (test set, accuracy without/with values):
    • IRNet scores 34.2/15.4, while IRNetExt reaches 54.3/50.1. Seq2SeqCopying performs worst, and SyntaxSQLNet was also evaluated.
    • Calculation questions make up 20.7% of the test set, and about 70% of test questions contain values.
    • Calculation questions remain hardest because they often require common-sense knowledge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

19 papers of our library cite it (of the 140673 whose bibliographies we read); 1 of them have a page here, 18 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.562

@inproceedings{wang2020dusql,
  title = {{DuSQL: A Large-Scale and Pragmatic Chinese Text-to-SQL Dataset}},
  author = {Wang, Lijie and Zhang, Ao and Wu, Kun and Sun, Ke and Li, Zhenghua and Wu, Hua and Zhang, Min and Wang, Haifeng},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.562},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.562}
}

References

30 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ali Mohamed Nabil Allam and Mohamed Hassan Hag-gag. 2012. The question answering systems: A survey. International Journal of Research and Reviews in Information Sciences (IJRRIS), 2(3). find
  2. Ben Bogin, Jonathan Berant, and Matt Gardner. 2019. Representing schema structure with graph neural networks for text-to-sql parsing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4560–4565. find
  3. Robin D Burke, Kristian J Hammond, Vladimir Ku-lyukin, Steven L Lytinen, Noriko Tomuro, and Scott Schoenberg. 1997. Question answering from frequently asked question files: Experiences with the faq finder system. AI magazine, 18(2):57–57. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  5. Li Dong and Mirella Lapata. 2016. Language to logical form with neural attention. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 33–43. find
  6. Catherine Finegan-Dollak, Jonathan K Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev. 2018. Improving text-to-sql evaluation methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages… find our card
  7. Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-sql in cross-domain database with intermediate representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4524– 4535. find
  8. Tong Guo and Huilin Gao. 2019. Content enhanced bert-based text-to-sql generation. arXiv preprint arXiv:1910.07179. — [arXiv:1910.07179](https://arxiv.org/abs/1910.07179) find
  9. Pengcheng He, Yi Mao, Kaushik Chakrabarti, and Weizhu Chen. 2019. X-sql: reinforce schema representation with context. arXiv preprint arXiv:1908.08113. — [arXiv:1908.08113](https://arxiv.org/abs/1908.08113) find
  10. Jonathan Herzig and Jonathan Berant. 2019. Dont paraphrase, detect! rapid and effective data collection for semantic parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJC… find
  11. Srinivasan Iyer, Ioannis Konstas, Alvin Cheung, Jayant Krishnamurthy, and Luke Zettlemoyer. 2017. Learning a neural semantic parser from user feedback. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 963–973. find
  12. Cody CT Kwok, Oren Etzioni, and Daniel S Weld. 2001. Scaling question answering to the web. In Proceedings of the 10th international conference on World Wide Web, pages 150–161. find
  13. Fei Li and HV Jagadish. 2014. Constructing an interactive natural language interface for relational databases. Proceedings of the VLDB Endowment, 8(1):73–84. find
  14. Percy Liang. 2013. Lambda dependency-based compositional semantics. arXiv preprint arXiv:1309.4408. find
  15. Wang Ling, Phil Blunsom, Edward Grefenstette, Karl Moritz Hermann, Tomáš Kočiskỳ, Fumin Wang, and Andrew Senior. 2016. Latent predictor networks for code generation. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 599– 609. find
  16. Qingkai Min, Yuefeng Shi, and Yue Zhang. 2019. A pilot study for chinese sql semantic parsing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3643–3649. find
  17. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: a human-generated machine reading comprehension dataset. find our card
  18. Ana-Maria Popescu, Oren Etzioni, and Henry Kautz. 2003. Towards a theory of natural language interfaces to databases. In Proceedings of the 8th international conference on Intelligent user interfaces, pages 149–157. find
  19. Ningyuan Sun, Xuefeng Yang, and Yunfeng Liu. 2020. Tableqa: a large-scale chinese text-to-sql dataset for table-aware sql generation. arXiv preprint arXiv:2006.06434. — [arXiv:2006.06434](https://arxiv.org/abs/2006.06434) find
  20. Yibo Sun, Duyu Tang, Nan Duan, Jianshu Ji, Guihong Cao, Xiaocheng Feng, Bing Qin, Ting Liu, and Ming Zhou. 2018. Semantic parsing with syntax-and table-aware sql generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 361–… find
  21. Lappoon R Tang and Raymond J Mooney. 2001. Using multiple clause constructors in inductive logic programming for semantic parsing. In European Conference on Machine Learning, pages 466–477. Springer. find
  22. Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2019. Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers. arXiv preprint arXiv:1911.04942. find
  23. Yushi Wang, Jonathan Berant, and Percy Liang. 2015. Building a semantic parser overnight. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1332–1342. find
  24. Xiaojun Xu, Chang Liu, and Dawn Song. 2017. Sqlnet: Generating structured queries from natural language without reinforcement learning. arXiv preprint arXiv:1711.04436. find
  25. Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig. 2017. Sqlizer: query synthesis from natural language. Proceedings of the ACM on Programming Languages, 1(OOPSLA):1–26. find
  26. Pengcheng Yin and Graham Neubig. 2017. A syntactic neural model for general-purpose code generation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 440–450. find our card
  27. Tao Yu, Michihiro Yasunaga, Kai Yang, Rui Zhang, Dongxu Wang, Zifan Li, and Dragomir Radev. 2018a. Syntaxsqlnet: Syntax tree networks for complex and cross-domain text-to-sql task. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1653–1663. find
  28. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn-ing Yao, Shanelle Roman, et al. 2018b. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In Proceedings of the 2018 Conference on Empirical… find our card
  29. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2sql: Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103. find
  30. SQLs ::= SQL ::= Select ::= Where GroupC ::= ::= OrderC ::= Dir Conditions ::= ::= Condition A C ::= ::= ::= mathop op ::= ::= SQL intersect SQLs \| SQL union SQLs \| SQL except SQLs \| SQL Select \| Select Where \| Select GroupC \| Select Where GroupC \| Select OrderC \| Select Where OrderC \| Select from S… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/wang-2020-dusql-large-scale-pragmatic-chinese/