Elamur Research⌕
☰
home / papers / 2020 / Grounded Adaptation for Zero-shot Executable Semantic Parsing

Grounded Adaptation for Zero-shot Executable Semantic Parsing

Paper · 2020 · Victor Zhong, Mike Lewis, Sida I. Wang et al.

Summary

Executable semantic parsers, such as text-to-SQL systems, tend to fail when deployed on a database they never encountered during training, and labelled examples for each new schema are rarely available. The authors present GAZP, a model-agnostic method that adapts a parser to such unseen environments without manual annotation. Two models are first trained on the original data. The first is a forward parser that uses BERT to cross-encode the question with each table schema, then applies BiLSTMs and pointer decoders for query templates and values. The second is a backward generator that turns SQL into questions. In the target database, SQL queries are sampled from coarse templates weighted by training-set frequency, converted into questions, and parsed again. Only pairs whose re-parse agrees with the sampled query, by execution result or exact match, are kept. Retraining on the original data plus the filtered synthetic data improves both logical form and execution accuracy on the zero-shot Spider, Sparc and CoSQL benchmarks. The results suggest that verified synthetic data built in the deployment environment is a cheap and effective route to domain transfer.

Key points

  • Evaluated on three language-to-SQL zero-shot tasks with unseen test databases: Spider, Sparc and CoSQL. GAZP improves the baseline parser's logical form and execution accuracy on all three.
  • Query sampling abstracts training queries into coarse templates with typed column and value slots and removes JOINs, which are later restored deterministically. Queries that are invalid or return an empty result are discarded.
  • The cycle-consistency filter discards roughly 60% of the synthesized examples. Keeping all of them ("nocycle") clearly hurts performance. String-match ("EM") consistency performs close to execution-based consistency but usually gives lower execution accuracy.
  • Adaptation quality grows with the amount of synthesized data, with gains levelling off after about 30–40k examples.
  • GAZP beats data augmentation carried out in the training environment. It can be applied to any parser for which a backward generator can be trained and logical-form equivalence can be checked.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States263 works · 180,581 citations🏢MetaMenlo Park · United States158 works · 211,459 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Semantic parsing & meaning representation · 135 works

Cite

Cite as Victor Zhong, Mike Lewis, Sida I. Wang and Luke Zettlemoyer. 2020. Grounded Adaptation for Zero-shot Executable Semantic Parsing. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.558

@inproceedings{zhong2020grounded,
  title = {{Grounded Adaptation for Zero-shot Executable Semantic Parsing}},
  author = {Victor Zhong, Mike Lewis, Sida I. Wang and Luke Zettlemoyer},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.558},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.558}
}

References

26 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. ICLR.
  2. Ruisheng Cao, Su Zhu, Chen Liu, Jieyu Li, and Kai Yu. 2019. Semantic parsing with dual learning. In ACL. find
  3. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL.
  4. Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. 2017. Learning to paraphrase for question answering. In EMNLP.
  5. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. In EMNLP. find
  6. Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. 2018. Speaker-follower models for vision-and-language navigation. In NeurIPS. find
  7. Arthur C Graesser, Patrick Chipman, Brian C Haynes, and Andrew Olney. 2005. AutoTutor: An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions on Education. find
  8. Jiaqi Guo, Zecheng Zhan, Yan Gao, Yan Xiao, Jian-Guang Lou, Ting Liu, and Dongmei Zhang. 2019. Towards complex text-to-SQL in cross-domain database with intermediate representation. In ACL. find
  9. Awni Y. Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Greg Diamos, Erich Elsen, Ryan Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and Andrew Y. Ng. 2014. Deep Speech: Scaling up end-to-end speech recognition. CoRR, abs/1412.5567.
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation. find
  11. Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isola, Kate Saenko, Alexei A. Efros, and Trevor Darrell. 2018. CyCADA: Cycle consistent adversarial domain adaptation. In ICML. find
  12. Cynthia Matuszek, Evan Herbst, Luke Zettlemoyer, and Dieter Fox. 2013. Learning to parse natural language commands to a robot control system. Experimental Robotics. find
  13. Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. In ACL.
  14. Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2020. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. In NeurIPS Workshop on Energy Efficient Machine Learning and Cognitive Computing. find
  15. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. In ACL. find
  16. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In NIPS. find
  17. Wen-tau Yih, Xiaodong He, and Christopher Meek. 2014. Semantic parsing for single-relation question answering. In ACL.
  18. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018a. Qanet: Combining local convolution with global self-attention for reading comprehension. In ICLR.
  19. Tao Yu, Rui Zhang, He Yang Er, Suyi Li, Eric Xue, Bo Pang, Xi Victoria Lin, Yi Chern Tan, Tianze Shi, Zihan Li, Youxuan Jiang, Michihiro Yasunaga, Sungrok Shim, Tao Chen, Alexander Fabbri, Zifan Li, Luyao Chen, Yuwen Zhang, Shreya Dixit, Vincent Zhang, Caiming Xiong, Richard Socher, Walter Lasecki,…
  20. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018b. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. In EMNLP.
  21. Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019b. Sparc: Cross-domain semantic parsing in…
  22. John M. Zelle and Raymond J. Mooney. 1996. Learning to parse database queries using inductive logic programming. In AAAI/IAAI.
  23. Luke S. Zettlemoyer and Michael Collins. 2005. Learning to map sentences to logical form: Structured classification with probabilistic categorial grammars. In UAI.
  24. Rui Zhang, Tao Yu, He Yang Er, Sungrok Shim, Eric Xue, Xi Victoria Lin, Tianze Shi, Caiming Xiong, Richard Socher, and Dragomir Radev. 2019. Editing-based sql query generation for cross-domain context-dependent questions. In EMNLP.
  25. Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.00103. find
  26. Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In ICCV. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhong-2020-grounded-adaptation-zero-shot-executable/