SEQZERO: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models
Paper · 2022 · Jingfeng Yang, Haoming Jiang, Qingyu Yin et al.
Summary
The paper tackles semantic parsing (mapping questions to formal queries such as SQL) when only a handful of annotated pairs exist. Earlier few-shot approaches fine-tune pretrained LMs to produce a canonical utterance that a grammar then converts into the query. Because these utterances are long and compositional, they need a lot of training data. Small fine-tuning sets also cause overfitting to spurious patterns and poor compositional out-of-distribution generalization.
SEQZERO breaks a query into its sub-clauses (FROM, SELECT, WHERE and so on) and predicts them one at a time. Each step fills a short slot in a natural-language prompt, with earlier clauses supplied as context. Every slot is scored by two models:
- a fine-tuned BART;
- an untuned (zero-shot) BART restricted to a set of candidate values.
The zero-shot model assigns very little probability to those candidates, so the authors rescale its constrained probabilities before a linear ensemble. On the compositional splits of GeoQuery and the newly released EcommerceQuery, the method reaches state-of-the-art results among BART-based parsers. This suggests that combining zero-shot and few-shot models can reduce bias-driven errors.
Key points
-
Directly fine-tuned BART on GeoQuery tends to latch onto spurious cues, for example predicting the wrong table for familiar question patterns. The authors attribute about 75% of the Base model's test errors to this kind of compositional OOD mistake.
-
The zero-shot model's candidate values come from:
- SQL grammar;
- table schema;
- the input utterance;
- training data.
A trie (prefix tree) determines which tokens are allowed at each decoding step.
-
Constrained probability rescaling is what lets the zero-shot model actually contribute to the ensemble. Weight-space ensembling in the style of Wortsman et al. (2021) did not work in this generation setting.
-
The paper contributes a new E-commerce product-search dataset, EcommerceQuery. Code and data are announced at github.com/amzn/SeqZero.
-
The work was published in Findings of NAACL 2022 by authors from Amazon and Georgia Tech.
Institutions
United StatesFields, topics and concepts
📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works
🧭 Topics 1Research subjects and languages the work is about.Semantic parsing & meaning representation · 185 works
Cite
Cite as Jingfeng Yang, Haoming Jiang, Qingyu Yin, Danqing Zhang, Bing Yin and Diyi Yang. 2022. SEQZERO: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models. Findings of the Association for Computational Linguistics: NAACL 2022.
@inproceedings{yang2022seqzero,
title = {{SEQZERO: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models}},
author = {Jingfeng Yang, Haoming Jiang, Qingyu Yin, Danqing Zhang, Bing Yin and Diyi Yang},
booktitle = {Findings of the Association for Computational Linguistics: NAACL 2022},
year = {2022},
doi = {10.18653/v1/2022.findings-naacl.5},
url = {https://doi.org/10.18653/v1/2022.findings-naacl.5}
}References
- Ekin Akyürek, Afra Feyza Akyürek, and Jacob Andreas. 2020. Learning to recombine and resample data for compositional generalization. arXiv preprint arXiv:2010.03706. \#\# Conclusion — [arXiv:2010.03706](https://arxiv.org/abs/2010.03706) find
Updated
This page in Markdown → https://md.research.elamur.ai/free/en/papers/yang-2022-seqzero-few-shot-compositional-semantic/