🗺️ Semantic parsing

Concept · 30 key papers · all concepts

Semantic parsing is the task of mapping a natural-language utterance to a formal, machine-interpretable representation of its meaning, such as a logical form, a database query, a meaning graph or a frame structure, which a computer can execute, query or reason over.

Also: natural language to logical form, meaning representation parsing, text-to-SQL (a major subtask), natural language interface to databases (NLIDB), AMR parsing, frame-semantic parsing, shallow semantic parsing / semantic role labeling (in the weaker sense)

What it is

A semantic parser takes a sentence or question and produces a structured object that captures what it means. Syntactic parsing describes grammatical form. A semantic parse instead has a model-theoretic interpretation: it can be executed against a database, checked against a knowledge base, or compared with other meanings. The field spans several kinds of target representation. They differ mainly in how deep the meaning goes and whether the output is meant to be executed.

Target representation	Typical use	Example work
Predicate roles and frames	identifying who did what to whom	(Pradhan et al. 2004; Das et al. 2014)
Lambda-calculus logical forms	questions over small domains such as U.S. geography or ATIS flight booking	(Wong & Mooney 2007; Zettlemoyer & Collins 2009)
Sentence-level meaning graphs	broad-coverage meaning of open text	(Banarescu et al. 2013)
SQL queries	database question answering	(Yu et al. 2018)
Intent–slot trees	voice assistants and task-oriented dialogue	(Gupta et al. 2018)

The shallow end of this spectrum is often called semantic role labeling. It finds the arguments of each verb and labels them with roles such as ARG0–ARG5 from PropBank, an annotation of verbs and their arguments in Wall Street Journal text (Pradhan et al. 2004). Frame-semantic parsing does the same with frames and roles from FrameNet, a lexicon of situation types and their participants (Das et al. 2014). Deep or executable semantic parsing, the more common sense of the term today, produces full logical forms or queries.

The key idea

Most semantic parsers search over candidate meaning representations and pick the best-scoring one. In the classic statistical formulation, a model scores pairs of a derivation and a logical form:

z* = argmax over (d, z) of  θ · φ(x, d, z)
p(d, z | x; θ) = exp(θ · φ(x, d, z)) / Σ over (d′, z′) of exp(θ · φ(x, d′, z′))
y = ⟦z*⟧_K

Here x is the input utterance and z is a candidate meaning representation, such as a lambda term, an SQL query or a graph. The derivation d records how z was built from the words, for example a tree of grammar rules. The feature function φ maps a triple (x, d, z) to a vector of features, and θ is a learned weight vector. The sum in the denominator runs over all (derivation, meaning) pairs that the grammar or decoder allows, which makes p a log-linear (maximum-entropy) distribution. That space is huge, so in practice the argmax is found by approximate search such as beam search (Zettlemoyer & Collins 2009). The last line says that the answer y is obtained by executing z* against a knowledge source K, such as a database.

This scheme appears in several forms. A maximum-entropy model selects among derivations of a synchronous grammar whose meaning side consists of λ-functions (Wong & Mooney 2007). CCG (Combinatory Categorial Grammar) parsers pair words with syntactic categories and lambda meanings. They learn a log-linear parse-selection model while treating derivations as hidden variables (Kwiatkowski et al. 2011). When only final logical forms are annotated, the intermediate steps can be learned as latent variables with a hidden-variable perceptron (Zettlemoyer & Collins 2009).

Compositionality is built into these derivations. In λ-WASP, each synchronous rule pairs a phrase with a λ-function. Lambda abstraction lets logical variables be bound once and shared correctly as rules combine (Wong & Mooney 2007). In CCG parsers, each lexical entry couples a category with a lambda term, and combining categories also combines their meanings (Kwiatkowski et al. 2011). Neural parsers replace hand-designed features φ with learned encoders and generate z token by token. The underlying search for a well-formed, executable structure remains.

History and founding papers

The compositional, model-theoretic view of meaning goes back to Montague (1970), and frame semantics to Fillmore (1976). Statistical semantic role labeling grew from the framework of Gildea and Jurafsky (2002). Pradhan and colleagues replaced its classifier with SVMs. They reached an identification-plus-classification F1 of 87 on hand-corrected PropBank parses and 79 on automatic ones (Pradhan et al. 2004).

For deep parsing, a key mid-2000s line, the WASP parser, borrowed machinery from statistical machine translation (Wong & Mooney 2006; Wong & Mooney 2007). Its extension λ-WASP learned grammars for full logical forms with variables. GeoQuery is a corpus of questions about a U.S. geography database. On it, λ-WASP reached an F-measure of 89.19 under 10-fold cross-validation, with 792 training examples per fold (Wong & Mooney 2007). It also performed similarly on Spanish, Japanese and Turkish versions of a smaller multilingual subset, known as Geo250 (Wong & Mooney 2007; Kwiatkowski et al. 2011).

In parallel, Zettlemoyer and Collins cast the task as structured classification with probabilistic categorial grammars (Zettlemoyer & Collins 2012). They later handled context-dependent dialogues from ATIS, a flight-booking corpus. Their system produced fully correct logical forms for 83.7% of sentences in DEC94, the standard ATIS test set (Zettlemoyer & Collins 2009). Factored CCG lexicons then improved generalization without language-specific rules (Kwiatkowski et al. 2011).

A different path removed supervision entirely. USP clustered lambda forms derived from dependency trees. On a biomedical question-answering evaluation it returned 295 correct answers at 88% accuracy (Poon & Domingos 2009).

The early 2010s shifted toward weaker supervision and larger knowledge bases. Parsers were trained on Freebase, a large structured knowledge base, from question–answer pairs instead of annotated logical forms (Berant et al. 2013). Paraphrasing was used to bridge the gap between language and KB predicates (Berant & Liang 2014). Single-relation questions were handled with convolutional similarity models (Yih et al. 2014). These reached a best F1 of 61 against 54 for PARALEX, an earlier system built on induced lexical rules, on the PARALEX test set. Compositional parsing was also extended to semi-structured web tables (Pasupat & Liang 2015).

Broad-coverage meaning representation took shape with Abstract Meaning Representation (AMR). AMR encodes a sentence as a rooted, directed, edge-labeled graph built on PropBank frames, using roughly 100 relations (Banarescu et al. 2013). AMR parsers soon followed:

JAMR, a discriminative graph-based parser (Flanigan et al. 2014);
a transition-based algorithm (Wang et al. 2015);
a CCG-based parser evaluated with Smatch, an F1 score over the overlap of predicted and gold AMR graph triples. On the test portion of the official LDC release it scored about 66 Smatch F1, against 63.2 for JAMR (Artzi et al. 2015).

Neural sequence-to-sequence models with attention then showed that logical forms could be generated without hand-built grammars (Dong & Lapata 2016).

How it is used today

The most active application is text-to-SQL. Two metrics are common. Exact (set) match compares the components of the predicted query with the gold query. Execution accuracy compares the results of running both queries. Spider set the reference benchmark, with 10,181 questions and 5,693 unique queries over 200 multi-table databases. Its test databases are unseen in training. When Spider appeared, the best tested model, TypeSQL, reached only 9.7% exact-match accuracy on the database split (Yu et al. 2018).

Earlier benchmarks had a known weakness. Question-based splits let most test queries also appear in training, so query-based splits were proposed as a more honest measure of generalization (Finegan-Dollak et al. 2018). On the single-table WikiSQL benchmark, type-aware encoding raised test execution accuracy from 68.0% for the SQLNet baseline to 73.5%, without using database content. A variant that reads table content reached 82.6% (Yu et al. 2018).

Cross-database parsers then focused on schema linking, that is, matching question words to tables and columns. The metrics below differ, so the numbers are not a single progression:

graph neural networks over table–column structure reached 39.4% test accuracy (exact match, the leaderboard metric of the time), roughly double the previous best (Bogin et al. 2019);
relation-aware encoding was used to link question words to schema items (Wang et al. 2020);
BRIDGE serialized the schema for BERT and reached 59.2% test exact match and 59.9% test execution accuracy (Lin et al. 2020);
RESDSQL separated schema linking from skeleton parsing, which produces the SQL keyword structure before the full query (Li et al. 2023).

In 2023 large language models came to dominate this benchmark. DIN-SQL decomposes the task into chained prompts. With GPT-4 it reached 85.3% execution accuracy on the Spider holdout test, although its exact set match was only about 60%. On the harder BIRD benchmark it reached 55.9% test execution accuracy (Pourreza & Rafiei 2023). DAIL-SQL, derived from a systematic study of prompt design, reached 86.6% execution accuracy on Spider (Gao et al. 2023).

Conversational settings extend the problem to multi-turn interaction. In SParC, accuracy collapses in later turns. On the dev set, the SyntaxSQL-con baseline scored 38.6% on first-turn questions but 1.1% from turn 4 onward (Yu et al. 2019). CoSQL is a dialogue corpus in which state tracking means predicting the SQL query that represents the dialogue state at each turn. The baselines reported with it in 2019 scored below 16% question-level accuracy on this task (Yu et al. 2019).

Virtual assistants use hierarchical intent–slot trees. In the TOP data, about 30% of queries need nesting that flat tagging cannot express (Gupta et al. 2018). MTOP extends such annotations to six languages (Li et al. 2021).

Open questions

Schema linking remains a major bottleneck. Even after decomposition it was still the largest category of LLM errors (Pourreza & Rafiei 2023). In one graph-based parser, removing the learned relevance score gave 37.0% dev accuracy, while an oracle relevance score raised it to 54.3%. This shows that much headroom is left (Bogin et al. 2019).

Evaluation is also unsettled. Execution accuracy and exact set match can diverge widely (Pourreza & Rafiei 2023), and benchmark splits can hide memorization (Finegan-Dollak et al. 2018).

Several other problems remain open:

Robustness to synonyms and domain knowledge is not solved. A strong 2023 parser reached 76.9% execution accuracy on the Spider-Syn variant and 66.0% on Spider-DK (Li et al. 2023).
LLM pipelines are costly per query (Pourreza & Rafiei 2023).
Fine-tuning open models can weaken their in-context learning ability (Gao et al. 2023).
As of 2015, broad-coverage AMR parsing trailed human agreement, with about 66 Smatch F1 against inter-annotator agreement near 83 (Artzi et al. 2015).
AMR itself is English-oriented and does not claim to be an interlingua (Banarescu et al. 2013).
Multi-turn context, clarification and unanswerable requests stay largely unsolved (Yu et al. 2019).

A broader question is whether a single general-purpose meaning representation can serve many tasks, or whether executable, task-specific targets will remain the practical norm.

Key papers

The works this page cites, the most cited first; citations as counted by OpenAlex plus the papers of this library that cite the work, read 2026-10-23.

Semantic role labeling — Semantic role labeling is the task of finding, for each predicate in a sentence, the phrases that act as its participants and labeling each…
Context-free grammar — A context-free grammar is a formal system of rewriting rules in which each rule replaces a single nonterminal symbol with a string of…
Knowledge representation and reasoning — Knowledge representation and reasoning is the area of artificial intelligence that studies how facts, concepts and rules about a domain can…
Construction grammar — Construction grammar is a family of linguistic theories holding that knowledge of a language consists of constructions. Constructions are…

More papers

Topics: 🧾 Semantic parsing & meaning representation

Shelves: 💬 Natural language processing

This page in Markdown → https://md.research.elamur.ai/free/en/concepts/semantic-parsing/