📄 Language to Logical Form with Neural Attention

Summary

The paper addresses semantic parsing: a natural language sentence must be converted into a formal meaning representation that a machine can execute, such as a logical form or a query. Previously, such systems typically relied on hand-crafted lexicons, templates and features, which made them difficult to transfer to another domain or another formalism. The authors propose a general neural approach based on an encoder-decoder architecture with LSTM. The encoder converts the sentence into vectors, and the decoder, conditioned on them, generates the logical form. The model comes in two variants: SEQ2SEQ outputs the logical form as an ordinary sequence of tokens, while SEQ2TREE builds it hierarchically, as a tree. An attention mechanism learns to softly align sentence words with predicates, and a separate argument identification step handles rare entities and numbers. On four corpora, the model performs on par with or better than the best systems of its time, even though it uses no domain- or formalism-specific features. It can therefore be easily adapted to new tasks.

Key points

• The model was evaluated on JOBS (Prolog-style logical forms), GEO and ATIS (lambda calculus), and IFTTT. In all cases the architecture was left unchanged.
• SEQ2TREE ranks second on JOBS and shares first place with ZC07 on ATIS. On IFTTT, both variants outperform posclass and the baseline methods of Quirk et al. (2015).
• The SEQ2TREE decoder marks subtrees with a special token <n>. The hidden state of the parent node is fed as input to the LSTM when generating the child tokens (parent-feeding).
• Training maximizes likelihood using RMSProp, and dropout between LSTM layers curbs overfitting on small corpora.
• The main weakness identified is under-mapping: attention does not take the history of alignments into account, so in long questions the model sometimes skips words. The authors propose explicitly modeling coverage.