📄 Task-Oriented Dialogue as Dataflow Synthesis

Summary

Task-oriented dialogue systems often force a choice between rigid slot-filling states, which are easy to learn but limited, and opaque neural states, which are flexible but hard to train. Andreas et al. (Microsoft Semantic Machines) propose a middle path. The dialogue state is a dataflow graph that grows over the conversation. At each turn, the agent converts the user's utterance into an executable program that adds nodes to this graph.

Two metacomputation operators handle the context-dependent parts of conversation:

• refer retrieves a salient earlier node, covering phenomena such as anaphora.
• revise splices a modified constraint into an earlier computation, covering corrections and follow-ups.

Errors, such as an ambiguous attendee, are represented as exceptions that later turns can repair.

The authors release SMCalFlow, a large English corpus with program annotations for every turn. On this corpus, the explicit reference machinery noticeably reduces the errors of a standard seq2seq-with-copying model. When MultiWOZ is converted into dataflow form, a generic seq2seq model performs on par with the best task-specific state tracker.

The work matters because it shows that compositional, cross-domain user goals can be represented and also learned well.

Key points

• SMCalFlow corpus: about 40K English dialogues covering calendar events, weather, places and people. The authors describe it as the largest goal-oriented dialogue dataset at the time. The dataset, code and a public leaderboard are released.
• Effect of explicit reference: on SMCalFlow, adding explicit reference operators cuts the mistakes made by a sequence-to-sequence model with a copy mechanism (See et al., 2017) by 5.9% across all turns. On turns that refer back to earlier turns, the reduction is 10.9%.
• MultiWOZ results: the authors provide scripts that automatically convert MultiWOZ into dataflow programs. With this representation, an off-the-shelf seq2seq model matches the strongest existing task-specific state tracking model.
• Core operators: refer and revise reuse fragments of the graph non-destructively. Exceptions such as UnderconstrainedException block evaluation until a later turn fixes them.
• Error analysis: a manual review of 100 mispredicted SMCalFlow turns found four main error types: underprediction, entity linking, fencing (out-of-scope detection) and ambiguity.