📄 Key-Value Retrieval Networks for Task-Oriented Dialogue

Summary

The paper addresses a problem of task-oriented dialogue agents. Such agents find it hard to make natural use of information from a knowledge base (KB) while remaining fully differentiable. Researchers from Stanford and Ford Research propose the Key-Value Retrieval Network. It is an LSTM-based seq2seq model with attention over the encoder states, extended with a separate attention mechanism over the keys of KB entries. Each database entry is decomposed into (subject, relation, object) triples. The key is the sum of the subject and relation embeddings. The attention score for a key becomes the logit for a canonicalized value token, and during generation this token is replaced with the actual value from the database. The model requires no belief tracker or explicit dialogue state modeling and is trained on all domains at once. The authors also released a new corpus of KB-grounded dialogues for an in-car assistant. On BLEU, entity F1 and human evaluations in live conversations, the model outperforms a rule-based system and Copy Net. The work shows that knowledge from a database can be incorporated into neural dialogue without manual state annotation.

Key points

• Corpus: 3,031 multi-turn dialogues in three in-car assistant domains: calendar scheduling, weather forecasting, and point-of-interest navigation. Each dialogue has its own KB containing at most 230 normalized triples. The data is freely available on the Stanford NLP website.
• Method: the output vocabulary is extended to |V| + n canonical KB tokens. The KB attention scores are added to the ordinary logits as a sparse vector. The idea derives from key-value memory networks (Miller et al., 2016).
• Automatic metrics: the model achieves the highest BLEU and aggregate entity F1. It outperforms the rule-based system by 4.2% in aggregate entity F1, and Copy Net by 2.2 BLEU points and by 11% in entity F1. It falls about 12.7% short of human performance in entity F1, although in BLEU it is roughly on par with humans.
• Ablation: Attn. Seq2Seq without KB attention has a very low entity F1. The variant without encoder attention is already considerably better, and the best result comes from combining both types of attention.
• Human evaluation: the experiment covered 120 live chat scenarios with Amazon Mechanical Turk workers, with ratings on a 1–5 scale. KV Ret. Net scored 3.36 for fluency, 3.43 for cooperativeness and 3.09 for humanlikeness. For comparison: rule-based — 3.20/3.39/2.87, Copy Net — 2.33/2.42/2.02, humans — 3.98/3.95/4.04.