📄 Few-shot Natural Language Generation for Task-Oriented Dialog

Summary

Task-oriented dialog systems need an NLG component that turns a dialog act, meaning an intent plus slot-value pairs, into a fluent reply. Template systems sound rigid, and statistical models such as SC-LSTM need large annotated sets for each domain, which new domains rarely have. To study this low-resource situation, the authors build FewShotWOZ, a benchmark derived from MultiWOZ and the Cambridge NLG data. It covers 7 domains, each with no more than 50 training utterances. They also propose SC-GPT, a GPT-2-based Transformer trained in three stages:

• generic language-model pre-training on plain text;
• further pre-training on roughly 400k utterances labeled with dialog acts;
• fine-tuning on a few dozen in-domain examples.

Dialog acts are written as plain token sequences rather than one-hot vectors, so new intents and slots can be handled without changing the model. SC-GPT beats SC-LSTM, HDSA and plain GPT-2 on FewShotWOZ and sets a new best result on MultiWOZ. This shows that pre-trained language models can make dialog NLG practical for domains with little labeled data.

Key points

• FewShotWOZ benchmark: It spans 7 domains (Restaurant, Hotel, Laptop, TV, Attraction, Taxi, Train), with 50 training instances per domain and 40 for Taxi. Only 8.82% of its delexicalised test dialog acts also occur in training, compared with 94–100% in E2E NLG, BAGEL and RNNLG.
• Dialog-act pre-training data: The second training stage draws on the Schema-Guided Dialog corpus, MultiWOZ, Frames and the Facebook Multilingual Dialog Corpus. Domains used later for fine-tuning are left out.
• MultiWOZ result: SC-GPT improves on the previous state of the art by about 4 BLEU points.
• Unseen dialog acts (restaurant domain): SC-GPT reaches 36.69 BLEU with a slot error rate of 4.96. SC-LSTM scores 12.83 BLEU with 51.98 ERR, and GPT-2 scores 27.92 BLEU with 17.36 ERR.
• New-domain demo: After fine-tuning on only 16 RASA restaurant examples, SC-GPT handled edited dialog acts better than GPT-2. The edits included added slots, removed slots and replaced slot values.