Elamur Research⌕
☰
home / papers / 2020 / PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable

PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable

Summary

PLATO addresses why general-purpose pre-trained models like BERT transfer poorly to open-domain dialogue generation. The authors identify three causes: conversational language differs from general text, generation runs left to right while BERT encodes in both directions, and a single context can admit many valid replies. Their framework starts from BERT weights and continues pre-training on large Reddit and Twitter conversation data. Following UniLM, it uses self-attention masks so one transformer can encode bi-directionally and decode uni-directionally. A K-way categorical latent variable, interpreted as an unsupervised "latent speech act," captures reply diversity. Two complementary tasks share the same parameters: response generation conditioned on context and latent value, and latent act recognition, which infers that value from a context–response pair. Training combines NLL, bag-of-words, and response selection losses. The model was evaluated on chit-chat, knowledge-grounded dialogue, and conversational QA, including Persona-Chat and DSTC7-AVSD, and beat prior state-of-the-art systems. The code and models are public, offering a reusable foundation for diverse, coherent dialogue systems.

Key points

  • Each input token's representation sums four embeddings: token, role, turn, and position. Turn indices are relative, so the response is always turn 0. Special [EOU] and [BOU] tokens mark utterance boundaries.
  • The latent act recognition and response selection objectives read the final hidden state of a special [M] mask token. Response selection scores also rank multiple candidate replies at inference time.
  • The paper reports a new best result on DSTC7-AVSD. A reported upper bound, obtained by picking the best of K generated candidates, suggests further headroom.
  • Persona-Chat ablations used 12-layer models trained on 1k, 5k, and 9k dialogues and compared them by perplexity. Flexible attention outperformed directly fine-tuning BERT or GPT-2. Further pre-training on Twitter/Reddit helped, and the latent variable improved results again.
  • Code and pre-trained models are released at github.com/PaddlePaddle/Research (NLP/Dialogue-PLATO).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Dialogue systems

Institutions

🏢BaiduBeijing · China26 works · 11 874 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.9

@inproceedings{bao2020plato,
  title = {{PLATO: Pre-trained Dialogue Generation Model with Discrete Latent Variable}},
  author = {Bao, Siqi and He, Huang and Wang, Fan and Wu, Hua and Wang, Haifeng},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.9},
  url = {https://doi.org/10.18653/v1/2020.acl-main.9}
}

References

5 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. find our card
  2. Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, et al. 2019. Audio visual scene-aware dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7558–7567. find
  3. Siqi Bao, Huang He, Fan Wang, Rongzhong Lian, and Hua Wu. 2019. Know more about each other: Evolving dialogue strategy via compound assessment. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5382– 5391. find
  4. Boxing Chen and Colin Cherry. 2014. A systematic comparison of smoothing techniques for sentence-level bleu. In Proceedings of the 9th Workshop on Statistical Machine Translation, pages 362–367. find our card
  5. Chaotao Chen, Jinhua Peng, Fan Wang, Jun Xu, and Hua Wu. 2019. Generating multiple diverse responses with multi-mapping and posterior mapping selection. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 4918–4924. \#\# Conclusion find

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bao-2020-plato-pre-trained-dialogue-generation/