Elamur Research
☰
home / papers / 2019 / The Second Conversational Intelligence Challenge (ConvAI2)

The Second Conversational Intelligence Challenge (ConvAI2)

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1902.00098

Summary

ConvAI2 was a NeurIPS 2018 competition designed to give open-domain chit-chat research a common benchmark and evaluation protocol, which the field had lacked. The organizers built it on Persona-Chat. In this dataset, crowdworkers who are assigned short persona profiles get acquainted through conversation. They added a newly crowdsourced hidden test set and rephrased persona versions so that models could not rely on simple word overlap. Baselines, training and evaluation code were released in ParlAI. Submissions were scored first with automatic metrics (perplexity, F1, hits@1). The strongest teams then went to human judging on Mechanical Turk and to unstructured chats with volunteers. Pretrained Transformer models performed best overall. Hugging Face led the automatic tracks, but Lost in Conversation won the human-judged grand prize. The analysis shows that automatic scores only partly track human judgments. They overlook multi-turn problems such as repetition, persona contradictions and an unbalanced mix of questions and answers. This motivates new conversation-level metrics, for example ones based on dialogue NLI.

Key points

  • Data splits (ConvAI2, based on Persona-Chat):
    • Training: 131,438 examples, 17,878 dialogues, 1,155 personas.
    • Validation: 7,801 examples, 1,000 dialogues, 100 personas.
    • Hidden test: 6,634 examples, 1,015 dialogues, 100 unseen personas.
  • Automatic metrics: perplexity (generative models only), word-level F1, and hits@1/20, which means picking the gold reply among 19 random distractors.
  • Human evaluation: the top 7 teams were evaluated on Mechanical Turk, where raters judged enjoyment and identified the bot's persona. Volunteers also chatted with the bots via Facebook Messenger and Telegram.
  • Results: 23 entrants took part. Hugging Face won the automatic tracks, while Lost in Conversation took the grand prize, with Hugging Face placing second.
  • Comparison with ConvAI 2017: the earlier edition was built around Wikipedia snippets and provided no initial training data. Ten systems took part, and the best bot scored 2.746 against 3.8 for humans.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

177 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 160 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1902.00098 · arXiv 1902.00098

@misc{dinan2019second,
  title = {{The Second Conversational Intelligence Challenge (ConvAI2)}},
  author = {Dinan, Emily and Logacheva, Varvara and Malykh, Valentin and Miller, Alexander and Shuster, Kurt and Urbanek, Jack and Kiela, Douwe and Szlam, Arthur and Serban, Iulian and Lowe, Ryan and Prabhumoye, Shrimai and Black, Alan W and Rudnicky, Alexander and Williams, Jason and Pineau, Joelle and Burtsev, Mikhail and Weston, Jason},
  year = {2019},
  doi = {10.48550/arxiv.1902.00098},
  eprint = {1902.00098},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1902.00098}
}

References

13 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: I have a dog, do you have pets too? arXiv preprint arXiv:1801.07243, 2018. find our card
  2. Iulian Vlad Serban, Ryan Lowe, Laurent Charlin, and Joelle Pineau. Generative deep neural networks for dialogue: A short review. arXiv preprint arXiv:1611.06216, 2016. — [arXiv:1611.06216](https://arxiv.org/abs/1611.06216) find
  3. Oriol Vinyals and Quoc Le. arXiv:1506.05869, 2015. A neural conversational model. http://workshop.colips.org/dstc7/arXiv preprint ↗ our card
  4. Jiwei Li, Michel Galley, Chris Brockett, Georgios P Spithourakis, Jianfeng Gao, and Bill Dolan. A persona-based neural conversation model. arXiv preprint arXiv:1603.06155, 2016. find our card
  5. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. A diversity-promoting objective function for neural conversation models. arXiv preprint arXiv:1510.03055, 2015. find our card
  6. Alexander H Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. Parlai: A dialog research software platform. arXiv preprint arXiv:1705.06476, 2017. find our card
  7. Chia-Wei Liu, Ryan Lowe, Iulian Vlad Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. CoRR, abs/1603.08023, 2016. find
  8. Ilya Kulikov, Alexander H. Miller, Kyunghyun Cho, and Jason Weston. Importance of a search strategy in neural dialogue modelling. CoRR, abs/1811.00907, 2018. — [arXiv:1811.00907](https://arxiv.org/abs/1811.00907) find
  9. Yu Wu, Wei Wu, Chen Xing, Ming Zhou, and Zhoujun Li. Sequential matching network: A new architecture for multi-turn response selection in retrieval-based chatbots. arXiv preprint arXiv:1612.01627, 2016. find
  10. Jason Weston, Emily Dinan, and Alexander H Miller. Retrieve and refine: Improved sequence generation models for dialogue. arXiv preprint arXiv:1808.04776, 2018. find
  11. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. find
  12. Sean Welleck, Jason Weston, Arthur Szlam, and Kyunghyun Cho. Dialogue natural language inference. arXiv preprint arXiv:1811.00671, 2018. find our card
  13. Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of wikipedia: Knowledge-powered conversational agents. arXiv preprint arXiv:1811.01241, 2018. find