A Dataset for Document Grounded Conversations

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1076.pdf

Summary

Most open conversational datasets either lack an external knowledge source or contain only short exchanges on a single topic. Zhou, Prabhumoye and Black address this by building CMU DoG (CMU Document Grounded Conversations), a corpus of chats between people that revolve around a given text.

They condensed Wikipedia articles on 30 popular movies into four-part documents: basic facts and reception, plus three key plot scenes. Pairs of Amazon Mechanical Turk workers then chatted about each film. Sections were revealed gradually as the dialogue progressed, so every turn is linked to a document section. In one setup only one partner saw the text and tried to persuade the other. In the other, both read it and shared opinions. Conversation quality was graded automatically by BLEU overlap with the document.

As baselines, the authors compared a sequence-to-sequence generator with and without access to the current section. Adding the document lowered perplexity and made responses more engaging and fluent in human judgments. This shows that grounding text helps response generation and gives the field a benchmark for knowledge-aware dialogue.

Key points

• CMU DoG has 4112 conversations averaging 21.43 turns: 2128 where one worker had the document and 1984 where both did.
• Each conversation is graded into three quality ratings by BLEU against its source document:
  • rating 1 means BLEU ≤ 0.1;
  • rating 3 requires more than 12 turns and BLEU above 0.587.
• Both baselines use a two-layer bidirectional LSTM encoder and an LSTM decoder with global attention and a copy mechanism:
  • SEQ conditions on the current utterance only;
  • SEQS additionally encodes the matching document section.
• Trigram language-model perplexity was 21.8 for SEQ and 10.11 for SEQS, an improvement of 11.69 points.
• Human raters preferred SEQS responses for engagement 43.9% of the time versus 36.4% for SEQ, with no preference in 19.6% of cases. SEQS also gained +0.96 in fluency MOS.