📄 Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-Based Chatbots

Summary

The paper addresses multi-turn response selection for retrieval-based chatbots: given a conversation history and a pool of candidate replies, the system must pick the reply that fits the whole dialogue, not just the last message. Earlier models either glue all utterances together, which discards their structure, or compress the context into one vector before comparing it with the reply, which drops details. The authors propose the Sequential Matching Network (SMN), which works in three stages:

• It compares the candidate with every utterance separately, using word-level and GRU-based segment-level similarity matrices.
• A CNN condenses each pair into a matching vector.
• A second GRU combines these vectors in dialogue order to produce a score.

SMN is evaluated on the English Ubuntu dialogue corpus and on a new Chinese, human-labeled Douban Conversation Corpus. It beats strong baselines on both, with a gain of more than 6% in R10@1 on Ubuntu. This shows that matching early, before compressing anything, pays off. The new corpus and the code are publicly released.

Key points

• SMN has three layers:
  • utterance–response matching, with two channels: a word-embedding similarity matrix M1 and a GRU hidden-state similarity matrix M2, both processed by convolution and pooling;
  • matching accumulation with a GRU;
  • matching prediction from the GRU hidden states.
• Results over the best baseline:
  • Ubuntu: more than 6% improvement in R10@1.
  • Douban: about 3% in R10@1 and 4% in P@1.
• Douban Conversation Corpus:
  • open-domain Chinese conversations;
  • candidates come from a retrieval engine and were labeled by three human judges;
  • the authors describe it as the first human-labeled dataset for this task.
• Ablation findings:
  • Swapping the "2D" matching for a neural tensor network causes a sharp drop.
  • Replacing the accumulation GRU with an MLP causes only a slight drop.
  • M2 is somewhat stronger than M1, and using both is best in almost all metrics.
• Context length:
  • Performance rises quickly up to about 5 context turns and levels off around 10, so 10 is recommended as a practical cap.
  • In the Douban test set, only 667 of 1000 contexts had any correct candidate after retrieval.

Interesting ideas

• I1. Match first, aggregate later. Comparing the candidate with each context piece before summarizing keeps the fine-grained signals that early compression discards. This could carry over to multi-document QA or to ranking against long histories.
• I2. Two-channel similarity images. Stacking a word-level matrix and a contextual (RNN-state) matrix as CNN input channels captures matches at several granularities at once. This is a cheap way to enrich any text-pair matching model.
• I3. Order-aware accumulation of pair scores. Feeding per-utterance matching vectors into a GRU lets the gates learn which turns matter. The visualization showed the gates nearly closing for uninformative utterances, which also gives an interpretability signal.
• I4. Realistic evaluation data. Building test candidates from an actual retrieval engine with human labels, instead of random negatives, gives a harder and more realistic benchmark. This is worth copying when building evaluation sets for Ukrainian dialogue or search.
• I5. Checking the retrieval step separately. The error analysis measured how often the first-stage retriever returns no correct candidate at all (333 of 1000 test contexts). This is a useful diagnostic for any retrieve-then-rank pipeline, because it shows how much of the error comes from the retriever rather than the ranker.