Elamur Research
☰
home / papers / 2015 / Neural Responding Machine for Short-Text Conversation

Neural Responding Machine for Short-Text Conversation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P15-1152.pdf

Summary

The paper addresses Short-Text Conversation (STC): producing a single reply to a short user post. Earlier approaches either retrieve existing human replies, which cannot adapt to new posts, or treat replying as statistical machine translation, which assumes the post and reply mean the same thing and often yields stiff or ungrammatical output. The authors propose Neural Responding Machine (NRM), an RNN encoder-decoder with GRU units that encodes the post and then generates the reply word by word. They test three encoder designs:

  • NRM-glo uses the final hidden state as a single global summary.
  • NRM-loc uses an attention mechanism that builds the context dynamically.
  • NRM-hyb combines the global and local designs.

The models were trained on about 4.4 million post–comment pairs collected from Sina Weibo. Human judges rated over 75% of the generated replies as fluent and appropriate. The hybrid model beat both retrieval-based and SMT-based baselines with statistical significance. The work matters because it was an early demonstration that purely neural generation is a viable approach to open-domain dialogue.

Key points

  • The dataset contains 219,905 posts and 4,308,211 responses, forming 4,435,959 pairs. Each post has roughly 20 responses on average, and the data was filtered to remove trivial replies and advertisements.
  • The test set has 110 unseen posts. Five annotators labelled the outputs, and their agreement, measured with Fleiss' kappa, was mostly "fair" (0.2–0.4).
  • On ranking:
    • NRM-hyb ranked best (Friedman test, p < 0.05).
    • NRM-glo and the retrieval baseline performed about the same.
    • NRM-loc's advantage over retrieval was only marginal (p = 0.062).
    • SMT was the weakest system.
  • Beam search with a beam size of 500 produced many distinct yet relevant replies for the same post.
  • BLEU scores fell below 2 for every system and diverged from human judgments, so the authors relied on manual evaluation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

264 papers of our library cite it (of the 140673 whose bibliographies we read); 15 of them have a page here, 249 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-1152 · arXiv 1503.02364

@inproceedings{shang2015neural,
  title = {{Neural Responding Machine for Short-Text Conversation}},
  author = {Shang, Lifeng and Lu, Zhengdong and Li, Hang},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-1152},
  eprint = {1503.02364},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.3115/v1/p15-1152}
}

References

24 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Michael Auli, Michel Galley, Chris Quirk, and Geoffrey Zweig. 2013. Joint language and translation modeling with recurrent neural networks. In EMNLP, pages 1044–1054. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  3. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. find
  4. Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555. find
  5. Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378. find
  6. Alex Graves. 2013. Generating sequences with recurrent neural networks. preprint arXiv:1308.0850. find
  7. Klaus Greff, Rupesh Kumar Srivastava, Jan Koutnı́k, Bas R. Steunebrink, and Jürgen Schmidhuber. 2015. LSTM: A search space odyssey. CoRR, abs/1503.04069. find
  8. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  9. David C. Howell. 2010. Fundamental Statistics for the Behavioral Sciences. PSY 200 (300) Quantitative Methods in Psychology Series. Wadsworth Cengage Learning. find
  10. Zongcheng Ji, Zhengdong Lu, and Hang Li. 2014. An information retrieval approach to short text conversation. arXiv preprint arXiv:1408.6988. — [arXiv:1408.6988](https://arxiv.org/abs/1408.6988) find
  11. Thorsten Joachims. 2006. Training linear svms in linear time. In SIGKDD, pages 217–226. ACM. find
  12. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In EMNLP, pages 1700–1709. find
  13. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th annual meeting of the ACL on interacti… find
  14. Diane Litman, Satinder Singh, Michael Kearns, and Marilyn Walker. 2000. Njfun: a reinforcement learning spoken dialogue system. In Proceedings of the 2000 ANLP/NAACL Workshop on Conversational systems, pages 17–20. ACL. find
  15. Tomas Mikolov, Martin Karafiát, Lukas Burget, Jan Cernockỳ, and Sanjeev Khudanpur. 2010. Recurrent neural network based language model. In IN-TERSPEECH 2010, pages 1045–1048. find
  16. Teruhisa Misu, Kallirroi Georgila, Anton Leuski, and David Traum. 2012. Reinforcement learning of question-answering dialogue policies for virtual museum guides. In SIGDIAL, pages 84–93. ACL. find
  17. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational linguistics, 29(1):19–51. find
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics. find our card
  19. Alan Ritter, Colin Cherry, and William B Dolan. 2011. Data-driven response generation in social media. In EMNLP, pages 583–593. Association for Computational Linguistics. find
  20. Jost Schatzmann, Karl Weilhammer, Matt Stuttle, and Steve Young. 2006. A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies. The knowledge engineering review, 21(02):97–126. find
  21. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Meg Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. Conference of the North American Chapter of the Association for Computa… find our card
  22. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In NIPS, pages 3104–3112. find
  23. Hao Wang, Zhengdong Lu, Hang Li, and Enhong Chen. 2013. A dataset for research on short-text conversations. In EMNLP, pages 935–945. find
  24. Jason D Williams and Steve Young. 2007. Partially observable markov decision processes for spoken dialog systems. Computer Speech & Language, 21(2):393–422. find