Elamur Research⌕
☰
home / papers / 2016 / Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation

Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation

Paper · 2016 · Lili Mou, Yiping Song, Rui Yan et al.

Summary

Neural seq2seq chatbots often produce bland, generic answers. Such replies are frequent in training data, and a short query may give the encoder too little to work with. Mou et al. address this for single-turn short-text conversation by splitting reply generation into two stages. First, a keyword for the reply is selected using pointwise mutual information between query words and candidate reply words. Candidates are limited to nouns, so the choice favours words specific to the query rather than merely common ones. Second, a new decoder, seq2BF, builds the reply around that keyword. One seq2seq network writes the words preceding the keyword in reverse order. A second network then reads the query plus the restored prefix and continues past the keyword to the end. This lets the keyword sit anywhere in the sentence. In human evaluation and an entropy measure, the method beats a plain seq2seq baseline, and the keyword appears in sensible positions. The work shows that explicitly injecting content is a simple and effective counter to generic replies.

Key points

  • Keyword prediction. The query-level PMI with a candidate reply word is approximated as the sum of word-level PMI scores, under independence assumptions. The highest-scoring noun is chosen.
  • seq2BF architecture. It adopts the asyn-B/F variant from the authors' earlier backward/forward language modeling work (Mou et al., 2015), not sep-B/F or syn-B/F. The backward and forward networks use GRU-based RNNs and do not share parameters, nor do encoders and decoders.
  • Training data. No extra labels are needed. A random word in each training reply serves as the split point, and the prefix is reversed to train the backward generator. The forward generator is an ordinary query-to-reply seq2seq model; at inference it is fed the keyword and backward prefix before it continues.
  • Entropy result. With the same seq2BF model, the non-keyword words reach an average character-level entropy of 7.422 when a keyword is supplied. Without keywords the figure is 6.971. This suggests that a concrete starting word pushes generation toward more informative content.
  • Overall comparison. The approach consistently outperforms pure seq2seq on human evaluation and entropy.

Interesting ideas

  • I1. Decode outward from a mandatory token. Generating left in reverse and then right guarantees that a required word actually appears in the output. Soft insertion mechanisms cannot promise this. The same trick could enforce terminology, entity names, or place names in generated text.
  • I2. PMI as a cheap content planner. Dividing by a word's prior probability penalises frequent filler words. Summing word-level PMI gives a fast, training-free way to pick a "topic" word that conditions a heavier generator.
  • I3. Self-supervised split points. Picking a random split word turns ordinary input–output pairs into training data for constrained generation. This avoids any annotation cost when building similar models for new languages or domains.
  • I4. Reuse a standard model via forced prefixes. The forward generator is just a normal seq2seq model whose first outputs are overridden at inference. Existing trained decoders can therefore be repurposed for infix-constrained generation without architectural changes.
  • I5. Diagnostic by holding the model fixed. The authors compare the same model with and without the injected keyword and measure entropy only on the non-keyword words. This isolates whether blandness comes from the decoding start point rather than the model's capacity, and it is a useful pattern for analysing other generation failures.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Peking UniversityBeijing · China100 works · 35,618 citations🏢BaiduBeijing · China40 works · 14,627 citations🏛️Chinese Academy of SciencesBeijing · China68 works · 22,753 citations

Also on the byline: Ministry of Education of the People's Republic of China.

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksArtificial Intelligence · 8,888 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

Cite

Cite as Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang and Zhi Jin. 2016. Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation. Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mou2016sequence,
  title = {{Sequence to Backward and Forward Sequences: A Content-Introducing Approach to Generative Short-Text Conversation}},
  author = {Lili Mou, Yiping Song, Rui Yan, Ge Li, Lu Zhang and Zhi Jin},
  booktitle = {Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers},
  year = {2016},
  url = {https://aclanthology.org/C16-1316/}
}

References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations.
  2. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of 8th Workshop on Syntax, Semtnatics and Structure in Statistical Translation, pages 103–111.
  3. George Ferguson, James Allen, and Brad Miller. 1996. TRAINS-95: Towards a mixed-initiative planning assistant. In Proceedings of Artificial Intelligence Planning Systems Conference, pages 70–77. find
  4. Measuring nominal scale agreement among many raters. Psychological Bulletin, Arthur C Graesser, Patrick Chipman, Brian C Haynes, and Andrew Olney. 2005. AutoTutor: An intelligent tutoring system with mixed-initiative dialogue. IEEE Transactions on Education, 48(4):612–618. find
  5. Sangdo Han, Jeesoo Bang, Seonghan Ryu, and Gary Geunbae Lee. 2015. Exploiting knowledge base to generate responses for natural language dialog listening agents. In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pages 129–133. find
  6. Charles Lee Isbell, Michael Kearns, Dave Kormann, Satinder Singh, and Peter Stone. 2000. Cobot in LambdaMOO: A social statistics agent. In Proceedings of the 17th National Conference on Artificial Intelligence, pages 36–41. find
  7. Andrej Karpathy, Justin Johnson, and Fei-Fei Li. 2015. Visualizing and understanding recurrent networks. arXiv preprint arXiv:1506.02078.
  8. Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a. A diversity-promoting objective function for neural conversation models. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1…
  9. Xiang Li, Lili Mou, Rui Yan, and Ming Zhang. 2016b. StalemateBreaker: A proactive content-introducing approach to automatic human-computer conversation. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, pages 2845–2851. find
  10. Teruhisa Misu and Tatsuya Kawahara. 2007. Speech-based interactive information guidance system using question-answering technique. In Proceedings of the International Conference on Acoustics, Speech and Signal Processing, volume IV, pages 145–148. find
  11. Lili Mou, Rui Yan, Ge Li, Lu Zhang, and Zhi Jin. 2015. Backward and forward language modeling for constrained natural language generation. arXiv preprint arXiv:1512.06612. — [arXiv:1512.06612](https://arxiv.org/abs/1512.06612) find
  12. Hao Peng, Lili Mou, Ge Li, Yunchuan Chen, Yangyang Lu, and Zhi Jin. 2015. A comparative study on regularization strategies for embedding-based neural networks. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 2106–2111. find
  13. Alan Ritter, Colin Cherry, and William B Dolan. 2011. Data-driven response generation in social media. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 583–593.
  14. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 379–389. find
  15. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016a. Building end-to-end dialogue systems using generative hierarchical neural network models. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, pages 3776–3783.
  16. Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016b. A hierarchical latent variable encoder-decoder model for generating dialogues. arXiv preprint arXiv:1605.06069.
  17. Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing, pages 1577–1586.
  18. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. In Proceedings of the Conference of the North American Chapter of…
  19. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems, pages 3104–3112. find
  20. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869.
  21. Hao Wang, Zhengdong Lu, Hang Li, and Enhong Chen. 2013. A dataset for research on short-text conversations. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 935–945. find
  22. Zhe Wang, Wei He, Hua Wu, Haiyang Wu, Wei Li, Haifeng Wang, and Enhong Chen. 2016. Chinese poetry generation with planning based neural network. In Proceedings the 26th International Conference on Computational Linguistics. find
  23. Chen Xing, Wei Wu, Yu Wu, Jie Liu, Yalou Huang, Ming Zhou, and Wei-Ying Ma. 2016. Topic augmented neural response generation with a joint attention mechanism. arXiv preprint arXiv:1606.08340. — [arXiv:1606.08340](https://arxiv.org/abs/1606.08340) find
  24. Kaisheng Yao, Geoffrey Zweig, and Baolin Peng. 2015. Attention with intention for a neural network conversation model. arXiv preprint arXiv:1510.08565. — [arXiv:1510.08565](https://arxiv.org/abs/1510.08565) find
  25. Jun Yin, Xin Jiang, Zhengdong Lu, Lifeng Shang, Hang Li, and Xiaoming Li. 2016. Neural generative question answering. In Proceedings of the 25th International Joint Conference on Artificial Intelligence, pages 2972– 2978. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mou-2016-sequence-backward-forward-sequences-content/