Elamur Research
☰
home / papers / 2016 / Incorporating Copying Mechanism in Sequence-to-Sequence Learning

Incorporating Copying Mechanism in Sequence-to-Sequence Learning

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1603.06393

Summary

Standard encoder-decoder networks, even with attention, often fail when an output has to repeat parts of the input exactly. Names, dates and rare words are typical cases, and they are often out-of-vocabulary (OOV). Gu, Lu, Li and Li propose COPYNET to handle this. It is an attention-based RNN encoder-decoder whose decoder can either generate a word from a fixed vocabulary or copy a token from the source. The two options share one softmax normalization, so they compete directly for each output position. A "selective read" passes the location of the previously copied word back into the decoder state, which helps the model copy several consecutive words. The whole model is differentiable and is trained end to end with gradient descent. The authors test it on synthetic sequence-transformation data, Chinese text summarization (LCSTS) and Chinese single-turn dialogue. COPYNET clearly beats an RNNSearch baseline, especially when the input contains words never seen in training. The work matters because it gives Seq2Seq systems a learned, built-in way to deal with OOV words and verbatim reuse of input. It was an early influence on pointer-generator style models.

Key points

  • Output distribution: each output word's probability adds a generate-mode score and a copy-mode score. A word can be produced by both modes, by copying only (OOV source tokens) or by neither (UNK). For each input, the effective vocabulary is V ∪ UNK ∪ the source words.
  • Copy scores: these come from a tanh projection that compares the decoder state with the encoder's bidirectional hidden states. Those states carry both what a word means and where it sits in the source.
  • Dialogue results: on the template-based dialogue sets (DS-I and DS-II), top-1 accuracy went from 44.1 to 61.2 on DS-I. On DS-II, where the filled-in entities never appear in training, it went from 13.5 to 50.5, and top-10 accuracy went from 15.9 to 64.8.
  • Example behaviour: in the case studies, the baseline replaces unseen entities with other names (it outputs Hainan instead of Shanghai). COPYNET copies the right entity and generates the rest of the reply normally.
  • Related work and limits: unlike Pointer Networks, COPYNET is not limited to copying from the input. It cannot be used directly for machine translation because it copies source words exactly, and the authors leave that extension for future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

449 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 429 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1603.06393 · arXiv 1603.06393

@misc{gu2016incorporating,
  title = {{Incorporating Copying Mechanism in Sequence-to-Sequence Learning}},
  author = {Gu, Jiatao and Lu, Zhengdong and Li, Hang and Li, Victor O. K.},
  year = {2016},
  doi = {10.48550/arxiv.1603.06393},
  eprint = {1603.06393},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1603.06393}
}

References

22 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. find our card
  2. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. arXiv preprint arXiv:1603.07252. find our card
  3. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078. find
  4. Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. arXiv preprint arXiv:1410.5401. find
  5. Caglar Gulcehre, Sungjin Ahn, Ramesh Nallapati, Bowen Zhou, and Yoshua Bengio. 2016. Pointing the unknown words. arXiv preprint arXiv:1603.08148. find
  6. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385. find
  7. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  8. Baotian Hu, Qingcai Chen, and Fangze Zhu. 2015. Lcsts: a large scale chinese short text summarization dataset. arXiv preprint arXiv:1506.05865. find
  9. Karol Kurach, Marcin Andrychowicz, and Ilya Sutskever. 2015. Neural random-access machines. arXiv preprint arXiv:1511.06392. find
  10. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Stan Szpakowicz Marie-Francine Moens, editor, Text Summarization Branches Out: Proceedings of the ACL-04 Workshop, pages 74–81, Barcelona, Spain, July. Association for Computational Linguistics. find
  11. Thang Luong, Ilya Sutskever, Quoc Le, Oriol Vinyals, and Wojciech Zaremba. 2015. Addressing the rare word problem in neural machine translation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language P… find our card
  12. Geoffrey J McLachlan and Kaye E Basford. 1988. Mixture models. inference and applications to clustering. find
  13. Statistics: Textbooks and Monographs, New York: Dekker, 1988, 1. find
  14. Alexander M Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. arXiv preprint arXiv:1509.00685. find
  15. Lifeng Shang, Zhengdong Lu, and Hang Li. 2015. Neural responding machine for short-text conversation. arXiv preprint arXiv:1503.02364. find our card
  16. Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2015. Minimum risk training for neural machine translation. CoRR, abs/1512.02433. find our card
  17. Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A neural network approach to context-sensitive generation of conversational responses. arXiv preprint arXiv:1506.06714. find our card
  18. Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway networks. arXiv preprint arXiv:1505.00387. find our card
  19. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  20. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869. find our card
  21. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015a. Pointer networks. In Advances in Neural Information Processing Systems, pages 2674–2682. find
  22. Oriol Vinyals, Łukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey Hinton. 2015b. Grammar as a foreign language. In Advances in Neural Information Processing Systems, pages 2755–2763. find