Elamur Research⌕
☰
home / papers / 2016 / Pointing the Unknown Words

Pointing the Unknown Words

Summary

Neural sequence models usually generate words from a fixed shortlist of frequent tokens, so rare items such as named entities are either poorly learned or collapsed into UNK. This paper introduces the Pointer Softmax (PS), an extension of attention-based encoder–decoder models with two output distributions. The first is the usual softmax over the shortlist. The second is a location softmax, built from the attention weights, that selects a position in the source sequence so its word can be copied. At each decoding step, a small switching MLP looks at the context and the decoder state and decides which distribution to use. The whole model is trained end-to-end with one likelihood objective.

The authors test PS on two tasks: Europarl English–French translation and Gigaword summarization. On Europarl, BLEU rises from 20.19 to 23.76, and training converges faster. On Gigaword, PS still helps when combined with the large-vocabulary trick. The approach matters because it lets a model output words it has never seen in training, as long as they appear in the input, without enlarging the vocabulary.

Key points

  • Architecture:
    • The baseline is the attention NMT of Bahdanau et al., with a bidirectional RNN encoder and a GRU decoder.
    • PS changes only the final output layer, replacing it with a shortlist softmax, a location softmax and a switching network.
    • If the target word appears in neither the shortlist nor the source sentence, the switch is meant to pick the shortlist, which then emits UNK.
  • Europarl setup:
    • Both shortlists contain 30,000 tokens, one of which is reserved for UNK.
    • The corpus has 134,831 unique English words and 153,083 unique French words.
    • About 49,490 words are shared between the two languages.
    • A French-to-English word dictionary covers 15,953 words that fall outside the shortlist and the common-word list.
  • Translation results:
    • BLEU-4 improves from 20.19 to 23.76, roughly +3.6.
    • Validation NLL reaches 63.91 after 200k updates with PS, versus 65.26 after 400k updates for the baseline.
  • Switching network design:
    • A two-layer MLP with noisy-tanh activations and a residual connection gave about +1 BLEU compared with a ReLU version.
    • The bias of the final sigmoid was initialized to −1 so the model leans toward the shortlist early in training.
    • Gradients were rescaled whenever their norm exceeded 1.
  • Related work: Pointer networks are a special case of PS in which the model always points. Luong et al.'s Positional All model is limited by its fixed relative pointing window.

Interesting ideas

  • I1.

    Learned gate between generating and copying. A per-step switch decides whether to produce a vocabulary word or point to a source position. This suits any task where outputs partly overlap with inputs, such as entity-heavy QA, data-to-text, or Ukrainian geographic and trade texts full of proper names.

  • I2.

    Reusing attention as a pointer. The existing attention weights serve directly as a distribution over source positions, so copying costs almost no extra parameters. The pointer also scales naturally to inputs of any length, unlike fixed relative-offset windows.

  • I3.

    Heuristic copy supervision. Training targets for the pointer come from a cascade of checks:

    • exact match of the word in the source;
    • a shared-word lookup between source and target languages;
    • a bilingual dictionary;
    • as a last resort, the position with the highest attention weight.

    This shows how cheap lexical resources can supply supervision for latent alignment decisions.

  • I4.

    Pointer targets as attention hints. The authors attribute PS's faster convergence to the copy targets acting as extra guidance for the attention. Auxiliary supervision of attention may therefore speed up training in other encoder–decoder setups.

  • I5.

    Biasing the gate at initialization. Setting the gate's bias so the model starts out favoring the default path, the shortlist, is a simple trick worth reusing in other mixture or gating models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

164 papers of our library cite it (of the 140884 whose bibliographies we read); 14 of them have a page here, 150 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1014

@inproceedings{gulcehre2016pointing,
  title = {{Pointing the Unknown Words}},
  author = {Gulcehre, Caglar and Ahn, Sungjin and Nallapati, Ramesh and Zhou, Bowen and Bengio, Yoshua},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1014},
  url = {https://doi.org/10.18653/v1/p16-1014}
}

References

27 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473. find our card
  2. Yoshua Bengio and Jean-Sébastien Senécal. 2008. Adaptive importance sampling to accelerate training of a neural probabilistic language model. Neural Networks, IEEE Transactions on, 19(4):713–722. find
  3. Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. arXiv preprint arXiv:1506.02075. find
  4. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by ex- find
  5. Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. CoRR, abs/1412.3555. find
  6. Dan Gillick, Cliff Brunk, Oriol Vinyals, and Amarnag Subramanya. 2015. Multilingual language processing from bytes. arXiv preprint arXiv:1512.00103. find
  7. Alex Graves. 2013. Generating sequences with recurrent neural networks. arXiv preprint arXiv:1308.0850. find
  8. Jiatao Gu, Zhengdong Lu, Hang Li, and Victor OK Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. arXiv preprint arXiv:1603.06393. find our card
  9. Caglar Gulcehre, Marcin Moczulski, Misha Denil, and Yoshua Bengio. 2016. Noisy activation functions. arXiv preprint arXiv:1603.00391. — [arXiv:1603.00391](https://arxiv.org/abs/1603.00391) find
  10. Michael U Gutmann and Aapo Hyvärinen. 2012. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. The Journal of Machine Learning Research, 13(1):307–361. find
  11. Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for mage recognition. arXiv preprint arXiv:1512.03385. find
  12. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1684–1692. find our card
  13. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2014. On using very large target vocabulary for neural machine translation. arXiv preprint arXiv:1412.2007. find our card
  14. Diederik P Kingma and Jimmy Ba Adam. 2015. A method for stochastic optimization. In International Conference on Learning Representation. find
  15. Minh-Thang Luong, Ilya Sutskever, Quoc V Le, Oriol Vinyals, and Wojciech Zaremba. 2015. Addressing the rare word problem in neural machine translation. In Proceedings of ACL. for scientific computing (Theano Development Team, 2016). We acknowledge the support of the following organizations for resea… find our card
  16. Danielle Matthews, Tanya Behne, Elena Lieven, and Michael Tomasello. 2012. Origins of the human pointing gesture: a training study. Developmental science, 15(6):817– 829. find
  17. Andriy Mnih and Koray Kavukcuoglu. 2013. Learning word embeddings efficiently with noise-contrastive estimation. In Advances in Neural Information Processing Systems, pages 2265–2273. find
  18. Frederic Morin and Yoshua Bengio. 2005. Hierarchical probabilistic neural network language model. In Aistats, volume 5, pages 246–252. Citeseer. find
  19. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063. find
  20. Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2013. How to construct deep recurrent neural networks. arXiv preprint arXiv:1312.6026. find
  21. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. CoRR, abs/1509.00685. find
  22. Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. Signal Processing, IEEE Transactions on, 45(11):2673–2681. find
  23. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909. find
  24. Theano Development Team. 2016. Theano: A Python framework for fast computation of mathematical expressions. arXiv e-prints, abs/1605.02688, May. find our card
  25. Michael Tomasello, Malinda Carpenter, and Ulf Liszkowski. 2007. A new look at infant pointing. Child development, 78(3):705–722. find
  26. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2674–2682. find
  27. Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701. \#\# Acknowledgments We would also like to thank the developers of Theano 5 , for developing such a powerful tool 5 http://deeplearning.net/software/ theano/ ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gulcehre-2016-pointing-unknown-words/