Elamur Research⌕
☰
home / papers / 2016 / Text Understanding with the Attention Sum Reader Network

Text Understanding with the Attention Sum Reader Network

Summary

The paper addresses cloze-style reading comprehension, where a system reads a context document and fills in a missing word in a question by choosing among candidate answers. Evaluation uses four large automatically generated benchmarks: the CNN and Daily Mail news datasets, which contain anonymized entities, and two parts of the Children's Book Test (CBT), common nouns and named entities. The authors propose the Attention Sum Reader (AS Reader). It encodes every document token in context with a bidirectional GRU and encodes the question as a single vector. Dot products between the two, passed through a softmax, give an attention distribution over token positions. The score for a candidate word is the total attention collected over all of its occurrences. The model never builds a blended document representation. Instead, attention works directly as a pointer to the answer. Although it is simpler than competing architectures, an ensemble of AS Readers set a new state of the art on every evaluated dataset. The result shows that a well-chosen inductive bias can outperform heavier designs whenever the answer is guaranteed to appear in the text.

Key points

  • Datasets:
    • CNN has about 380k training queries and Daily Mail about 879k, each with roughly 25–26 candidate entities on average.
    • CBT examples use 20 sentences of a children's story as context, ask about the 21st, and always offer 10 candidates.
  • Architecture:
    • A shared word embedding feeds two encoders.
    • The document encoder is a bidirectional GRU whose concatenated forward and backward states serve as contextual embeddings.
    • The query encoder produces a vector of matching size.
    • Answer probability is the sum of softmax attention over every position where a word occurs ("pointer sum attention", inspired by Pointer Networks).
  • Results:
    • On CNN, the single model with the best validation score reaches 69.5% test accuracy, while the top 20% of models average 69.9%.
    • On CBT named entities, the best single model scores 68.6%, about 2 points above MemNN with self-supervision, and the averaging ensemble beats the previous best by about 4 points.
    • On CBT common nouns, the ensemble reaches 69%, roughly 6 points above MemNN.
  • Top-k accuracy of the greedy ensemble: with five guesses, the correct answer is included 94.8% of the time on CNN and Daily Mail, 96.8% on CBT NE and 95.4% on CBT CN.
  • Limitations and release:
    • The model cannot output an answer absent from the document.
    • Accuracy falls as documents get longer, as candidates multiply, and when the correct answer is rare in the text.
    • The implementation is public (github.com/rkadlec/asreader).

Interesting ideas

  • I1. Attention as the output itself. Using attention scores directly as the prediction, rather than as weights for building a new vector, is a cheap and strong baseline. It could serve any extractive task, such as picking entities, spans or slot values from a source text.
  • I2. Summing over repeated mentions. Pooling probability across all occurrences of a word builds frequency evidence into the model. The authors' error analysis also shows the downside: frequent distractors can win, which is worth watching for when reusing the trick.
  • I3. Anonymizing and shuffling entities per example. This forces a model to rely on the context rather than memorized world knowledge. It is a useful recipe for building comprehension tests in any language that cannot be solved by world knowledge alone.
  • I4. Slicing accuracy by data properties. Breaking results down by document length, number of candidates and answer frequency rank exposes where and why a model fails. The same protocol transfers easily to evaluating other QA or extraction systems.
  • I5. Reporting top-k recall and model averages. Top-k numbers, together with averages over the best validation models, give a fuller picture than a single best-validation score. This helps when a model's outputs feed a reranker or a human-in-the-loop pipeline.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

84 papers of our library cite it (of the 140884 whose bibliographies we read); 19 of them have a page here, 65 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p16-1086

@inproceedings{kadlec2016text,
  title = {{Text Understanding with the Attention Sum Reader Network}},
  author = {Kadlec, Rudolf and Schmid, Martin and Bajgar, Ondrej and Kleindienst, Jan},
  booktitle = {Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2016},
  doi = {10.18653/v1/p16-1086},
  url = {https://doi.org/10.18653/v1/p16-1086}
}

References

22 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1253–1262. find
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. International Conference on Learning Representations. find our card
  3. Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. 2012. On the difficulty of training recurrent neural networks. Proceedings of The 30th International Conference on Machine Learning, pages 1310–1318. find
  4. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. 2012. Theano: new features and speed improvements. find
  5. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop. find
  6. Matthew Richardson, Christopher J C Burges, and Erin Renshaw. 2013. MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text. Empirical Methods in Natural Language Processing (EMNLP), pages 193–203. find our card
  7. Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A Thorough Examination of the CNN / Daily Mail Reading Comprehension Task. In Association for Computational Linguistics (ACL). find our card
  8. Mrinmaya Sachan, Avinava Dubey, Eric P Xing, and Matthew Richardson. 2015. Learning Answer-Entailing Structures for Machine Comprehension. Association for Computational Linguistics (ACL), pages 239–249. find
  9. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. Empirical Methods in Natural Language Processing (EMNLP). find
  10. Andrew M Saxe, James L Mcclelland, and Surya Ganguli. 2014. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. International Conference on Learning Representations. find
  11. Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. arXiv, pages 1–9. find
  12. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: prevent NN from overfitting. Journal of Machine Learning Research, 15:1929–1958. find
  13. David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya a. Kalyanpur, Adam Lally, J. William Murdock, Eric Nyberg, John Prager, Nico Schlaefer, and Chris Welty. 2010. Building Watson: An Overview of the DeepQA Project. AI Magazine, 31(3):59–79. find
  14. Wilson L Taylor. 1953. Cloze procedure: a new tool for measuring readability. Journalism and Mass Communication Quarterly, 30(4):415. find
  15. Bart van Merrienboer, Dzmitry Bahdanau, Vincent Dumoulin, Dmitriy Serdyuk, David Warde-farley, Jan Chorowski, and Yoshua Bengio. 2015. Blocks and Fuel : Frameworks for deep learning. pages 1–5. find
  16. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, pages 1684– 1692. find our card
  17. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2674–2682. find
  18. Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. arXiv preprint arXiv:1410.3916. find
  19. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The goldilocks principle: Reading children’s books with explicit memory representations. arXiv preprint arXiv:1511.02301. find our card
  20. Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: a Method for Stochastic Optimization. International Conference on Learning Representations, pages 1– 13. find
  21. Sosuke Kobayashi, Ran Tian, Naoaki Okazaki, and Kentaro Inui. 2016. Dynamic Entity Representation with Max-pooling Improves Machine Reading. Proceedings of the North American Chapter of the Association for Computational Linguistics and Human Language Technologies (NAACL-HLT). find
  22. Karthik Narasimhan and Regina Barzilay. 2015. Machine Comprehension with Discourse Relations. Proceedings of the 53rd Annual Meeting of the find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/kadlec-2016-text-understanding-attention-sum-reader/