Machine Comprehension Using Match-LSTM and Answer Pointer

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1608.07905

Summary

The paper addresses extractive reading comprehension on SQuAD, a crowdsourced dataset of Wikipedia paragraphs where an answer can be any span of the passage. Earlier end-to-end neural readers either chose among given candidates or returned a single token, so they did not fit this setting. The authors propose a fully neural system with three stages:

Encoding: LSTMs encode the passage and the question.
Matching: a bidirectional match-LSTM, taken from their earlier textual entailment work, reads the passage while attending over the question. The question plays the premise and the passage the hypothesis.
Answer Pointer: a layer built on Pointer Net (Vinyals et al., 2015) outputs positions in the passage rather than words from a vocabulary.

They compare two decoders. A sequence model emits answer positions one by one, and a boundary model predicts only the start and end, optionally with an added span search. The best reported test result is 67.9% exact match and 77.0% F1. This clearly beats the feature-engineered baseline of Rajpurkar et al. (2016) and comes close to the leading system of the time. It shows that answer spans can be learned end to end without hand-built linguistic pipelines.

Key points

Architecture: LSTM preprocessing layer, then a bidirectional match-LSTM layer, then an Answer Pointer (Ans-Ptr) layer. The two proposed models differ only in the final layer: the sequence model allows non-contiguous positions, while the boundary model predicts start and end indices.
Test results: the best system reaches 67.9% EM and 77.0% F1 on the hidden SQuAD test set, and the introduction notes that an ensemble boosts results. The conclusion credits the boundary model with 67.6% EM and 77% F1, so the paper's own figures differ slightly.
Comparison:
- Both models outperform the logistic regression baseline with manual features from Rajpurkar et al. (2016).
- The boundary model beats the sequence model.
- The state of the art at the time (Salesforce Research) was 71.6% EM and 80.4% F1.
Analysis: performance is broken down by answer length and question type. "Why" questions remain weak and are flagged for future work.
Code: released on GitHub as SeqMatchSeq (shuohangwang/SeqMatchSeq).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

120 papers of our library cite it (of the 140673 whose bibliographies we read); 13 of them have a page here, 107 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1608.07905 · arXiv 1608.07905

@misc{wang2016machine,
  title = {{Machine Comprehension Using Match-LSTM and Answer Pointer}},
  author = {Wang, Shuohang and Jiang, Jing},
  year = {2016},
  doi = {10.48550/arxiv.1608.07905},
  eprint = {1608.07905},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1608.07905}
}

References

22 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

Danqi Chen, Jason Bolton, and Christopher D. Manning. A thorough examination of the CNN/Daily Mail reading comprehension task. In Proceedings of the Conference on Association for Computational Linguistics, 2016. find our card
Yiming Cui, Ting Liu, Zhipeng Chen, Shijin Wang, and Guoping Hu. Consensus attention-based neural networks for chinese reading comprehension. In arXiv preprint arXiv:1607.02250, 2016. find
Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. Incorporating copying mechanism in sequence-to-sequence learning. In Proceedings of the Conference on Association for Computational Linguistics, 2016. find our card
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Proceedings of the Conference on Advances in Neural Information Processing Systems, pp. 1693–1701, 2015. find our card
Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. WIKIREADING: A novel large-scale language understanding task over wikipedia. In Proceedings of the Conference on Association for Computational Linguistics, 2016. find
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. The Goldilocks principle: Reading children’s books with explicit memory representations. In Proceedings of the International Conference on Learning Representations, 2016. find our card
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735–1780, 1997. find
Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. Text understanding with the attention sum reader network. In Proceedings of the Conference on Association for Computational Linguistics, 2016. find
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations, 2015. find
Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. Ask me anything: Dynamic memory networks for natural language processing. In Proceedings of the International Conference on Machine Learning, 2016. find our card
Jeffrey Pennington, Richard Socher, and Christopher D Manning. GloVe: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2014. find
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2016. find our card
Matthew Richardson, Christopher JC Burges, and Erin Renshaw. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2013. find our card
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. End-to-end memory networks. In Proceedings of the Conference on Advances in neural information processing systems, 2015. find
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. MovieQA: Understanding stories in movies through question-answering. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2016. find
Adam Trischler, Zheng Ye, Xingdi Yuan, and Kaheer Suleman. Natural language comprehension with the EpiReader. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2016. find
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. In Proceedings of the Conference on Advances in Neural Information Processing Systems, 2015. find
Shuohang Wang and Jing Jiang. Learning natural language inference with LSTM. In Proceedings of the Conference on the North American Chapter of the Association for Computational Linguistics, 2016. find
Jason Weston, Sumit Chopra, and Antoine Bordes. Memory networks. In Proceedings of the International Conference on Learning Representations, 2015. find
Jason Weston, Antoine Bordes, Sumit Chopra, Alexander M Rush, Bart van Merriënboer, Armand Joulin, and Tomas Mikolov. Towards AI-complete question answering: A set of prerequisite toy tasks. In Proceedings of the International Conference on Learning Representations, 2016. find our card
Wenpeng Yin, Sebastian Ebert, and Hinrich Schütze. Attention-based convolutional neural network for machine comprehension. arXiv preprint arXiv:1602.04341, 2016. find
Yang Yu, Wei Zhang, Kazi Hasan, Mo Yu, Bing Xiang, and Bowen Zhou. End-to-end answer chunk extraction and ranking for reading comprehension. arXiv preprint arXiv:1610.09996, 2016. — [arXiv:1610.09996](https://arxiv.org/abs/1610.09996) find