Simple and Effective Multi-Paragraph Reading Comprehension
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1710.10723
Summary
Neural reading comprehension models usually read one paragraph at a time, but real QA systems receive whole documents. Clark and Gardner study how to apply paragraph-level models to document-level input. They first build a stronger pipeline. It picks a paragraph by TF-IDF similarity to the question and trains with an objective that sums probability over every occurrence of the answer string, which handles the noise of distant supervision. The reader combines BiDAF-style attention, residual self-attention, GRUs and variational dropout. They then target confidence-based methods, which run the model on many paragraphs and return the highest-scoring answer. Normally trained models give scores that cannot be compared across paragraphs. The fix is to also train on paragraphs without the answer and to use a shared-normalization objective, where the softmax is normalized jointly over all paragraphs sampled from one document. On TriviaQA web this reaches 71.3 F1, up from 56.7 F1 for the prior best system. This makes the approach practical for open-domain QA.
Key points
- On TriviaQA web the system scores 71.3 F1 on the test set, roughly 15 points above earlier work. Multi-paragraph gains are also shown on TriviaQA unfiltered and on a document-level variant of SQuAD.
- Choosing paragraphs by TF-IDF, with document frequencies computed only within the given documents, raises answer coverage from 83.1% (first paragraph) to 85.1% on TriviaQA web.
- On document-level SQuAD the base model degrades once it sees more than two paragraphs. The shared-norm model peaks at 72.37 F1 / 64.08 EM with 15 paragraphs.
- On the Wikipedia documents used by Chen et al. (2017), the model gets 59.14 EM / 67.34 F1, compared with their 49.7 EM.
- Four alternatives were compared: shared-norm, merge, no-answer and sigmoid. Shared-norm worked best, and sigmoid suffered from label noise. Code is released at github.com/allenai/document-qa, along with an end-to-end demo.
Cited by
- Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
- MS MARCO: A Human Generated MAchine Reading COmprehension Dataset
- InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks
- Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling
- How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites
- QuAC: Question Answering in Context
- Quoref: A Reading Comprehension Dataset with Questions Requiring Coreferential Reasoning
- DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
- AllenNLP: A Deep Semantic Natural Language Processing Platform
- Pretrained Transformers Improve Out-of-Distribution Robustness
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
- Evaluating Open-Domain Question Answering in the Era of Large Language Models
- Which Linguist Invented the Lightbulb? Presupposition Verification for Question-Answering
- Natural Questions: A Benchmark for Question Answering Research
- Augmented Language Models: a Survey
- Passage Re-ranking with BERT
- How Context Affects Language Models' Factual Predictions
- Know What You Don't Know: Unanswerable Questions for SQuAD
- Recent Advances in Natural Language Inference: A Survey of Benchmarks, Resources, and Approaches
- MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
- End-to-End Open-Domain Question Answering with BERTserini
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
- Big Bird: Transformers for Longer Sequences
- ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
- InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models
Cite
@misc{clark2017simple,
title = {{Simple and Effective Multi-Paragraph Reading Comprehension}},
author = {Clark, Christopher and Gardner, Matt},
year = {2017},
doi = {10.48550/arxiv.1710.10723},
eprint = {1710.10723},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.1710.10723}
}References
- Petr Baudiš. 2015. YodaQA: A Modular Question Answering System Pipeline. In POSTER 2015-19th International Student Conference on Electrical Engineering. pages 1156–1165. find
- Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic Parsing on Freebase from Question-Answer Pairs. In EMNLP. find
- Jamie Callan, Mark Hoy, Changkuk Yoo, and Le Zhao. 2009. Clueweb09 Data Set. find
- Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to Answer Open-Domain Questions. arXiv preprint arXiv:1704.00051 . find our card
- Jianpeng Cheng, Li Dong, and Mirella Lapata. 2016. Long Short-Term Memory-Networks for Machine Reading. arXiv preprint arXiv:1601.06733 . find our card
- Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 . find
- Bhuwan Dhingra, Kathryn Mazaitis, and William W Cohen. 2017. Quasar: Datasets for Question Answering by Search and Reading. arXiv preprint arXiv:1707.03904 . — [arXiv:1707.03904](https://arxiv.org/abs/1707.03904) find
- Yarin Gal and Zoubin Ghahramani. 2016. A Theoretically Grounded Application of Dropout in Recurrent Neural Networks. In Advances in neural information processing systems. find our card
- Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching Machines to Read and Comprehend. In Advances in Neural Information Processing Systems. find our card
- Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. 2016. Wikireading: A Novel Large-scale Language Understanding Task over Wikipedia. arXiv preprint arXiv:1608.03542 . find
- Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The Goldilocks Principle: Reading Children’s Books with Explicit Memory Representations. arXiv preprint arXiv:1511.02301 . find our card
- Minghao Hu, Yuxing Peng, and Xipeng Qiu. 2017. Mnemonic Reader: Machine Comprehension with Iterative Aligning and Multi-hop Answer Pointing . find
- Robin Jia and Percy Liang. 2017. Adversarial Examples for Evaluating Reading Comprehension Systems. arXiv preprint arXiv:1707.07328 . find our card
- Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. arXiv preprint arXiv:1705.03551 . find our card
- Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, and Jan Kleindienst. 2016. Text understanding with the attention sum reader network. arXiv preprint arXiv:1603.01547 . find
- Sewon Min, Minjoon Seo, and Hannaneh Hajishirzi. 2017. Question Answering through Transfer Learning from Large Fine-grained Supervision Data. arXiv preprint arXiv:1702.02171 . find
- Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv preprint arXiv:1611.09268 . find our card
- Boyuan Pan, Hao Li, Zhou Zhao, Bin Cao, Deng Cai, and Xiaofei He. 2017. MEMEN: Multi-layer Embedding with Memory Networks for Machine Comprehension. arXiv preprint arXiv:1707.09098 . — [arXiv:1707.09098](https://arxiv.org/abs/1707.09098) find
- Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. In Empirical Methods in Natural Language Processing (EMNLP). find
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. arXiv preprint arXiv:1606.05250 . find our card
- Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional Attention Flow for Machine Comprehension. CoRR abs/1611.01603. find our card
- Chuanqi Tan, Furu Wei, Nan Yang, Weifeng Lv, and Ming Zhou. 2017. S-net: From answer extraction to answer generation for machine reading comprehension. arXiv preprint arXiv:1706.04815 . — [arXiv:1706.04815](https://arxiv.org/abs/1706.04815) find
- Ellen M Voorhees et al. 1999. The TREC-8 Question Answering Track Report. In Trec. find
- Shuohang Wang and Jing Jiang. 2016. Machine Comprehension Using Match-LSTM and Answer Pointer. arXiv preprint arXiv:1608.07905 . find our card
- Shuohang Wang, Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang, Shiyu Chang, Gerald Tesauro, Bowen Zhou, and Jing Jiang. 2017a. R: Reinforced Reader-Ranker for Open-Domain Question Answering. arXiv preprint arXiv:1709.00023 . — [arXiv:1709.00023](https://arxiv.org/abs/1709.00023) find
- Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017b. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). volume 1, pages 189–198. find
- Dirk Weissenborn, Tom Koisk, and Chris Dyer. 2017a. Dynamic Integration of Background Knowledge in Neural NLU Systems. arXiv preprint arXiv:1706.02596 . — [arXiv:1706.02596](https://arxiv.org/abs/1706.02596) find
- Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017b. FastQA: A Simple and Efficient Neural Architecture for Question Answering. arXiv preprint arXiv:1703.04816 . find
- Matthew D Zeiler. 2012. Adadelta: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 . find our card