Reading Wikipedia to Answer Open-Domain Questions
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/P17-1171.pdf
Summary
The paper tackles open-domain factoid question answering where Wikipedia is the only knowledge source, so each answer must be a text span in one of its more than 5 million articles. The authors call this setting machine reading at scale (MRS): a system has to find the right documents first, then read them closely enough to pick out the exact answer. Their system, DrQA, has two parts. Document Retriever is a search module with no learning that ranks articles against the question using TF-IDF over hashed unigram and bigram features. Document Reader is a multi-layer bidirectional LSTM that predicts answer spans in paragraphs. Its inputs are GloVe embeddings, token features, exact-match signals and an attention-based soft alignment with the question. On its own, the reader is competitive with the best single models on SQuAD. For the full system, the authors build distantly supervised training sets from CuratedTREC, WebQuestions and WikiMovies and train one model on all of them together. This beats SQuAD-only training on all four benchmarks. The study shows that reading comprehension alone does not solve open-domain QA and offers a simple retriever-plus-reader design that works on any document collection.
Key points
- Document Retriever hashes bigrams into 2^24 bins with an unsigned murmur3 hash and returns the top 5 articles per question. It beats the built-in ElasticSearch-based Wikipedia Search API and finds an article containing the answer for 77.8% of SQuAD questions.
- Document Reader scores 69.5 EM / 78.8 F1 on the SQuAD dev set and 70.0 EM / 79.0 F1 on test. If both the aligned-embedding and exact-match features are removed, dev F1 falls to 59.4.
- On full Wikipedia, the Multitask (DS) model reaches top-1 exact match of 29.8 on SQuAD, 25.4 on CuratedTREC, 20.7 on WebQuestions and 36.5 on WikiMovies.
- On SQuAD, moving from a given paragraph to all of Wikipedia cuts accuracy sharply (69.5 → 27.1 for the SQuAD-only model). Many errors come from sentences that are on topic but don't contain the answer, and from SQuAD questions that are unclear without their source paragraph.
- YodaQA uses several sources (Freebase, DBpedia and others) and still stays ahead: 31.3 vs 25.4 on CuratedTREC and 39.8 vs 20.7 on WebQuestions.
Cited by
- Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
- Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
- HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
- FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios
- Simple and Effective Multi-Paragraph Reading Comprehension
- Mind2Web: Towards a Generalist Agent for the Web
- ELI5: Long Form Question Answering
- A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models
- MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension
- RARR: Researching and Revising What Language Models Say, Using Language Models
- Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering
- Unsupervised Dense Information Retrieval with Contrastive Learning
- Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
- HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification
- What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams
- Evaluating Open-Domain Question Answering in the Era of Large Language Models
- Dense Passage Retrieval for Open-Domain Question Answering
- RealTime QA: What's the Answer Right Now?
- Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
- Internet-Augmented Dialogue Generation
- Multi-Relational Question Answering from Narratives: Machine Reading and Reasoning in Simulated Worlds
- Latent Retrieval for Weakly Supervised Open Domain Question Answering
- MLQA: Evaluating Cross-lingual Extractive Question Answering
- PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them
- A Unified MRC Framework for Named Entity Recognition
- Entity-Based Knowledge Conflicts in Question Answering
- Fine-Tuning LLaMA for Multi-Stage Text Retrieval
- Query Rewriting in Retrieval-Augmented Large Language Models
- Generation-Augmented Retrieval for Open-domain Question Answering
- Augmented Language Models: a Survey
- Advances in Pre-Training Distributed Word Representations
- ParlAI: A Dialog Research Software Platform
- Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
- Passage Re-ranking with BERT
- Language Models as Knowledge Bases?
- How Context Affects Language Models' Factual Predictions
- KILT: a Benchmark for Knowledge Intensive Language Tasks
- PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations
- RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering
- Learning To Retrieve Prompts for In-Context Learning
- Improving Passage Retrieval with Zero-Shot Question Generation
- RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
- A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge
- Retrieval Augmentation Reduces Hallucination in Conversation
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- FEVER: a Large-scale Dataset for Fact Extraction and VERification
- Break It Down: A Question Understanding Benchmark
- BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis
- HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
- End-to-End Open-Domain Question Answering with BERTserini
- Multilingual Universal Sentence Encoder for Semantic Retrieval
- LinkBERT: Pretraining Language Models with Document Links
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension
- Generate rather than Retrieve: Large Language Models are Strong Context Generators
- IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions
- Defending Against Neural Fake News
- Personalizing Dialogue Agents: I have a dog, do you have pets too?
Cite
@inproceedings{chen2017reading,
title = {{Reading Wikipedia to Answer Open-Domain Questions}},
author = {Chen, Danqi and Fisch, Adam and Weston, Jason and Bordes, Antoine},
booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
year = {2017},
doi = {10.18653/v1/p17-1171},
eprint = {1704.00051},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/p17-1171}
}References
- Eric Brill, Susan Dumais, and Michele Banko. 2002. An analysis of the AskMSR question-answering system. In Empirical Methods in Natural Language Processing (EMNLP). pages 257–264. find
- David Ahn, Valentin Jijkoun, Gilad Mishne, Karin Mller, Maarten de Rijke, and Stefan Schlobach. 2004. Using wikipedia at the trec qa track. In Proceedings of TREC 2004. find
- Davide Buscaldi and Paolo Rosso. 2006. Mining knowledge from Wikipedia for the question answering task. In International Conference on Language Resources and Evaluation (LREC). pages 727–730. find
- Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. 2007. Dbpedia: A nucleus for a web of open data. In The semantic web, Springer, pages 722–735. find
- Rich Caruana. 1998. Multitask learning. In Learning to learn, Springer, pages 95–133. find
- Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In International Conference on Learning Representations (ICLR). find our card
- Petr Baudiš. 2015. YodaQA: a modular question answering system pipeline. In POSTER 2015-19th International Student Conference on Electrical Engineering. pages 1156–1165. find
- Petr Baudiš and Jan Šedivỳ. 2015. Modeling of the question answering task in the YodaQA system. In International Conference of the Cross-Language Evaluation Forum for European Languages. Springer, pages 222–228. find
- Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase Danqi Chen, Jason Bolton, and Christopher D Manning. 2016. A thorough examination of the CNN/Daily Mail reading comprehension task. In Association for Computational Linguistics (ACL). find
- Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In International Conference on Machine Learning (ICML). find
- Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In ACM SIGKDD international conference on Knowledge discovery and data mining. pages 1156–1165. find
- David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A Kalyanpur, Adam Lally, J William Murdock, Eric Nyberg, John Prager, et al. 2010. Building Watson: An overview of the DeepQA project. AI magazine 31(3):59–79. find
- Clinton Gormley and Zachary Tong. 2015. Elasticsearch: The Definitive Guide. ” O’Reilly Media, Inc.”. find
- Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural turing machines. arXiv preprint arXiv:1410.5401 . find
- Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems (NIPS). find our card
- Daniel Hewlett, Alexandre Lacoste, Llion Jones, Illia Polosukhin, Andrew Fandrianto, Jay Han, Matthew Kelcey, and David Berthelot. 2016. Wikireading: A novel large-scale language understanding task over wikipedia. In Association for Computational Linguistics (ACL). pages 1535–1545. find
- Mike Mintz, Steven Bills, Rion Snow, and Daniel Jurafsky. 2009. Distant supervision for relation extraction without labeled data. In Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL/IJCNLP). pages 1003–1011. find
- Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP). pages 1532–1543. find
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). find our card
- Pum-Mo Ryu, Myung-Gil Jang, and Hyun-Ki Kim. 2014. Open domain question answering using Wikipedia-based knowledge model. Information Processing & Management 50(5):683–692. find
- Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The Goldilocks Principle: Reading children’s books with explicit memory representations. In International Conference on Learning Representations (ICLR). find our card
- Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603 . find our card
- Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. 2016. What makes ImageNet good for transfer learning? arXiv preprint arXiv:1608.08614 . — [arXiv:1608.08614](https://arxiv.org/abs/1608.08614) find
- Huan Sun, Hao Ma, Wen-tau Yih, Chen-Tse Tsai, Jingjing Liu, and Ming-Wei Chang. 2015. Open domain question answering via semantic enrichment. In Proceedings of the 24th International Conference on World Wide Web. ACM, pages 1045–1055. find
- Mohit Iyyer, Jordan L Boyd-Graber, Leonardo Max Batista Claudino, Richard Socher, and Hal Daumé III. 2014. A neural network for factoid question answering over paragraphs. In Empirical Methods in Natural Language Processing (EMNLP). pages 633–644. find
- Zhiguo Wang, Haitao Mi, Wael Hamza, and Radu Florian. 2016. Multi-perspective context matching for machine comprehension. arXiv preprint arXiv:1612.04211 . — [arXiv:1612.04211](https://arxiv.org/abs/1612.04211) find
- Rudolf Kadlec, Ondrej Bajgar, and Jan Kleindienst. 2016. From particular to general: A preliminary case study of transfer learning in reading comprehension. Machine Intelligence Workshop, NIPS . find
- Kilian Weinberger, Anirban Dasgupta, John Langford, Alex Smola, and Josh Attenberg. 2009. Feature hashing for large scale multitask learning. In International Conference on Machine Learning (ICML). pages 1113–1120. find
- Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 . find
- Jason Weston, Sumit Chopra, and Antoine Bordes. 2015. Memory networks. In International Conference on Learning Representations (ICLR). find
- Kenton Lee, Tom Kwiatkowski, Ankur Parikh, and Dipanjan Das. 2016. Learning recurrent span representations for extractive question answering. arXiv preprint arXiv:1611.01436 . — [arXiv:1611.01436](https://arxiv.org/abs/1611.01436) find
- Caiming Xiong, Victor Zhong, and Richard Socher. 2016. Dynamic coattention networks for question answering. arXiv preprint arXiv:1611.01604 . find our card
- Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J Bethard, and David Mc-Closky. 2014. The stanford corenlp natural language processing toolkit. In Association for Computational Linguistics (ACL). pages 55–60. find our card
- Alexander H. Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In Empirical Methods in Natural Language Processing (EMNLP). pages 1400– 1409. find